Scaffolding: Federation gateway setup

[klpoland]

Note

High Risk
Touches API authentication scoping, new internal export surface with network ACLs, and irreversible publish rules on datasets—mistakes could block sync, leak export data, or lock dataset lifecycle incorrectly.

Overview
Adds gateway-side federation scaffolding so a local sync service can pull export payloads, with shared secrets and compose wiring at the repo root.

Export API and pipeline: New /api/.../federation/export/* routes (datasets/captures list + detail) gated by FEDERATION_ENABLED, operational health (sync health, Redis, FederationSync API key), and CIDR allowlists for internal callers. Public final datasets and linked captures are serialized for export, written to fed-* OpenSearch indices on change, and announced on a configurable Redis channel; post_save signals schedule reindex after commit.

Auth: Introduces KeySources.FederationSync, mint via get-federation-sync-api-key (DRF token from federation-shared.env), init_federation_sync_token in prepare_gateway when federation is on, and global DisallowFederationSyncKey so sync keys cannot use normal asset APIs. APIKeyAuthentication now returns None without a header (multi-auth friendly) and attaches the UserAPIKey as request.auth.

Data rules: Datasets gain federation_exportable() queryset helpers and validation that final status and public visibility cannot be reverted; detach-from-datasets only clears draft/private links (published links preserved), with federation reindex on dataset disconnect.

Tooling: generate-secrets.sh writes root federation-shared.env; local compose loads it; extensive federation tests and Pydantic contract checks against sds_federation.

^{Reviewed by Cursor Bugbot for commit 16c53ff. Bugbot is set up for automated code reviews on this repo. Configure here.}

[semanticdiff-com]

Review changes with  

Changed Files

File	Status
gateway/sds_gateway/api_methods/views/capture_endpoints.py	63% smaller
gateway/sds_gateway/api_methods/views/file_endpoints.py	48% smaller
gateway/sds_gateway/api_methods/views/dataset_endpoints.py	42% smaller
gateway/sds_gateway/api_methods/tests/test_authenticate.py	41% smaller
gateway/sds_gateway/users/tests/test_quick_add_views.py	39% smaller
gateway/sds_gateway/users/urls.py	27% smaller
gateway/sds_gateway/users/api/views.py	24% smaller
gateway/sds_gateway/api_methods/helpers/extract_drf_metadata.py	15% smaller
gateway/sds_gateway/api_methods/utils/relationship_utils.py	13% smaller
gateway/sds_gateway/api_methods/utils/asset_access_control.py	10% smaller
gateway/config/settings/base.py	5% smaller
gateway/sds_gateway/api_methods/tests/factories.py	5% smaller
.gitignore	Unsupported file format
federation-shared.example.env	Unsupported file format
gateway/.envs/example/django.env	Unsupported file format
gateway/compose.local.yaml	0% smaller
gateway/config/api_router.py	0% smaller
gateway/pyproject.toml	Unsupported file format
gateway/scripts/fallow-cross-file-dupes.sh	Unsupported file format
gateway/scripts/generate-secrets.sh	Unsupported file format
gateway/sds_gateway/api_methods/apps.py	0% smaller
gateway/sds_gateway/api_methods/authentication.py	0% smaller
gateway/sds_gateway/api_methods/federation/__init__.py	0% smaller
gateway/sds_gateway/api_methods/federation/availability.py	0% smaller
gateway/sds_gateway/api_methods/federation/compile_federated_data.py	0% smaller
gateway/sds_gateway/api_methods/federation/events.py	0% smaller
gateway/sds_gateway/api_methods/federation/export_contract.py	0% smaller
gateway/sds_gateway/api_methods/federation/fed_index.py	0% smaller
gateway/sds_gateway/api_methods/federation/permissions.py	0% smaller
gateway/sds_gateway/api_methods/federation/redis_channel.py	0% smaller
gateway/sds_gateway/api_methods/federation/reindex.py	0% smaller
gateway/sds_gateway/api_methods/federation/signals.py	0% smaller
gateway/sds_gateway/api_methods/management/commands/prepare_gateway.py	0% smaller
gateway/sds_gateway/api_methods/models.py	0% smaller
gateway/sds_gateway/api_methods/permissions.py	0% smaller
gateway/sds_gateway/api_methods/serializers/capture_serializers.py	0% smaller
gateway/sds_gateway/api_methods/serializers/dataset_serializers.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_dataset_endpoints.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_dataset_publish_immutability.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_federation_events.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_federation_export.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_federation_export_contract.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_federation_hardening.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_federation_signals.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_federation_sync_api_key.py	0% smaller
gateway/sds_gateway/api_methods/tests/test_file_endpoints.py	0% smaller
gateway/sds_gateway/api_methods/views/federation_endpoints.py	0% smaller
gateway/sds_gateway/users/backend_service_key_utils.py	0% smaller
gateway/sds_gateway/users/management/commands/create_test_files.py	0% smaller
gateway/sds_gateway/users/management/commands/init_federation_sync_token.py	0% smaller
gateway/sds_gateway/users/management/commands/init_svi_token.py	0% smaller
gateway/sds_gateway/users/migrations/0012_alter_userapikey_source_federation_sync.py	0% smaller
gateway/sds_gateway/users/migrations/max_migration.txt	Unsupported file format
gateway/sds_gateway/users/models.py	0% smaller
gateway/sds_gateway/users/svi_utils.py	0% smaller
gateway/sds_gateway/users/tests/test_federation_sync_drf_init.py	0% smaller
gateway/sds_gateway/users/tests/test_svi_utils.py	0% smaller

[klpoland]

Added some dedicated API endpoints, serializers, and signals for federation and configuration that defaults to disabling federation sync.

[klpoland]

Refactor:

• Index into local federated index FIRST then send Redis signal.
• Redis triggers export API call on sync service which queries Opensearch, not Postgres, for doc body.

Hardening:

• Added some cleaning logic on the Dataset model to enforce publishing (final status and/or public) immutability on save, preventing reversion (we may revisit this, I was just noticing as I was building out the logic for federation signals that unpublishing a public dataset would have strange effects on the federation flow. This should prevent/warn if an admin tries to manually unpublish on a peer site for example.)

Logic Updates:

• When dataset is published, the signal checks that it is not deleted, if deleted (before signal) AND an active doc doesn't currently exist, skip indexing/notify if so to prevent stale/deleted pre-bootstrap data from being indexed.
• If the signal is for soft deletion, it checks that the federated doc hasn't been marked as deleted previously and if not, updates and notifies. (this is for BOTH captures and datasets)
• When datasets are indexed/updated, so are their attached (not deleted) captures.
• When a capture is updated (say, when metadata is re-indexed) the signal will also check if it is in a public dataset and index/update/notify accordingly.

[cursor]

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 16c53ff. Configure here.}