Skip to content

[tooling] Script to test compatiblity between releases #1624

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
the-glu wants to merge 1 commit into
base: master
Choose a base branch
from
+658 −1
Open

[tooling] Script to test compatiblity between releases #1624

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion release/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@ unset GOOGLE_CLOUD_PROJECT
* `export ZONE_NAME=(google zone name)`
* `export AWS_PROFILE=...` (if needed)


#### Deploy clusters
* `./release/scripts/spawn-clusters.sh`

Expand Down Expand Up @@ -81,3 +80,29 @@ Cleanup resources by:
* Applying terraform destroy to release clusters

No manual cleaning operations are needed after this step.

#### Compatibility matrix
* `./release/scripts/run-compat-matrix.sh`

Test which upgrade paths are possible between released DSS versions, so that the compatibility matrix of the documentation can be updated.

The list of versions to test is the `VERSIONS` array at the top of the script, ordered from oldest to newest. The version being released must be added to it.

Any local DSS instance must be stopped first with `make stop-locally`, otherwise it interferes with the stack started by the script.

For each pair of versions A (older) to B (newer), the script starts a local stack with a single datastore shared by two `core-service` instances, one running version A and the other version B. The datastore is migrated to the latest schema of version B. Pairs where B is older than A are not tested and reported as not evaluated.

Each pair is then validated by running the prober against both instances and the USS qualifier against the pool. A pair passes only if the three runs pass.

The prober and the qualifier run from `MONITORING_IMAGE`, except for pairs involving v0.20.2 which use `V0_20_2_MONITORING_IMAGE`: those results are flagged with a footnote in the generated table.

A local `dummy-oauth` image is built if it is not already present.

The docker compose stack and the qualifier configuration are in `release/compat`.

After this step, the script prints:

* A summary matrix in the terminal
* A Markdown compatibility matrix to copy into the documentation / GitHub release notes

Container logs are available in `release/logs` and qualifier reports in `release/uss_qualifier_output/compat`.
101 changes: 101 additions & 0 deletions release/compat/docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
name: dss-compat

networks:
default:
name: dss-compat

services:

crdb:
image: cockroachdb/cockroach:v24.1.3
command: start-single-node --insecure
healthcheck:
test: curl -f 'http://localhost:8080/health?ready=1' || exit 1
interval: 1s
timeout: 1s
retries: 10

rid-bootstrapper:
image: ${DSS_IMAGE_B}
command: /usr/bin/db-manager migrate --schemas_dir=/db-schemas/rid --db_version latest ${DATASTORE_FLAG_B} crdb
depends_on:
crdb:
condition: service_healthy

scd-bootstrapper:
image: ${DSS_IMAGE_B}
command: /usr/bin/db-manager migrate --schemas_dir=/db-schemas/scd --db_version latest ${DATASTORE_FLAG_B} crdb
depends_on:
crdb:
condition: service_healthy

aux-bootstrapper:
image: ${DSS_IMAGE_B}
command: /usr/bin/db-manager migrate --schemas_dir=/db-schemas/aux_ --db_version latest ${DATASTORE_FLAG_B} crdb
profiles: ["with-aux"]
depends_on:
crdb:
condition: service_healthy

core-service-a:
image: ${DSS_IMAGE_A}
volumes:
- ../../build/test-certs:/var/test-certs:ro
command: >
/usr/bin/core-service ${DATASTORE_FLAG_A} crdb ${PUBLIC_ENDPOINT_A}
-public_key_files /var/test-certs/auth2.pem
-log_format console
-addr :8082
-accepted_jwt_audiences localhost,core-service-a,core-service-b
-enable_scd
-allow_http_base_urls
-locality dss-compat-a
depends_on:
rid-bootstrapper:
condition: service_completed_successfully
scd-bootstrapper:
condition: service_completed_successfully
aux-bootstrapper:
condition: service_completed_successfully
required: false
healthcheck:
test: wget -O - 'http://localhost:8082/healthy' || exit 1
interval: 1s
timeout: 1s
retries: 10

core-service-b:
image: ${DSS_IMAGE_B}
volumes:
- ../../build/test-certs:/var/test-certs:ro
command: >
/usr/bin/core-service ${DATASTORE_FLAG_B} crdb ${PUBLIC_ENDPOINT_B}
-public_key_files /var/test-certs/auth2.pem
-log_format console
-addr :8082
-accepted_jwt_audiences localhost,core-service-a,core-service-b
-enable_scd
-allow_http_base_urls
-locality dss-compat-b
depends_on:
rid-bootstrapper:
condition: service_completed_successfully
scd-bootstrapper:
condition: service_completed_successfully
aux-bootstrapper:
condition: service_completed_successfully
required: false
healthcheck:
test: wget -O - 'http://localhost:8082/healthy' || exit 1
interval: 1s
timeout: 1s
retries: 10

oauth:
image: interuss-local/dummy-oauth
command: -private_key_file /var/test-certs/auth2.key
healthcheck:
test: wget -O - 'http://localhost:8085/token?intended_audience=-&scope=-' || exit 1
interval: 5s
timeout: 5s
retries: 20
254 changes: 254 additions & 0 deletions release/compat/qualifier_config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,254 @@
v1:
artifacts:
output_path: output/compat
raw_report:
indent: null
redact_access_tokens: true
tested_requirements:
- participant_requirements:
uss1: all_astm_dss_requirements
uss2: all_astm_dss_requirements
report_name: requirements
requirement_collections:
all_astm_dss_requirements:
requirement_collections:
- requirement_sets:
- astm.f3411.v22a.dss_provider
- astm.f3411.v19.dss_provider
- astm.f3548.v21.dss_provider
test_run:
action:
on_failure: Continue
test_suite:
resources:
f3411v19_dss_instances: netrid_dss_instances_v19
f3411v22a_dss_instances: netrid_dss_instances_v22a
f3548v21_dss_instances: scd_dss_instances
flight_intents: che_non_conflicting_flights
id_generator: id_generator
planning_area: kentland_planning_area
planning_area_volume: kentland_planning_area_volume
problematically_big_area: kentland_problematically_big_area
second_utm_auth: second_utm_auth
service_area: kentland_service_area
test_exclusions: test_exclusions
utm_client_identity: utm_client_identity
suite_type: suites.interuss.dss.all_tests
execution:
include_action_when: null
skip_action_when: null
stop_fast: false
stop_when_resource_not_created: false
non_baseline_inputs:
- v1.test_run.resources.resource_declarations.utm_auth
- v1.test_run.resources.resource_declarations.second_utm_auth
- v1.test_run.resources.resource_declarations.scd_dss_instances
- v1.test_run.resources.resource_declarations.netrid_dss_instances_v22a
- v1.test_run.resources.resource_declarations.netrid_dss_instances_v19
resources:
resource_declarations:
che_non_conflicting_flights:
dependencies: {}
resource_type: resources.flight_planning.FlightIntentsResource
specification:
file:
path: test_data.flight_intents.standard.non_conflicting
transformations:
- relative_translation:
degrees_east: 7.4774
degrees_north: 46.9748
meters_up: 605
id_generator:
dependencies:
client_identity: utm_client_identity
resource_type: resources.interuss.IDGeneratorResource
specification: {}
kentland_planning_area_volume:
resource_type: resources.VolumeResource
specification:
template:
outline_polygon:
vertices:
- lat: 37.1853
lng: -80.6140
- lat: 37.2148
lng: -80.6140
- lat: 37.2148
lng: -80.5440
- lat: 37.1853
lng: -80.5440
altitude_lower:
value: 0
reference: W84
units: M
altitude_upper:
value: 3048
reference: W84
units: M
kentland_planning_area:
resource_type: resources.PlanningAreaResource
dependencies:
volume: kentland_planning_area_volume
specification:
base_url: https://uss_qualifier.test.utm/interuss/dss/build/dev/dss_probing_qualifier_config/kentland_planning_area
kentland_problematically_big_area:
resource_type: resources.VolumeResource
specification:
template:
outline_polygon:
vertices:
- lat: 38
lng: -81
- lat: 37
lng: -81
- lat: 37
lng: -80
- lat: 38
lng: -80
kentland_service_area_volume:
resource_type: resources.VolumeResource
specification:
template:
outline_polygon:
vertices:
- lat: 37.1853
lng: -80.6140
- lat: 37.2148
lng: -80.6140
- lat: 37.2148
lng: -80.5440
- lat: 37.1853
lng: -80.5440
altitude_lower:
value: 0
reference: W84
units: M
altitude_upper:
value: 3048
reference: W84
units: M
start_time:
offset_from:
starting_from:
time_during_test: TimeOfEvaluation
offset: 1s
end_time:
offset_from:
starting_from:
time_during_test: TimeOfEvaluation
offset: 1h0m1s
kentland_service_area:
resource_type: resources.netrid.ServiceAreaResource
dependencies:
volume: kentland_service_area_volume
specification:
base_url: https://uss_qualifier.test.utm/interuss/dss/build/dev/dss_probing_qualifier_config/kentland_service_area
netrid_dss_instances_v19:
dependencies:
auth_adapter: utm_auth
resource_type: resources.astm.f3411.DSSInstancesResource
specification:
dss_instances:
- base_url: http://core-service-a:8082
has_private_address: true
participant_id: uss1
rid_version: F3411-19
- base_url: http://core-service-b:8082
has_private_address: true
participant_id: uss2
rid_version: F3411-19
netrid_dss_instances_v22a:
dependencies:
auth_adapter: utm_auth
resource_type: resources.astm.f3411.DSSInstancesResource
specification:
dss_instances:
- base_url: http://core-service-a:8082/rid/v2
has_private_address: true
participant_id: uss1
rid_version: F3411-22a
- base_url: http://core-service-b:8082/rid/v2
has_private_address: true
participant_id: uss2
rid_version: F3411-22a
scd_dss_instances:
dependencies:
auth_adapter: utm_auth
resource_type: resources.astm.f3548.v21.DSSInstancesResource
specification:
dss_instances:
- base_url: http://core-service-a:8082
has_private_address: true
participant_id: uss1
supports_ovn_request: true
user_participant_ids:
- mock_uss
- base_url: http://core-service-b:8082
has_private_address: true
participant_id: uss2
supports_ovn_request: true
user_participant_ids:
- mock_uss
second_utm_auth:
dependencies: {}
resource_type: resources.communications.AuthAdapterResource
specification:
environment_variable_containing_auth_spec: AUTH_SPEC_2
scopes_authorized:
- utm.strategic_coordination
test_exclusions:
dependencies: {}
resource_type: resources.dev.TestExclusionsResource
specification:
allow_cleartext_queries: true
allow_private_addresses: true
utm_auth:
dependencies: {}
resource_type: resources.communications.AuthAdapterResource
specification:
environment_variable_containing_auth_spec: AUTH_SPEC
scopes_authorized:
- rid.inject_test_data
- dss.read.identification_service_areas
- rid.service_provider
- rid.display_provider
- dss.write.identification_service_areas
- dss.read.identification_service_areas
- interuss.flight_planning.direct_automated_test
- interuss.flight_planning.plan
- utm.inject_test_data
- utm.strategic_coordination
- utm.conformance_monitoring_sa
- utm.availability_arbitration
- utm.constraint_management
- interuss.versioning.read_system_versions
- interuss.geospatial_map.query
- ''
utm_client_identity:
dependencies:
auth_adapter: utm_auth
resource_type: resources.communications.ClientIdentityResource
specification:
whoami_audience: localhost
whoami_scope: rid.display_provider
validation:
criteria:
- applicability:
test_scenarios: {}
pass_condition:
each_element:
has_execution_error: false
- applicability:
failed_checks:
has_severity:
higher_than: Low
pass_condition:
elements:
count:
equal_to: 0.0
- applicability:
skipped_actions: {}
pass_condition:
elements:
count:
equal_to: 6.0
Loading
Loading