Reach READY in headless Kubernetes deployments, and test k8s in CI - #460
Merged
Merged
Conversation
Expansion of HDFGroup#459 Changes from that PR: To the k8s test itself: - Expands k8s GH testing to run the full HSDS test suite instead of just a smoke check. - Adds tests/k8s/rescale_check.py to verify that objects are distributed correctly across DNs on k8s during a rescale. To to hsds/: - k8s_update_dn_info() now sets cluster_state to WAITING when the pod query fails, rather than returning with the last value in place. The failure is per pod, so one pod's query cold fail while another's succeeds, and the two would then have partition results for the same obj_id (which is what the gate was supposed to prevent in the first place). - Set an explicit probe timeoutSeconds in all four manifests. To CI/test config: - Pin k3d rather than tracking main. - Set max_task_count: 0 in the test cluster. kubectl port-forward pins to one pod instead of load balancing, so the whole suite lands on a single service node, where the chunk-heavy tests push active_tasks past the default 100 and causes the HSDS node to be overloaded. Ideally the test would load balance, but this allows the tests to at least run for now. - Widens diagnose() to return more output and grep a few more relevant strings. Misc refactors: - Factors in_cluster() in run.sh, replacing four copies of the "start port-forward, run, report, stop, diagnose" process. - Simplify the cluster_state writing process in k8s_update_dn_info.
To admin/kubernetes/: - Drops HDFGroup#459's probe changes from the three manifests in CI, since they depend upon hsds-node-state, which is only added here and doesn't exist in any external image yet. Eventually these will likely be re-added. - Fixes the AWS manifest's HEAD_PORT. It was set as `value: null`. Kubernetes parsed that as an empty string, causing the HSDS config to override it, and so so head_port kept its default of 5100 and update_dn_info took the head-node branch. This resulted in trying to register forever against a non-existent head node. To hsds/: - Add a guards the dn_ids assignment. - hsds-node-state now bypasses proxies. urlopen's default opener honours http_proxy, so a proxy injected into the pod environment would capture the localhost probe and fail a healthy node. - hsds-node-state now resolves NODE_TYPE=head_node, which is what entrypoint.sh uses, to head_port. To the k8s test: - assert_serving no longer loses its own error handling to errexit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expansion of #459
Changes from that PR:
To the k8s test itself:
check.
across DNs on k8s during a rescale.
To hsds/:
rather than returning with the last value in place. The failure is per pod, so
one pod's query could fail while another's succeeds, and
the two would then have partition results for the same obj_id (which is
what the gate was supposed to prevent in the first place).
To CI/test config:
pod instead of load balancing, so the whole suite lands on a single service
node, where the chunk-heavy tests push active_tasks past the default 100 and
causes the HSDS node to be overloaded. Ideally the test would load balance,
but this allows the tests to at least run for now.
Misc refactors:
run, report, stop, diagnose" process.