Repository navigation
Deploy: restart the worker set as one unit, verify the inbound drain, unbuffer worker logs - #82
Open
alexeygrigorev wants to merge 1 commit into
Open
alexeygrigorev wants to merge 1 commit into
alexeygrigorev wants to merge 1 commit into
Conversation
The Oct 10 deploy left datamailer-inbound-email-worker running Oct 1 code while db-worker got the fresh deploy: the restart batch ran under set -e, so one failing unit between them orphaned the rest, and nothing in the deploy path verified the drain afterwards. Restart all worker units in a single transaction, verify the inbound drain post-deploy, and run the workers with PYTHONUNBUFFERED=1 so failures reach journald immediately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the datamailer-side ops gap from the Oct 10-13 inbound-email outage (27 messages dead-lettered over three days while the inbound drain sat stale).
What happened. After the Oct 10 deploy,
datamailer-db-workerwas fresh butdatamailer-inbound-email-workerwas still running code from Oct 1; its enqueue path failed deterministically and every inbound message was retriedmaxReceiveCounttimes into the DLQ. The install script has restarted all workers since 754d940 (Aug 7) underset -euo pipefail, so the evidence (db-worker restarted Oct 10 06:15, drain still "active since Oct 1") means the restart batch aborted between units - and the deploy went red mid-script without anyone connecting that to the stale drain. The post-deployis-activelist did not check the drain at all, and the drain's block-buffered stdout hid three days ofreceived=1 deleted=0 failed=1lines until the manual restart flushed them.Changes
scripts/install_sandbox_worker_service.sh: restart all five worker units in a singlesystemctl restarttransaction - every unit gets a restart attempt even if another fails, and the existingsystemctl is-activeblock still fails the script when a unit did not come up. All worker units now run withPYTHONUNBUFFERED=1so failures land in journald immediately..github/workflows/deploy-sandbox.yml: adddatamailer-inbound-email-workerto the post-deployis-activeverification, so a stale or dead drain fails the deploy instead of passing silently.Sandbox-only (this repo has no production environment).