fix(dinghy-layer): prune orphaned config files on initial scan

[abarnto]

User description

🤖 This was written by an AI agent on behalf of @abarnto.

What this fixes

A container with VIRTUAL_HOST that is recreated with a new IP could keep a stale backend endpoint in its Traefik route. The URL then returned 502 while the backend was healthy, and no proxy or container restart reconciled it. Only a full teardown that removed the traefik_dynamic volume cleared it.

Closes #109.

Root cause

dinghy-layer writes one Traefik dynamic config file per container, named by the container ID, and removes that file only on the container die event. The Traefik service inside each file is keyed by the container name, not the ID.

HandleInitialScan writes a file for each running managed container but never prunes. When a die event is missed (for example dinghy-layer was not running at the moment the old container was removed), the old <id>.yaml survives as an orphan. Two files then define the same Traefik service name, the file provider merges them, and the stale endpoint can win. A restart does not help, because the scan only adds files, so only wiping the volume cleared the orphan.

The change

HandleInitialScan now reconciles the dynamic directory against the running containers. It collects the config file name of every container it processes into a keep set, then calls a new reconcileConfigs that removes any config file this service owns whose container is no longer present.

The prune is deliberately narrow.

• Only this service's own files are eligible. They match ^[0-9a-f]{12}\.yaml$, the 12-character short container ID plus .yaml. Certificate configs, the entrypoint-generated auto-tls.yml, and the middlewares/ subdirectory sharing the volume are left untouched.
• Reconciliation is skipped when any container fails to inspect during the scan. A transient inspect failure would leave a still-running container out of keep, so pruning in that state could delete a live route. The scan reconciles only when it saw every container cleanly; otherwise the next restart heals the orphan.

processContainer now returns the base name of the file it wrote (empty when it skips the container), which is what feeds the keep set.

This makes spark-http-proxy restart recover a stale state that previously needed a full teardown.

Verification

• go test -race ./... passes. New unit tests cover the file-name matcher (TestIsDinghyConfigFile) and the reconcile behavior: an orphan is removed while the current file and the protected auto-tls.yml, certificate, and middlewares/ entries survive (TestReconcileConfigsRemovesOnlyOrphans), dry-run keeps files, and a missing directory is a no-op.
• gofmt -l ./cmd ./pkg and go vet ./... are clean.
• make test (Docker integration suite) passes locally.

---

PR Type

Bug fix, Tests, Documentation

---

Description

• Prune orphaned dinghy-layer configs after scans

• Preserve shared Traefik files and directories

• Skip pruning when container inspection fails

• Add reconciliation and filename matcher coverage

---

Diagram Walkthrough

flowchart LR
  Scan["Initial container scan"]
  Keep["Current generated config files"]
  Reconcile["Reconcile dynamic config directory"]
  Prune["Remove orphaned container configs"]
  Scan -- "collects" --> Keep
  Keep -- "drives" --> Reconcile
  Reconcile -- "prunes stale files" --> Prune

Loading

File Walkthrough

	Relevant files
Bug fix	main.go Reconcile stale generated Traefik container configurations cmd/dinghy-layer/main.go Track generated config filenames for successfully scanned containers. Reconcile top-level generated config files after error-free scans. Remove only short hexadecimal container-ID .yaml orphan files. Preserve dry-run behavior and skip pruning after inspection failures. +105/-8
Tests	main_test.go Test orphaned configuration reconciliation safeguards        cmd/dinghy-layer/main_test.go Test generated config filename matching boundaries. Verify orphan removal preserves shared files and directories. Cover dry-run and missing dynamic directory behavior. +96/-0
Documentation	CHANGELOG.md Document stale backend configuration fix                                  CHANGELOG.md Document the stale backend IP reconciliation fix. Reference issue #109 and affected 502 behavior. +1/-0

---

---

Assisted-by: pr-agent/gpt-5.6-terra

[sparkfabrik-ai-bot]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis ✅ 109 - PR Code Verified Compliant requirements: Reconcile dinghy-layer dynamic config files against live containers during the initial scan. Remove orphaned top-level files owned by dinghy-layer when their container no longer exists. Restrict pruning to filenames matching ^[0-9a-f]{12}\.yaml$ so shared middleware, TLS, and certificate files are preserved. Avoid pruning when container inspection errors could make the live-container set incomplete. Add unit coverage for the file-name matcher. Requires further human verification: Run go test -race ./.... Run make test. Manually recreate a VIRTUAL_HOST container with a changed IP, restart http-proxy, and confirm routing uses the new backend IP.

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ No major issues detected

---

Assisted-by: pr-agent/gpt-5.6-terra

[sparkfabrik-ai-bot]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Serialize scans with container events Serialize the initial scan, reconciliation, and event handling with a shared CompatibilityLayer mutex. Without coordination, a start event can write a config after the container snapshot is taken but before reconcileConfigs runs, causing the new config to be immediately deleted because it is absent from keep. cmd/dinghy-layer/main.go [109-168] func (cl *CompatibilityLayer) HandleInitialScan(ctx context.Context) error { + cl.processingMu.Lock() + defer cl.processingMu.Unlock() + containers, err := utils.RetryContainerList(ctx, cl.dockerClient, container.ListOptions{}) ... if err := cl.reconcileConfigs(keep); err != nil { cl.logger.Error("Failed to reconcile Traefik configs", "error", err) } return nil } func (cl *CompatibilityLayer) HandleEvent(ctx context.Context, event events.Message) error { + cl.processingMu.Lock() + defer cl.processingMu.Unlock() + switch event.Action { case "start": _, err := cl.processContainer(ctx, event.Actor.ID) return err case "die": return cl.removeTraefikConfig(event.Actor.ID) Suggestion importance[1-10]: 7 __ Why: Concurrent event handling can write a newly started container's config after the scan snapshot is built but before reconcileConfigs prunes unkept files, causing that valid config to be deleted. Serializing HandleInitialScan and HandleEvent would prevent this race, assuming processingMu is available on CompatibilityLayer.	Medium

---

Assisted-by: pr-agent/gpt-5.6-terra

[paolomainardi]

🤖 This was written by an AI agent on behalf of @paolomainardi.

Review summary

The fix is correct and narrowly scoped. Verified against the code:

• No concurrency race: the event loop runs the initial scan to completion before subscribing to Docker events, so reconciliation cannot delete a config written by a concurrent start event.
• Prune scope is safe: the ^[0-9a-f]{12}\.yaml$ pattern matches only files generated by this service. auto-tls.yml, certificate configs, and middlewares/ are untouched.
• The scan-error guard is conservative: any inspect or write failure skips reconciliation entirely, so a transient failure cannot prune a live route.
• Swallowing reconcileConfigs errors is right, since returning them would kill the service on a filesystem hiccup.

End-to-end verification

Reproduced the bug and verified the fix locally, and added an e2e test to the integration suite (commit 2c5446b):

• On main, the orphaned config file survives a dinghy-layer restart (bug reproduced).
• On this branch, the restart prunes the orphan and the recreated container routes correctly.
• Full make test suite passes (7/7) including the new case, and go test -race ./... is green.

Non-blocking follow-up idea

After a Docker daemon restart the event stream resubscribes with a 5s backoff; a die event lost in that window recreates an orphan until the next restart. Running reconciliation after each event-stream reconnect would close that gap.