Skip to content

fix(deployment): mirror kubernetes-secret-generator chart via GHCR - #7199

Merged
corneliusroemer merged 6 commits into
mainfrom
fix/mirror-secret-generator-chart
Sep 2, 2026
Merged

fix(deployment): mirror kubernetes-secret-generator chart via GHCR#7199
corneliusroemer merged 6 commits into
mainfrom
fix/mirror-secret-generator-chart

Conversation

@corneliusroemer-agent

@corneliusroemer-agent corneliusroemer-agent commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

@anna-parker noticed CI failure due to helm chart pull failing: https://loculus.slack.com/archives/C05G172HL6L/p1787904000499649?thread_ts=1787903822.341089&cid=C05G172HL6L

helm.mittwald.de (the repo hosting the kubernetes-secret-generator chart used by secrets.yaml) times out intermittently on helm repo add/helm repo update, and install_secret_generator() runs that on every ./deploy.py cluster invocation, i.e. every integration-test run — see mittwald/kubernetes-secret-generator#115, which I filed a year ago and is still open with no maintainer response.

OCI chart references don't need a repo index, so this also removes the helm repo add/helm repo update calls entirely rather than just pointing them somewhere else.

Mirroring the chart

Done once, manually, rather than via a GitHub Action, to keep this simple. To refresh after a version bump upstream:

helm pull kubernetes-secret-generator --repo https://helm.mittwald.de --version <version>
gh auth refresh -h github.com -s write:packages
gh auth token | helm registry login ghcr.io -u <your-github-username> --password-stdin
helm push kubernetes-secret-generator-<version>.tgz oci://ghcr.io/loculus-project

The package defaults to private on first push — needs gh api --method PATCH orgs/loculus-project/packages/container/kubernetes-secret-generator -f visibility=public (or the same via the package's GitHub settings page) afterward, otherwise CI's anonymous pull 401s.

🚀 Preview: Add preview label to enable

helm.mittwald.de times out intermittently (mittwald/kubernetes-secret-generator#115,
filed a year ago, still unfixed), and install_secret_generator() re-runs
`helm repo add`/`repo update` against it on every cluster setup, so this was
flaking integration-test CI runs regularly.

Mirrored kubernetes-secret-generator 3.4.1 as an OCI artifact at
ghcr.io/loculus-project/kubernetes-secret-generator and switched deploy.py to
pull from there instead, pinned to that version. Upstream releases are rare
(3.4.0 -> 3.4.1 was a 3-year gap) so pinning costs little, and OCI charts skip
the repo-add/update step entirely.
@claude claude Bot added the deployment Code changes targetting the deployment infrastructure label Aug 28, 2026
@corneliusroemer
corneliusroemer marked this pull request as ready for review August 28, 2026 14:04
Copilot AI lite review requested due to automatic review settings August 28, 2026 14:04
@claude

claude Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Claude finished @corneliusroemer's task in 1m 54s —— View job


Code review

The change is the right fix for the flake: OCI refs skip the repo index entirely, so dropping helm repo add/helm repo update removes two network round-trips per ./deploy.py cluster rather than just relocating them. Helm 3.18.3 (pinned in all three workflows via azure/setup-helm@v5) has full OCI support, so no version concerns there. Adding --version alongside the mirror is a good call — an unpinned OCI ref would otherwise resolve to whatever tag happens to be in GHCR.

Two inline comments on the in-code refresh recipe and the pin's maintenance story. Beyond those:

Docs are now inconsistentdocs/src/content/docs/for-administrators/setup-with-k3d-and-nginx.mdx:106-111 still instructs administrators to helm repo add mittwald https://helm.mittwald.de and install unpinned from there. That's a defensible choice (external operators arguably shouldn't depend on the Loculus GHCR mirror), but it means the exact endpoint this PR is routing around is still the documented path for anyone following the setup guide, and they hit it during initial setup where a timeout is most confusing. Either point the docs at the mirror too, or add a sentence noting the upstream repo is flaky and the mirror exists as a fallback. Fix this →

Scope of the fix — this mirrors the chart only. The controller image the chart references is still pulled from its upstream registry at pod-start, so image-pull flakes remain possible; they're just far less likely than the helm repo calls since kubelet retries with backoff. Worth knowing when reading future CI failures, not something to change here.

Nothing blocking. I could not verify the mirrored artifact is anonymously pullable (helm show chart oci://... needs network access I don't have here) — worth eyeballing that the next CI run gets past install_secret_generator before merging, since a private package would fail identically for every contributor.

  • Gather context
  • Review deploy.py changes
  • Check other references to the mittwald repo/chart
  • Post review
    · Branch

Comment thread deploy.py Outdated
Comment thread deploy.py

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves deployment and CI reliability by eliminating intermittent timeouts when installing the kubernetes-secret-generator Helm chart from https://helm.mittwald.de. Instead, it switches installation to a mirrored OCI-hosted chart on GHCR and pins the chart version to avoid reliance on a Helm repo index.

Changes:

  • Remove helm repo add / helm repo update usage from install_secret_generator() to avoid flaky network dependency.
  • Install kubernetes-secret-generator from oci://ghcr.io/loculus-project/kubernetes-secret-generator and pin to version 3.4.1.
  • Add inline instructions in deploy.py for how to refresh the mirrored chart after an upstream bump.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread deploy.py Outdated
corneliusroemer and others added 3 commits September 2, 2026 14:47
Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

@anna-parker anna-parker left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!!!

@corneliusroemer

Copy link
Copy Markdown
Contributor

CI failure in dev test ubuntu is unrelated to the code touched here. I'm making a separate PR to deflake that particular CI failure

@corneliusroemer
corneliusroemer merged commit c2ff65a into main Sep 2, 2026
43 of 44 checks passed
@corneliusroemer
corneliusroemer deleted the fix/mirror-secret-generator-chart branch September 2, 2026 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

deployment Code changes targetting the deployment infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants