Repository navigation
Conversation
Generation-1 aes keys stay active so existing volume mounts keep authenticating. Cipher restriction is a later change. Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Deletes the aes keys kept for existing mounts. Every ceph-block mount has been rescheduled onto the aes256k key (sleds drained one at a time, no client.csi-*.1 sessions on any mon). Re-landing #5656 against rebuild: it merged into its stacked base branch instead of rebuild.
….npmjs.org (#5661) The node:alpine plugin manager (musl) stops on the router's NOERROR/no-data answer for registry.npmjs.org.holthome.net, so npm fails with ENOTFOUND and the pod crash-loops. A post-render patch adds dnsConfig ndots:1.
…ions (#5662) Add gpt-6.1-sol, gpt-6.1-terra, gpt-6-astra and gpt-6-luna from Azure resource ryholt-foundry (project proj-default, separate subscription) on the v1 API with a new AZURE_FOUNDRY_API_KEY. The pinned image predates GPT-6, so the azure/gpt5_series/ route supplies reasoning-model param mapping; cost tracking reads $0 until the image is bumped. Move every ryholt-simplechat-aifoundry deployment to 2025-04-01-preview. gpt-5-codex and gpt-5-pro are bridged to the Responses API, which 2024-12-01-preview does not serve. Shared params are YAML anchors. Enable the prometheus callback so the ServiceMonitor's /metrics scrape returns data. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… resource (#5663) Every deployment on ryholt-simplechat-aifoundry (gpt-4o, gpt-5, gpt-5-chat, gpt-5-codex, gpt-5-pro, gpt-5.1, gpt-5.1-codex, o3-mini, text-embedding-3-small) returns 401 from Azure. That environment is retired; the GPT-6 deployments on ryholt-foundry replace them and all four answer through the proxy. Remove the entries and AZURE_API_KEY. No Azure embedding model remains. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Bump ghcr.io/berriai/litellm v1.100.1 -> v1.106.6, pinned by digest. Renovate never proposed this because it does not scan the rebuild branch. The new cost map prices gpt-6.1-sol, gpt-6-astra and gpt-6-luna. Add decision-1, the Decision-1 deployment of Microsoft-Decision-1 on ryholt-foundry, via azure_ai/ on the services.ai.azure.com host. It is served on /v1/decisions and /v1/systemone (1.105+). base_model points cost tracking at azure_ai/Microsoft-Decision-1. Prisma migrates the schema forward on startup. CNPG Backup postgres17-pre-litellm-1-106-6 was taken first for rollback. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Runner pods run workflow code and could reach the node LAN, the router, the LoadBalancer pool, other pods and the Kubernetes API. The new CiliumNetworkPolicy selects runner pods by ARC's actions-ephemeral-runner label and allows only kube-dns and public IPv4. Listeners and the controller are not selected. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Repo-scoped ARC scale sets for atrium, mcp, network-config, schoolhouse and whiskey-whiskey-whiskey (runs-on: <name>-k8s), moving their CI off the Forge runners. These jobs use no Docker, so the overlays drop dind and the runner pods are not privileged. atrium's and mcp's Nix jobs get their own scale sets later, once their privilege needs are known. The GitHub App installation must include these repos first. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Kata VM sandboxes do not see the host cgroup socket-LB hook. With kube-proxy replacement already on, hostNamespaceOnly leaves host socket LB in place and sends pod service traffic through the tc load balancer. Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The Image Factory schematic for Talos v1.14.5 now includes siderolabs/kata-containers beside intel-ucode and nut-client. Flux reconciles the kata and kata-qemu RuntimeClasses that extension registers in containerd. Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The chart default leaves ceph-filesystem with no MDS placement, so both pods can share a node. Require hostname anti-affinity on app=rook-ceph-mds. Control planes are schedulable, so no extra toleration is added. Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…et site (#5670) * feat(matrix): add Tuwunel homeserver and Hermes bot provisioning Tuwunel v1.9.4 in namespace matrix, server name ryanholt.net, client and federation host matrix.ryanholt.net on envoy-external, apex /.well-known/matrix delegation. RocksDB on a ceph-block PVC, one replica, Recreate, Pod Security restricted, default-deny Cilium policy. Registration is token-only from a SOPS Secret; no first-user admin. Appservice hermes-bots owns @hermes-.*:ryanholt.net exclusively, rate limiting off. VolSync restic backup waits on a new 1Password item. A separate provisioner Kustomization ships the script and a per-bot template (Job, get/patch-only RBAC on one Secret). No bot is wired. ADR 0010 and docs/matrix.md. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * chore(matrix): drop stray __pycache__ from provisioner Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(web): serve ryanholt.net apex site; back up Tuwunel to R2 Add namespace web with a static ryanholt.net site (nginx-unprivileged, ConfigMap content, restricted, default-deny Cilium, no egress) and a www -> apex redirect. /.well-known/matrix/ keeps its longer-prefix route to Tuwunel, so the apex is shared rather than given to Matrix. Point the Tuwunel VolSync source at v1's existing 1Password item volsync-r2-template instead of a new item, using v1's per-app path. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(matrix): apex has no stale record; no manual DNS step Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Merge only after @ryan:ryanholt.net has registered. The pod restarts on the ConfigMap change and runs the command at startup. Revert in the next pull request. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Set encryption_enabled_by_default_for_room_type = "all". Tested against v1.9.4: rooms created with the public_chat and private_chat presets both get m.room.encryption (m.megolm.v1.aes-sha2) without the client asking. Document which layers are encrypted and what bots need. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The previous token was shown in a terminal during first-admin setup. New 32-byte value, encrypted to the same age and PGP recipients. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…LM key provisioning (#5675) * feat(hermes): deploy Chief, a Hermes agent on Matrix as @chief Namespace hermes (restricted, default-deny Cilium). Chief runs Hermes Agent v0.22.6 in a kata-qemu VM as uid 10000 with no capabilities and a read-only root: shareProcessNamespace keeps the entrypoint off PID 1, so the image bootstraps without s6 and accepts its own hermes UID. Model claude-sonnet via in-cluster LiteLLM (custom provider, key from 1Password item hermes-chief). Matrix E2EE required, allowlist @ryan. Data volume backed up to R2. Bot Matrix IDs drop the hermes- prefix. The hermes-bots appservice namespace becomes an explicit name list (^@(chief):ryanholt\.net$) so the as_token cannot act as other users; the registration was re-created with fresh as_token/hs_token (nothing used the old ones). Tested locally under podman with the pod's flags: bootstrap and config set on an empty volume, gateway joins Tuwunel v1.9.4 with cross-signing and auto-joins an encrypted DM; the appservice registers @chief and gets M_EXCLUSIVE for @ryan. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(litellm): provision virtual keys in-cluster; use it for Chief Add kubernetes/apps/litellm/key-provisioner: a shared script and a per-key template. The Job runs in namespace litellm with the master key from litellm-env, finds or creates the team, reconciles an existing key's models and budget, or deletes and regenerates the key when the consumer's Secret has lost it, then patches LITELLM_API_KEY into that one Secret (get/patch RBAC on that name only; egress DNS, LiteLLM, kube API). Chief gets key hermes-chief in team hermes, claude-sonnet only, 25 USD per 30 days, through Kustomization chief-litellm. The 1Password item for the key and its ExternalSecret are gone. BUDGET is one "<USD>/<duration>" string because Flux substitutes after kustomize build drops quotes; a bare number would reach the API server as an int. Found by substituting the built template and running kubeconform. Tested against LiteLLM v1.106.6 with Postgres: first run, re-run (key unchanged), budget change applied in place, lost Secret (old key 401, one key left under the alias). Ignore __pycache__ from local test runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Both provisioning Jobs (matrix/hermes-provision-chief and litellm/litellm-key-hermes-chief) failed on their first call to an in-cluster Service: "Name has no usable address". Pods search holthome.net last with ndots:5, so tuwunel.matrix.svc.cluster.local is first tried with the cluster suffixes (NXDOMAIN) and then as ....holthome.net, which Cloudflare and the router answer NODATA. musl (Alpine) treats that NODATA as final; glibc keeps searching and tries the name as given. Reproduced with podman: Alpine cannot resolve even matrix.ryanholt.net with that search list, Debian can. Same cause as the Headlamp ndots note and the CoreDNS holthome.net.holthome.net block. - Image python:3.15.2-alpine3.24 -> python:3.15.2-slim-trixie (glibc). - restartPolicy Never, so failed attempts keep their pods and logs; with OnFailure the Job deleted the pod when it gave up and the logs went too. - request() returns status 0 with "host: reason" for DNS, refused, and timeout errors instead of an uncaught traceback. The pod spec changes, so the bots' Kustomizations (force: true) replace the failed Jobs. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Chief's cross-signing bootstrap failed with M_UNRECOGNIZED: Not Found. Envoy Gateway 1.9.3 defaults escapedSlashesAction to UnescapeAndRedirect, so PUT .../account_data/m.secret_storage.key.ab%2Fcd got a 307 to .../ab/cd, a path Tuwunel does not route. Confirmed against envoy-external directly (10.45.10.51), not only through Cloudflare; %2B, %3D, and + already passed. Matrix clients percent-encode "/" in path parameters. ClientTrafficPolicy envoy-external-https sets KeepUnchanged on the external HTTPS listener only. gateway-auth.md notes what that means for any future path-based rule there. Chief: set database.journal_mode delete in the init step. /opt/data is virtiofs inside the Kata VM; Hermes already fell back to DELETE and warned for every database. The HelmRelease change also restarts Chief, which retries the cross-signing bootstrap (no recovery key file was written). Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ExternalSecret chief-matrix-recovery maps 1Password hermes-chief/ matrix_recovery_key to MATRIX_RECOVERY_KEY, wired into Chief as an optional envFrom. With it, Hermes loads the cross-signing keys from secret storage on every start and signs the current device, so a rebuilt or restored crypto store comes back verified. Until the item exists the pod starts without it; Reloader restarts the pod when the Secret appears. envFrom rather than env valueFrom: app-template 5.3.0's values schema rejects optional on an env secretKeyRef. kubeconform does not check chart values; helm template against the pulled chart does. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#5680) kubeconform validates the HelmRelease objects but not spec.values, so a value the chart's values.schema.json rejects (for example optional: true on an app-template 5.3.0 secretKeyRef) only failed at helm upgrade. scripts/helm-template.sh builds each Flux Kustomization, applies postBuild substitution from cluster-secrets with flux envsubst --strict, pulls the chart from its OCIRepository or HelmRepository (cached by url and ref), and runs helm template on the values. validate.sh and a new CI job run it. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Kubelet copies the node search list into every pod, after the cluster
suffixes, with ndots:5. holthome.net is DNSSEC-signed on Cloudflare, which
answers a name that does not exist with NOERROR and no records ("black
lies"), and the router passes that on. musl (Alpine) stops a search at
that answer, so Alpine pods could not resolve x.ns.svc.cluster.local or
any public name; glibc keeps going. CoreDNS autopath does not help: it
only moves on after NXDOMAIN.
Reproduced with podman against the router: with holthome.net in the
search list Alpine fails matrix.ryanholt.net; without it, it resolves.
ndots:1 also works but makes musl skip the search list for any dotted
relative name (tuwunel.matrix, kubernetes.default.svc).
domains: [] also clears a DHCP-provided list. Internal names are written
in full (forge.holthome.net). The CoreDNS holthome.net.holthome.net block
and the Headlamp ndots:1 patch come out in a follow-up, after this is
applied and pods have been recreated with the new search list.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…hief to hermes-chief (#5681) * feat(hermes): screen pod image with sshd as pn images/hermes-screen vendors NousResearch/hermes-agent v0.22.6 docker/sandbox-desktop.Dockerfile (Xvnc, Xfce, headed Playwright Chromium, cua-driver, user pn). nousresearch/hermes-sandbox is not published for v0.22.x. Local changes: base pinned by digest, Playwright pinned to 1.64.1, pn's NOPASSWD sudo removed, openssh-server run as pn on port 22 (key-only, no root, no PAM, no AcceptEnv, no forwarding). Tested locally on the pinned base: sshd listens on 22 as uid 1000 with all caps dropped and no-new-privs (ip_unprivileged_port_start=22); the agent key logs in; a wrong key, root, and password auth are refused; scp (SFTP) and tar-over-ssh work; SendEnv values do not arrive; -L forwarding is refused. hermes-screen-image.yaml builds it to ghcr.io/carpenike/hermes-screen with the built-in GITHUB_TOKEN. AGENTS.md gets the one carve-out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(hermes): per-bot component with agent and screen pods; move Chief to hermes-chief kubernetes/components/hermes-bot runs each Hermes bot in its own hermes-<bot> namespace as two pods: - agent (runc): hermes gateway run, the only holder of the Matrix token and LiteLLM key (as OPENAI_API_KEY, which Hermes scrubs from child processes). terminal.backend is ssh to the screen; no local fallback. Skills are mounted read-only so file_sync cannot pull changes back. - screen (kata-qemu): upstream sandbox-desktop image plus a key-only, non-root sshd. No credentials, egress to the public internet only. A bot is a configMapGenerator of eight inputs under kubernetes/apps/hermes/<bot>. The component provisions the Matrix login, LiteLLM key, and ssh keys with Jobs that refuse unlabelled Secrets. Chief's state restores from its existing R2 path; the old namespace stays at zero replicas until Chief answers from the new one. Also: trusted_servers = [] in Tuwunel, and %2F KeepUnchanged scoped to a dedicated https-matrix listener (with cloudflared originServerName). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hermes): set the screen's session env in sshd, not /etc/environment UsePAM no means pam_env never reads /etc/environment, so ssh sessions lacked XDG_RUNTIME_DIR and PLAYWRIGHT_BROWSERS_PATH. SetEnv applies them server-side. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(hermes): review fixes for the per-bot component - envoy-external: both HTTPS listeners share ryanholt-net-production-tls, so Envoy Gateway set TLSOverlaps and forced ALPN http/1.1 on both. Set h2,http/1.1 explicitly with ClientTrafficPolicies on each listener; cloudflared keeps one transport per ingress rule, so no coalescing. - agent: TERMINAL_ENV=ssh as a backstop for terminal.backend. - screen: pin the rebuilt image (sshd SetEnv for XDG_RUNTIME_DIR and PLAYWRIGHT_BROWSERS_PATH). - scripts/helm-template.sh: also render Flux Kustomizations built from directories that use a Kustomize Component, resolving substituteFrom ConfigMaps from the same build. - docs/hermes.md: gate the pre-merge backup on a restic "snapshot saved" line, check the restore before trusting Chief, real rollback and cleanup (legacy is not a rollback point; PVs are reclaim Delete), Chromium runs with --no-sandbox, OPENAI_API_KEY fallback in media tools, backup time in UTC. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…5682) Chief runs from kubernetes/components/hermes-bot in hermes-chief and has answered in an encrypted room, so the stopped legacy deployment goes. Flux prunes Namespace hermes with it; the chief-data PV was set to Retain beforehand. docs/hermes.md: the restore check now greps restic's "restoring snapshot" line (VolSync's log filter drops "Selected restic snapshot"). The Chief-specific migration and rollback steps become generic backup, restore-check and in-place restore steps, plus a record of how Chief moved. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ef (#5683) New input API_SERVER_ENABLED (default false). When true, the hermes HelmRelease adds, all gated on that one value: - ExternalSecret api-server: API_SERVER_KEY from 1Password hermes-<bot>/api_server_key, plus API_SERVER_ENABLED/HOST/PORT/ MODEL_NAME; the agent reads it through an optional envFrom - Service agent-api (8642, agent pod only) - HTTPRoute agent-api on envoy-internal only, <bot>-api.${SECRET_DOMAIN}, labelled public (bearer clients; Hermes checks the key) - CiliumNetworkPolicy agent-api: 8642 from envoy-internal proxies only - ReferenceGrant for pocket-id-exempt The screen pod is unchanged. The component now includes the sops component so the app Kustomization can substitute SECRET_DOMAIN. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…y wins (#5684) Hermes loads $HERMES_HOME/.env with override=True, and the image's docker/stage2-hook.sh writes a generated API_SERVER_KEY there on any boot whose env has none. Chief's volume had one from earlier boots, so the API server checked that key and every 1Password-key client got 401. The config init container now gets the optional api-server envFrom (so its stage2 run skips generation instead of writing a new key each boot) and deletes any ^API_SERVER_KEY= line from /opt/data/.env when that key is present. docs/hermes.md explains why, and corrects the claim that a bot with the API off has no listener: Hermes keeps a loopback-only one. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#5685) Serves the gpt-image-2 deployment from Azure resource ryholt-foundry-eastus2-resource on the v1 API. It uses its own key, AZURE_FOUNDRY_EASTUS2_API_KEY, read from 1Password field litellm/azure_foundry_eastus2_api_key. model_info.mode is image_generation; the v1.106.6 cost map prices azure/gpt-image-2. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… pytest workers (#5918) On the 8-CPU pod, va-acr-app's tests need 3,200-3,600 CPU-seconds. At 12 workers the pod was throttled in about half of its periods, so the tests are CPU-bound. Limits go to 16 CPU / 24Gi and PYTEST_XDIST_AUTO_NUM_WORKERS to 24 (1.5 per CPU). Memory peaked at 8.6 GiB with 12 workers and grows about 0.3 GiB per worker. Requests go to 8 CPU / 8Gi so concurrent test pods spread across the nodes; each node has about 47 CPUs unrequested. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Prometheus only selected ServiceMonitors, PodMonitors and rules labeled release=kube-prometheus-stack, the chart default. Rook's rook-ceph-mgr and rook-ceph-exporter monitors carry team=rook, so the TSDB had no ceph_* series. 18 ServiceMonitors and 2 PodMonitors were skipped in all, among them Cilium, cert-manager, external-secrets, flux-operator, volsync, tuppr, the DNS and tunnel exporters, CNPG and reloader. - kube-prometheus-stack: set the five *SelectorNilUsesHelmValues to false so Prometheus selects every monitor, rule, probe and ScrapeConfig in every namespace. - Rook's operator-owned monitors relabel cluster=rook-ceph, which wins over the cluster=k8s-holthome external label. An alert relabel rule puts Ceph alerts back on k8s-holthome before they reach Forge. - rook-ceph-cluster: createPrometheusRules (chart default false), which renders prometheus-ceph-rules (16 groups, 89 rules). The Kustomization now waits on prometheus-operator-crds. - New monitors for the Flux controllers, the Envoy Gateway controller and the Envoy proxies. They sit in kube-prometheus-stack so that flux-instance and envoy-gateway do not wait on the Prometheus CRDs. - docs/architecture.md records the selection policy. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
) GrafanaOnCallServiceDown matched up{job="grafana-oncall-engine"}, but the ServiceMonitor sets job from app.kubernetes.io/name, so targets carry job="grafana-oncall". absent() was always true and Forge paged Pushover critical. Match the Service name instead, which forceRename pins. The Finance Signal status read drops out in 2-32 minute bursts, and the listener, report status, attention and weekly fulfilled gauges move with it. ReportMissed had no for, and the others had 1-2m, so each burst fired and resolved. Use for: 5m and keep_firing_for: 30m on ListenerUnhealthy, ReportUnavailable, ReportAttention, ReportMissed and SnapshotUnavailable. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…t fail auth (#5922) At the default 60 s TTL every active virtual key went back to postgres17 each minute. When the database stalled, key auth failed, the proxy answered 503, and Hermes read that as "provider overloaded" and fell back to claude-sonnet. Set general_settings.user_api_key_cache_ttl to 3600 and document the tradeoff and the readiness-probe gap it leaves. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The volsync chart's ServiceMonitor scrapes /metrics over HTTPS without a bearer token. The controller's authn/authz filter answers 401, so both volsync targets have been down since Prometheus began selecting every ServiceMonitor (#5919). metrics.disableAuth: true adds --metrics-require-rbac=false to the controller and drops the chart's metrics-auth ClusterRoles and binding. Metrics stay on HTTPS :8443, which the existing ServiceMonitor uses. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
sled-1's ens1f0 has dropped about 4.9% of received frames as CRC/alignment errors since 12 Nov and lost link twice in a day, which stalled Ceph RBD writes across the cluster for 4-44 s. The chart's NodeNetworkReceiveErrs never fired: it needs a 2m ratio above 1% for an hour, and the 2m ratio dipped often enough to restart the timer. NodeNicErrors takes the RX+TX error ratio over 1h on physical ports (> 0.1% and > 1000 errors, for 15m, keep_firing_for 1h). NodeNicLinkFlapped fires on any carrier change in 30m, skipping nodes up for less than 45m so a reboot's counter reset does not count. Backtested over 9 days: NodeNicErrors matches only sled-1 ens1f0 (lowest 1h ratio 0.58%), and NodeNicLinkFlapped matches the 11-18 02:53 sled-1 outage and an 11-17 08:03 drop on all four nodes. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…mpty the Service (#5924) /health/readiness answers 503 while the database is unreachable. With one replica that removed LiteLLM's only endpoint at 11-15 01:05 and 11-17 03:49 UTC, the starts of two Hermes fallback bursts, although cached keys could still be served. /health/liveliness still answers 503 during graceful shutdown, so rollouts drain as before. Docs and ADR 0006 updated. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…5925) hindsight-api was OOMKilled (exit 137) seven times, three of them between 11:00 and 11:10Z on 2026-11-18. During a Ceph alert storm, Hank woke once per CephPGsUnclean group and recalled each time. Four recalls started within 2s, each ran the in-process cross-encoder, and the container passed its 4Gi limit within seconds of a ~1.3GiB baseline. Raise the limit to 8Gi (request stays 2Gi). Limit the local reranker to two concurrent passes (HINDSIGHT_API_RERANKER_LOCAL_MAX_CONCURRENT, default 4) over at most 150 candidates (HINDSIGHT_API_RERANKER_MAX_CANDIDATES, default 300). Both are read by hindsight-api 1.2.0 config.py. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…domains (#5926) * docs(mail): ryanholt.net and stefanieholt.com as alias domains Accounts stay in holthome.net because OIDC sign-ins append usernameDomain; the other domains are aliases. Records the edge rules for both domains and the per-domain cutover from Microsoft 365. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(mail): devialea.com from Cloudflare Email Routing Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(mail): agent mailboxes and the agent-auth-results script Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(mail): chief@holthome.net agent mailbox Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
EMAIL_ENABLED gives a bot Hermes' bundled email platform on its own Stalwart mailbox: an ExternalSecret with the app password and fixed settings (mx host, IMAP 993, SMTP 465, authserv-id pin), egress to the Stalwart pods on those ports, and a matching ingress rule on Stalwart. Chief gets chief@holthome.net, answering only Ryan's addresses. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Without a home channel Hermes mails a "No home channel is set for Email" notice when an email conversation starts. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Headlamp ran every request as its ServiceAccount, which was bound to view, so Secrets and other panes were empty and nothing could be edited. Headlamp now has its own Pocket ID client and passes the user's ID token to the API server. The control planes trust id.holthome.net for audience headlamp (talos/control-plane/20-authentication.yaml), mapping usernames and groups to the oidc: prefix. Group oidc:admins is bound to cluster-admin. The ServiceAccount no longer has a ClusterRoleBinding. The route opts out of the gateway's Pocket ID login (gateway.holthome.net/auth: native, ReferenceGrant headlamp-gateway-auth), so a user logs in once. The client secret comes from 1Password item headlamp through ExternalSecret headlamp-oidc. The Talos config is already applied to sled-0, sled-1 and sled-2. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Add withings_get_measurements and withings_trend to the /mcp/agent/grokbot view's tool list. Both are read-only; Withings has no write tools. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…on (#5931) Add every mfp_* tool (the same 33 Grokbot has) to /mcp/steffi. The view is credentials per_user, so each call uses mfp.steffi from homelab-mcp-user-credentials, never Ryan's shared jar; a missing or expired file is an error, not a fall-back. Refresh the comments that still said no view uses per_user and that mfp has only one cookie jar. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…solidation (#5932) Meetings, People, Projects, Research, Reviews, Voice, HHS and TASKS.md moved under Work/ on 2026-11-19, so their roots are dropped. The roots left are tooling that stays at the vault root (Claude's memory, Templates, the Excalidraw and ExcaliBrain plugin folders). No client's access changes. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…5933) Every Obsidian community plugin was removed on 2026-11-20 and the Templates, Excalidraw and Excalibrain folders were archived, so the work domain is now Work, memory and CLAUDE.md. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…laude, not Ryan (#5934) Skill incident-response 2.1.0: a new alert group joins an open note about the same component from the last six hours (group_keys), at most one board task per incident, and Needs Ryan only for a decision or hands-on work that is Ryan's. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… research (#5935) Personal/Research, personal sensitivity, owned by the Grok Chief of Staff; grokbot reads and writes it. Plus a Research project under Personal on the OpenProject board. Guardrails live in the vault's Personal/Research/Plan.md. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Vault tools only: read and write Work/, read Personal/Research. New vault client copilot (internal, this client only) and a Claude-only assistant domain for memory/ and CLAUDE.md, split out of work. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(replog): deploy the workout log on the internal gateway SQLite on a Ceph RBD volume, one replica, VolSync snapshot backup. The image tag is a placeholder until the first v* release exists. Co-authored-by: Ryan Holt <ryan@ryanholt.net> * feat(replog): leave MCP and OIDC commented until a later enable The process serves /api/mcp only when the Pocket ID env is set. The internal route is already native, with a commented /api/mcp rule to uncomment alongside that env. Co-authored-by: Ryan Holt <ryan@ryanholt.net> * feat(replog): match Forge's public site and leave MCP to homelab-mcp The hostname stays replog.holthome.net on both gateways, Pocket ID login stays, and the Deployment starts at zero replicas. The website route does not single out /api/mcp. A fresh database gets a new secret key. The Pocket ID client secret is copied from Forge. Co-authored-by: Ryan Holt <ryan@ryanholt.net> * feat(replog): deny /api/mcp on the public route and add replog-k8s The website rule is a prefix of /, which would forward /api/mcp. A longer prefix returns 404 from Envoy and does not reach the pod. The runner scale set follows va-acr-app-k8s and stays at minRunners 0. The app image remains the v0.0.0 placeholder at 0 replicas. Co-authored-by: Ryan Holt <ryan@ryanholt.net> * revert(replog): drop the replog-k8s runner scale set carpenike/replog is public. GitHub-hosted runners are free for it, and a privileged Docker runner will not be registered. The /api/mcp deny stays. Co-authored-by: Ryan Holt <ryan@ryanholt.net> * feat(replog): pin the published image and run one replica ghcr.io/carpenike/replog is published. Pin sha-f448ffd and its digest, and set replicas to 1. Merging still does not move the name off Forge. Co-authored-by: Ryan Holt <ryan@ryanholt.net> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Main published sha-9e33a7f after the earlier pin. replicas stays 1. Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(homelab-mcp): add Ryan's /mcp/replog view Pin homelab-mcp to the image published for 450891b, add the view from mcp#169, reference the coach token from 1Password, and allow port 8080 between homelab-mcp and replog. * feat(replog): map the coach service token from 1Password ExternalSecret replog writes REPLOG_SERVICE_TOKEN from item replog, field service_token. The bootstrap admin is a coach, so REPLOG_SERVICE_USER is not set. No token value is committed. --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Pin ghcr.io/carpenike/replog to the image published for replog 252ae15d9e3297a2eef7e881ad13f6d0f090c59a (#54). Digest confirmed from GHCR. Replicas, ExternalSecret, network policy, ndots, and homelab-mcp are unchanged. Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The /mcp/replog view filters to scope:atrium-personal-replog, but the replog module was never in HOMELAB_MCP_TOOL_CATEGORIES, so it never registered and tools/list returned nothing.
Pin ghcr.io/carpenike/replog to the image published for replog beb93c1c224cc7a51aad0f9ceed5d4491e636c8f (#55). Digest confirmed from GHCR. Replicas, ExternalSecret, network policy, ndots, and homelab-mcp are unchanged. Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(replog): per-user MCP credential and image pin Set /mcp/replog to credentials per_user so the bearer is replog.ryan from item homelab-mcp-user-credentials. Leave homelab-mcp/replog_token mapped to HOMELAB_MCP_REPLOG_TOKEN. Pin replog to main 147ec5a by the tag's Docker-Content-Digest. * homelab-mcp: pin image to carpenike/mcp#170 merge (f3bb75e8) --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft only. Do not merge.
ARC scale sets
sam-tracker(runs-on: sam-tracker-k8s) andsam-tracker-app-k8s(runs-on: sam-tracker-app-k8s, repocarpenike/sam-tracker-app) share one Kustomize component:containerMode.type: dind, runner memory limit4Gi,minRunners0,maxRunners3. Both use ExternalSecretactions-runner-controller. Installation175991169already includes both repositories. Namespaceactions-runner-systemispod-security.kubernetes.io/enforce: privileged(audit and warn too). A local render of chart 0.15.1 with kube version 1.37 puts privilegeddocker:dind(restartPolicy: Always) on the runner pod. Neither repository's workflows were changed. This environment still cannot open10.20.10.10:6443and has no kubeconfig, soflux reconcileand the live AutoscalingRunnerSets were not observed from here.LAN DNS for echo
echo.holthome.netis an HTTPRoute onenvoy-external, notenvoy-internal.10.45.10.51is that gateway's LoadBalancer address.envoy-internalis10.45.10.50.cloudflare-dnswatchesenvoy-externalonly and publishes a proxied record whose target isexternal.holthome.net.1.1.1.1returns the same pair the Mac got,104.21.60.153and172.67.197.253, forecho.holthome.net,external.holthome.net, andflux-webhook.holthome.net.not-a-real-app.holthome.net,headlamp.holthome.net, andinternal.holthome.netare NODATA (SOA only). There is no Cloudflare wildcard answering the zone.mikrotik-dnswatchesenvoy-internalonly, so it does not create a static record for echo. A resolver with no static override, including the router forwarding upstream, returns the Cloudflare addresses. This environment cannot query10.20.0.1:53or see the Mac's resolver list. The Mac's answer matches public DNS, so nothing in that lookup path had a different A record for echo.envoy-internalpreviously targeted the hostnameinternal.holthome.net. external-dns turns a hostname target into a CNAME, andmikrotik-dnsmanages A and TXT only, so internal routes were dropped and no static A was written. That target is now10.45.10.50. Echo stays on Cloudflare. Reaching the gateway directly remainscurl --resolve echo.holthome.net:443:10.45.10.51, which returns the Pocket ID 302.Auth Kustomization
The object is named
envoy-gateway-authin namespacenetwork.kubernetes/apps/networksets that namespace, so it does not appear underflux-system. Its health check is ExternalSecretpocket-id-gateway. This environment still cannot open10.20.10.10:6443, so Ready was not read from the API. The synced 1Password item and the 302 toid.holthome.netare the policy that Kustomization installs.Headlamp
Official Headlamp chart
0.46.0fromhttps://kubernetes-sigs.github.io/headlamp/, namespaceheadlamp. URLhttps://headlamp.holthome.net. The HTTPRoute parent is onlyenvoy-internal, with nogateway.holthome.net/authlabel, so Pocket ID on the gateway is the login andmikrotik-dnsis the publisher.cloudflare-dnsselectsenvoy-externalonly.The chart ServiceAccount
headlampis bound to the built-inviewClusterRole (the binding is namedheadlamp-adminbecause that is the chart's name).config.unsafeUseServiceAccountTokenmakes the UI use that token, which is the chart's in-cluster setting for a deployment behind an auth proxy. Headlamp OIDC is off. ClusterRoleheadlamp-flux-viewadds Flux read and is aggregated intoview. The Flux plugin isheadlamp_flux0.7.0 via the chartpluginsManager. Details and the later write grant are indocs/headlamp.md.Actions runners
GitHub Actions Runner Controller charts
gha-runner-scale-set-controllerandgha-runner-scale-set, both0.15.1, in namespaceactions-runner-system. Two repo-scoped scale sets share one component and ExternalSecretactions-runner-controller. HelmReleasesam-trackerishttps://github.com/carpenike/sam-trackerwithruns-onsam-tracker-k8s. HelmReleasesam-tracker-app-k8sishttps://github.com/carpenike/sam-tracker-appwithruns-onsam-tracker-app-k8s. Both are ephemeral,minRunners0,maxRunners3,containerMode.type: dind, runner memory limit4Gi.The scale set Kustomization depends on the controller and on external-secrets. Its health checks are the ExternalSecret and both HelmReleases. Nothing depends on it. The 1Password item was not rewired.
DNS: curl and am.holthome.net
Pods inherit
search holthome.netfrom TalosResolverConfigand kubelet addsndots:5.am.holthome.nethas two dots, so the resolver asksam.holthome.net.holthome.netbefore the real name.curlimages/curlis Alpine curl built with c-ares. c-ares aborts that search on SERVFAIL or REFUSED and returnsCould not resolve host. busyboxnslookup, glibc, and Go continue, so nslookup still printsforge.holthome.net/10.20.0.30and Prometheus still reaches the name. A NODATA answer for the doubled name does not trigger this. Public resolvers return NODATA for it. AAAA NXDOMAIN and a glued CNAME plus A also do not. A CNAME with no address fails Go as well as curl, which does not match Prometheus reaching the host.CoreDNS now answers
holthome.net.holthome.netwith NXDOMAIN, before the query is forwarded to10.20.0.1. c-ares treats that as a miss and resolvesam.holthome.net. Short names such asforgestill search toforge.holthome.net.This environment cannot open UDP/TCP 53 on
10.20.0.1, so the live rcode was not captured. The router was not changed. If/ip dnshas no upstream, RouterOS SERVFAILs names that are not static records. Setting upstream turns that into the NODATA public resolvers already return:Do not forward the zone
holthome.netat CoreDNS. Theloopplugin turns that loop into SERVFAIL.Prometheus control-plane scrapes
kubeProxy.enabled: falsedrops the kube-proxy ServiceMonitor and theKubeProxyDownrules. Cilium runs withkubeProxyReplacement: true, and TalosKubeProxyConfigis already disabled.kubeEtcd.enabled: falsedrops the etcd ServiceMonitor and etcd rules. Talos 1.14 serves etcd HTTP metrics on port 2383 with client mTLS, etcd is not a pod the chart can select, and this repo has no etcd client cert for Prometheus. The chart default is HTTP port 2381.kube-controller-manager and kube-scheduler stay enabled. Their Talos static pods are hostNetwork and labeled
component=kube-controller-managerandcomponent=kube-scheduler.talos/control-plane/00-cluster.yamlalready setsbind-address: 0.0.0.0, so they listen on 10257 and 10259. The chart scrapes those HTTPS ports with the Prometheus token.Checks
On
992175894cc9362728939b3a4291c9b68bed4e16:scripts/validate.sh— passkustomize buildof the scale-set app — HelmReleasessam-trackerandsam-tracker-app-k8sdiffer only by name,githubConfigUrl, andrunnerScaleSetName; both are dind, memory limit4Gi, min 0, max 3, secretactions-runner-controllerCluster API
10.20.10.10:6443— timeout. No kubeconfig on this machine, soflux reconciledid not runHuman review (agents leave this box unchecked)