Skip to content

Start the public Talos rebuild skeleton - #5639

Draft
carpenike wants to merge 295 commits into
mainfrom
rebuild
Draft

carpenike wants to merge 295 commits into
mainfrom
rebuild

Conversation

@carpenike

@carpenike carpenike commented Oct 27, 2026 •

Copy link
Copy Markdown
Owner

Draft only. Do not merge.

ARC scale sets

sam-tracker (runs-on: sam-tracker-k8s) and sam-tracker-app-k8s (runs-on: sam-tracker-app-k8s, repo carpenike/sam-tracker-app) share one Kustomize component: containerMode.type: dind, runner memory limit 4Gi, minRunners 0, maxRunners 3. Both use ExternalSecret actions-runner-controller. Installation 175991169 already includes both repositories. Namespace actions-runner-system is pod-security.kubernetes.io/enforce: privileged (audit and warn too). A local render of chart 0.15.1 with kube version 1.37 puts privileged docker:dind (restartPolicy: Always) on the runner pod. Neither repository's workflows were changed. This environment still cannot open 10.20.10.10:6443 and has no kubeconfig, so flux reconcile and the live AutoscalingRunnerSets were not observed from here.

LAN DNS for echo

echo.holthome.net is an HTTPRoute on envoy-external, not envoy-internal. 10.45.10.51 is that gateway's LoadBalancer address. envoy-internal is 10.45.10.50.

cloudflare-dns watches envoy-external only and publishes a proxied record whose target is external.holthome.net. 1.1.1.1 returns the same pair the Mac got, 104.21.60.153 and 172.67.197.253, for echo.holthome.net, external.holthome.net, and flux-webhook.holthome.net. not-a-real-app.holthome.net, headlamp.holthome.net, and internal.holthome.net are NODATA (SOA only). There is no Cloudflare wildcard answering the zone.

mikrotik-dns watches envoy-internal only, so it does not create a static record for echo. A resolver with no static override, including the router forwarding upstream, returns the Cloudflare addresses. This environment cannot query 10.20.0.1:53 or see the Mac's resolver list. The Mac's answer matches public DNS, so nothing in that lookup path had a different A record for echo.

envoy-internal previously targeted the hostname internal.holthome.net. external-dns turns a hostname target into a CNAME, and mikrotik-dns manages A and TXT only, so internal routes were dropped and no static A was written. That target is now 10.45.10.50. Echo stays on Cloudflare. Reaching the gateway directly remains curl --resolve echo.holthome.net:443:10.45.10.51, which returns the Pocket ID 302.

Auth Kustomization

The object is named envoy-gateway-auth in namespace network. kubernetes/apps/network sets that namespace, so it does not appear under flux-system. Its health check is ExternalSecret pocket-id-gateway. This environment still cannot open 10.20.10.10:6443, so Ready was not read from the API. The synced 1Password item and the 302 to id.holthome.net are the policy that Kustomization installs.

Headlamp

Official Headlamp chart 0.46.0 from https://kubernetes-sigs.github.io/headlamp/, namespace headlamp. URL https://headlamp.holthome.net. The HTTPRoute parent is only envoy-internal, with no gateway.holthome.net/auth label, so Pocket ID on the gateway is the login and mikrotik-dns is the publisher. cloudflare-dns selects envoy-external only.

The chart ServiceAccount headlamp is bound to the built-in view ClusterRole (the binding is named headlamp-admin because that is the chart's name). config.unsafeUseServiceAccountToken makes the UI use that token, which is the chart's in-cluster setting for a deployment behind an auth proxy. Headlamp OIDC is off. ClusterRole headlamp-flux-view adds Flux read and is aggregated into view. The Flux plugin is headlamp_flux 0.7.0 via the chart pluginsManager. Details and the later write grant are in docs/headlamp.md.

Actions runners

GitHub Actions Runner Controller charts gha-runner-scale-set-controller and gha-runner-scale-set, both 0.15.1, in namespace actions-runner-system. Two repo-scoped scale sets share one component and ExternalSecret actions-runner-controller. HelmRelease sam-tracker is https://github.com/carpenike/sam-tracker with runs-on sam-tracker-k8s. HelmRelease sam-tracker-app-k8s is https://github.com/carpenike/sam-tracker-app with runs-on sam-tracker-app-k8s. Both are ephemeral, minRunners 0, maxRunners 3, containerMode.type: dind, runner memory limit 4Gi.

The scale set Kustomization depends on the controller and on external-secrets. Its health checks are the ExternalSecret and both HelmReleases. Nothing depends on it. The 1Password item was not rewired.

DNS: curl and am.holthome.net

Pods inherit search holthome.net from Talos ResolverConfig and kubelet adds ndots:5. am.holthome.net has two dots, so the resolver asks am.holthome.net.holthome.net before the real name.

curlimages/curl is Alpine curl built with c-ares. c-ares aborts that search on SERVFAIL or REFUSED and returns Could not resolve host. busybox nslookup, glibc, and Go continue, so nslookup still prints forge.holthome.net / 10.20.0.30 and Prometheus still reaches the name. A NODATA answer for the doubled name does not trigger this. Public resolvers return NODATA for it. AAAA NXDOMAIN and a glued CNAME plus A also do not. A CNAME with no address fails Go as well as curl, which does not match Prometheus reaching the host.

CoreDNS now answers holthome.net.holthome.net with NXDOMAIN, before the query is forwarded to 10.20.0.1. c-ares treats that as a miss and resolves am.holthome.net. Short names such as forge still search to forge.holthome.net.

This environment cannot open UDP/TCP 53 on 10.20.0.1, so the live rcode was not captured. The router was not changed. If /ip dns has no upstream, RouterOS SERVFAILs names that are not static records. Setting upstream turns that into the NODATA public resolvers already return:

/ip dns set servers=1.1.1.1,9.9.9.9

Do not forward the zone holthome.net at CoreDNS. The loop plugin turns that loop into SERVFAIL.

Prometheus control-plane scrapes

kubeProxy.enabled: false drops the kube-proxy ServiceMonitor and the KubeProxyDown rules. Cilium runs with kubeProxyReplacement: true, and Talos KubeProxyConfig is already disabled.

kubeEtcd.enabled: false drops the etcd ServiceMonitor and etcd rules. Talos 1.14 serves etcd HTTP metrics on port 2383 with client mTLS, etcd is not a pod the chart can select, and this repo has no etcd client cert for Prometheus. The chart default is HTTP port 2381.

kube-controller-manager and kube-scheduler stay enabled. Their Talos static pods are hostNetwork and labeled component=kube-controller-manager and component=kube-scheduler. talos/control-plane/00-cluster.yaml already sets bind-address: 0.0.0.0, so they listen on 10257 and 10259. The chart scrapes those HTTPS ports with the Prometheus token.

Checks

On 992175894cc9362728939b3a4291c9b68bed4e16:

  • scripts/validate.sh — pass

  • kustomize build of the scale-set app — HelmReleases sam-tracker and sam-tracker-app-k8s differ only by name, githubConfigUrl, and runnerScaleSetName; both are dind, memory limit 4Gi, min 0, max 3, secret actions-runner-controller

  • Cluster API 10.20.10.10:6443 — timeout. No kubeconfig on this machine, so flux reconcile did not run

  • Human review (agents leave this box unchecked)

Open in Web Open in Cursor 

carpenike and others added 30 commits October 31, 2026 13:03
Generation-1 aes keys stay active so existing volume mounts keep
authenticating. Cipher restriction is a later change.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Deletes the aes keys kept for existing mounts. Every ceph-block mount
has been rescheduled onto the aes256k key (sleds drained one at a time,
no client.csi-*.1 sessions on any mon).

Re-landing #5656 against rebuild: it merged into its stacked base branch
instead of rebuild.
Stops the mons authenticating or creating aes keys. All 25 keys in ceph auth ls
are aes256k and no client.csi-*.1 key or session remains.

Re-landing #5658 against rebuild: its base was the stacked #5656 branch.
….npmjs.org (#5661)

The node:alpine plugin manager (musl) stops on the router's NOERROR/no-data
answer for registry.npmjs.org.holthome.net, so npm fails with ENOTFOUND and
the pod crash-loops. A post-render patch adds dnsConfig ndots:1.
…ions (#5662)

Add gpt-6.1-sol, gpt-6.1-terra, gpt-6-astra and gpt-6-luna from Azure
resource ryholt-foundry (project proj-default, separate subscription) on
the v1 API with a new AZURE_FOUNDRY_API_KEY. The pinned image predates
GPT-6, so the azure/gpt5_series/ route supplies reasoning-model param
mapping; cost tracking reads $0 until the image is bumped.

Move every ryholt-simplechat-aifoundry deployment to 2025-04-01-preview.
gpt-5-codex and gpt-5-pro are bridged to the Responses API, which
2024-12-01-preview does not serve. Shared params are YAML anchors.

Enable the prometheus callback so the ServiceMonitor's /metrics scrape
returns data.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… resource (#5663)

Every deployment on ryholt-simplechat-aifoundry (gpt-4o, gpt-5, gpt-5-chat,
gpt-5-codex, gpt-5-pro, gpt-5.1, gpt-5.1-codex, o3-mini,
text-embedding-3-small) returns 401 from Azure. That environment is
retired; the GPT-6 deployments on ryholt-foundry replace them and all four
answer through the proxy. Remove the entries and AZURE_API_KEY.

No Azure embedding model remains.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Bump ghcr.io/berriai/litellm v1.100.1 -> v1.106.6, pinned by digest.
Renovate never proposed this because it does not scan the rebuild branch.
The new cost map prices gpt-6.1-sol, gpt-6-astra and gpt-6-luna.

Add decision-1, the Decision-1 deployment of Microsoft-Decision-1 on
ryholt-foundry, via azure_ai/ on the services.ai.azure.com host. It is
served on /v1/decisions and /v1/systemone (1.105+). base_model points
cost tracking at azure_ai/Microsoft-Decision-1.

Prisma migrates the schema forward on startup. CNPG Backup
postgres17-pre-litellm-1-106-6 was taken first for rollback.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Runner pods run workflow code and could reach the node LAN, the router,
the LoadBalancer pool, other pods and the Kubernetes API. The new
CiliumNetworkPolicy selects runner pods by ARC's
actions-ephemeral-runner label and allows only kube-dns and public IPv4.
Listeners and the controller are not selected.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Repo-scoped ARC scale sets for atrium, mcp, network-config, schoolhouse
and whiskey-whiskey-whiskey (runs-on: <name>-k8s), moving their CI off
the Forge runners. These jobs use no Docker, so the overlays drop dind
and the runner pods are not privileged. atrium's and mcp's Nix jobs get
their own scale sets later, once their privilege needs are known.

The GitHub App installation must include these repos first.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Kata VM sandboxes do not see the host cgroup socket-LB hook. With
kube-proxy replacement already on, hostNamespaceOnly leaves host
socket LB in place and sends pod service traffic through the tc
load balancer.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The Image Factory schematic for Talos v1.14.5 now includes
siderolabs/kata-containers beside intel-ucode and nut-client. Flux
reconciles the kata and kata-qemu RuntimeClasses that extension
registers in containerd.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The chart default leaves ceph-filesystem with no MDS placement, so both
pods can share a node. Require hostname anti-affinity on app=rook-ceph-mds.
Control planes are schedulable, so no extra toleration is added.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
…et site (#5670)

* feat(matrix): add Tuwunel homeserver and Hermes bot provisioning

Tuwunel v1.9.4 in namespace matrix, server name ryanholt.net, client and
federation host matrix.ryanholt.net on envoy-external, apex
/.well-known/matrix delegation. RocksDB on a ceph-block PVC, one replica,
Recreate, Pod Security restricted, default-deny Cilium policy.

Registration is token-only from a SOPS Secret; no first-user admin.
Appservice hermes-bots owns @hermes-.*:ryanholt.net exclusively, rate
limiting off. VolSync restic backup waits on a new 1Password item.

A separate provisioner Kustomization ships the script and a per-bot
template (Job, get/patch-only RBAC on one Secret). No bot is wired.

ADR 0010 and docs/matrix.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* chore(matrix): drop stray __pycache__ from provisioner

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(web): serve ryanholt.net apex site; back up Tuwunel to R2

Add namespace web with a static ryanholt.net site (nginx-unprivileged,
ConfigMap content, restricted, default-deny Cilium, no egress) and a
www -> apex redirect. /.well-known/matrix/ keeps its longer-prefix route
to Tuwunel, so the apex is shared rather than given to Matrix.

Point the Tuwunel VolSync source at v1's existing 1Password item
volsync-r2-template instead of a new item, using v1's per-app path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(matrix): apex has no stale record; no manual DNS step

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Merge only after @ryan:ryanholt.net has registered. The pod restarts on
the ConfigMap change and runs the command at startup. Revert in the next
pull request.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@ryan:ryanholt.net is admin. admin_execute runs on every start, so the
line comes out now. Later admins are promoted from the admin room with
!admin users make-user-admin.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Set encryption_enabled_by_default_for_room_type = "all". Tested against
v1.9.4: rooms created with the public_chat and private_chat presets both
get m.room.encryption (m.megolm.v1.aes-sha2) without the client asking.
Document which layers are encrypted and what bots need.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The previous token was shown in a terminal during first-admin setup.
New 32-byte value, encrypted to the same age and PGP recipients.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…LM key provisioning (#5675)

* feat(hermes): deploy Chief, a Hermes agent on Matrix as @chief

Namespace hermes (restricted, default-deny Cilium). Chief runs Hermes
Agent v0.22.6 in a kata-qemu VM as uid 10000 with no capabilities and a
read-only root: shareProcessNamespace keeps the entrypoint off PID 1, so
the image bootstraps without s6 and accepts its own hermes UID. Model
claude-sonnet via in-cluster LiteLLM (custom provider, key from 1Password
item hermes-chief). Matrix E2EE required, allowlist @ryan. Data volume
backed up to R2.

Bot Matrix IDs drop the hermes- prefix. The hermes-bots appservice
namespace becomes an explicit name list (^@(chief):ryanholt\.net$) so the
as_token cannot act as other users; the registration was re-created with
fresh as_token/hs_token (nothing used the old ones).

Tested locally under podman with the pod's flags: bootstrap and config set
on an empty volume, gateway joins Tuwunel v1.9.4 with cross-signing and
auto-joins an encrypted DM; the appservice registers @chief and gets
M_EXCLUSIVE for @ryan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(litellm): provision virtual keys in-cluster; use it for Chief

Add kubernetes/apps/litellm/key-provisioner: a shared script and a
per-key template. The Job runs in namespace litellm with the master key
from litellm-env, finds or creates the team, reconciles an existing key's
models and budget, or deletes and regenerates the key when the consumer's
Secret has lost it, then patches LITELLM_API_KEY into that one Secret
(get/patch RBAC on that name only; egress DNS, LiteLLM, kube API).

Chief gets key hermes-chief in team hermes, claude-sonnet only, 25 USD per
30 days, through Kustomization chief-litellm. The 1Password item for the
key and its ExternalSecret are gone.

BUDGET is one "<USD>/<duration>" string because Flux substitutes after
kustomize build drops quotes; a bare number would reach the API server as
an int. Found by substituting the built template and running kubeconform.

Tested against LiteLLM v1.106.6 with Postgres: first run, re-run (key
unchanged), budget change applied in place, lost Secret (old key 401, one
key left under the alias). Ignore __pycache__ from local test runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Both provisioning Jobs (matrix/hermes-provision-chief and
litellm/litellm-key-hermes-chief) failed on their first call to an
in-cluster Service: "Name has no usable address". Pods search
holthome.net last with ndots:5, so tuwunel.matrix.svc.cluster.local is
first tried with the cluster suffixes (NXDOMAIN) and then as
....holthome.net, which Cloudflare and the router answer NODATA. musl
(Alpine) treats that NODATA as final; glibc keeps searching and tries the
name as given. Reproduced with podman: Alpine cannot resolve even
matrix.ryanholt.net with that search list, Debian can. Same cause as the
Headlamp ndots note and the CoreDNS holthome.net.holthome.net block.

- Image python:3.15.2-alpine3.24 -> python:3.15.2-slim-trixie (glibc).
- restartPolicy Never, so failed attempts keep their pods and logs; with
  OnFailure the Job deleted the pod when it gave up and the logs went too.
- request() returns status 0 with "host: reason" for DNS, refused, and
  timeout errors instead of an uncaught traceback.

The pod spec changes, so the bots' Kustomizations (force: true) replace
the failed Jobs.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Chief's cross-signing bootstrap failed with M_UNRECOGNIZED: Not Found.
Envoy Gateway 1.9.3 defaults escapedSlashesAction to UnescapeAndRedirect,
so PUT .../account_data/m.secret_storage.key.ab%2Fcd got a 307 to
.../ab/cd, a path Tuwunel does not route. Confirmed against envoy-external
directly (10.45.10.51), not only through Cloudflare; %2B, %3D, and +
already passed. Matrix clients percent-encode "/" in path parameters.

ClientTrafficPolicy envoy-external-https sets KeepUnchanged on the
external HTTPS listener only. gateway-auth.md notes what that means for
any future path-based rule there.

Chief: set database.journal_mode delete in the init step. /opt/data is
virtiofs inside the Kata VM; Hermes already fell back to DELETE and warned
for every database. The HelmRelease change also restarts Chief, which
retries the cross-signing bootstrap (no recovery key file was written).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ExternalSecret chief-matrix-recovery maps 1Password hermes-chief/
matrix_recovery_key to MATRIX_RECOVERY_KEY, wired into Chief as an
optional envFrom. With it, Hermes loads the cross-signing keys from
secret storage on every start and signs the current device, so a rebuilt
or restored crypto store comes back verified. Until the item exists the
pod starts without it; Reloader restarts the pod when the Secret appears.

envFrom rather than env valueFrom: app-template 5.3.0's values schema
rejects optional on an env secretKeyRef. kubeconform does not check
chart values; helm template against the pulled chart does.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#5680)

kubeconform validates the HelmRelease objects but not spec.values, so a
value the chart's values.schema.json rejects (for example optional: true
on an app-template 5.3.0 secretKeyRef) only failed at helm upgrade.

scripts/helm-template.sh builds each Flux Kustomization, applies postBuild
substitution from cluster-secrets with flux envsubst --strict, pulls the
chart from its OCIRepository or HelmRepository (cached by url and ref),
and runs helm template on the values. validate.sh and a new CI job run it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Kubelet copies the node search list into every pod, after the cluster
suffixes, with ndots:5. holthome.net is DNSSEC-signed on Cloudflare, which
answers a name that does not exist with NOERROR and no records ("black
lies"), and the router passes that on. musl (Alpine) stops a search at
that answer, so Alpine pods could not resolve x.ns.svc.cluster.local or
any public name; glibc keeps going. CoreDNS autopath does not help: it
only moves on after NXDOMAIN.

Reproduced with podman against the router: with holthome.net in the
search list Alpine fails matrix.ryanholt.net; without it, it resolves.
ndots:1 also works but makes musl skip the search list for any dotted
relative name (tuwunel.matrix, kubernetes.default.svc).

domains: [] also clears a DHCP-provided list. Internal names are written
in full (forge.holthome.net). The CoreDNS holthome.net.holthome.net block
and the Headlamp ndots:1 patch come out in a follow-up, after this is
applied and pods have been recreated with the new search list.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…hief to hermes-chief (#5681)

* feat(hermes): screen pod image with sshd as pn

images/hermes-screen vendors NousResearch/hermes-agent v0.22.6
docker/sandbox-desktop.Dockerfile (Xvnc, Xfce, headed Playwright
Chromium, cua-driver, user pn). nousresearch/hermes-sandbox is not
published for v0.22.x. Local changes: base pinned by digest, Playwright
pinned to 1.64.1, pn's NOPASSWD sudo removed, openssh-server run as pn on
port 22 (key-only, no root, no PAM, no AcceptEnv, no forwarding).

Tested locally on the pinned base: sshd listens on 22 as uid 1000 with
all caps dropped and no-new-privs (ip_unprivileged_port_start=22); the
agent key logs in; a wrong key, root, and password auth are refused; scp
(SFTP) and tar-over-ssh work; SendEnv values do not arrive; -L forwarding
is refused.

hermes-screen-image.yaml builds it to ghcr.io/carpenike/hermes-screen
with the built-in GITHUB_TOKEN. AGENTS.md gets the one carve-out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(hermes): per-bot component with agent and screen pods; move Chief to hermes-chief

kubernetes/components/hermes-bot runs each Hermes bot in its own
hermes-<bot> namespace as two pods:

- agent (runc): hermes gateway run, the only holder of the Matrix token
  and LiteLLM key (as OPENAI_API_KEY, which Hermes scrubs from child
  processes). terminal.backend is ssh to the screen; no local fallback.
  Skills are mounted read-only so file_sync cannot pull changes back.
- screen (kata-qemu): upstream sandbox-desktop image plus a key-only,
  non-root sshd. No credentials, egress to the public internet only.

A bot is a configMapGenerator of eight inputs under
kubernetes/apps/hermes/<bot>. The component provisions the Matrix login,
LiteLLM key, and ssh keys with Jobs that refuse unlabelled Secrets.
Chief's state restores from its existing R2 path; the old namespace stays
at zero replicas until Chief answers from the new one.

Also: trusted_servers = [] in Tuwunel, and %2F KeepUnchanged scoped to a
dedicated https-matrix listener (with cloudflared originServerName).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hermes): set the screen's session env in sshd, not /etc/environment

UsePAM no means pam_env never reads /etc/environment, so ssh sessions
lacked XDG_RUNTIME_DIR and PLAYWRIGHT_BROWSERS_PATH. SetEnv applies them
server-side.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(hermes): review fixes for the per-bot component

- envoy-external: both HTTPS listeners share ryanholt-net-production-tls,
  so Envoy Gateway set TLSOverlaps and forced ALPN http/1.1 on both.
  Set h2,http/1.1 explicitly with ClientTrafficPolicies on each listener;
  cloudflared keeps one transport per ingress rule, so no coalescing.
- agent: TERMINAL_ENV=ssh as a backstop for terminal.backend.
- screen: pin the rebuilt image (sshd SetEnv for XDG_RUNTIME_DIR and
  PLAYWRIGHT_BROWSERS_PATH).
- scripts/helm-template.sh: also render Flux Kustomizations built from
  directories that use a Kustomize Component, resolving substituteFrom
  ConfigMaps from the same build.
- docs/hermes.md: gate the pre-merge backup on a restic "snapshot saved"
  line, check the restore before trusting Chief, real rollback and
  cleanup (legacy is not a rollback point; PVs are reclaim Delete),
  Chromium runs with --no-sandbox, OPENAI_API_KEY fallback in media
  tools, backup time in UTC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…5682)

Chief runs from kubernetes/components/hermes-bot in hermes-chief and has
answered in an encrypted room, so the stopped legacy deployment goes.
Flux prunes Namespace hermes with it; the chief-data PV was set to Retain
beforehand.

docs/hermes.md: the restore check now greps restic's "restoring snapshot"
line (VolSync's log filter drops "Selected restic snapshot"). The
Chief-specific migration and rollback steps become generic backup,
restore-check and in-place restore steps, plus a record of how Chief
moved.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ef (#5683)

New input API_SERVER_ENABLED (default false). When true, the hermes
HelmRelease adds, all gated on that one value:

- ExternalSecret api-server: API_SERVER_KEY from 1Password
  hermes-<bot>/api_server_key, plus API_SERVER_ENABLED/HOST/PORT/
  MODEL_NAME; the agent reads it through an optional envFrom
- Service agent-api (8642, agent pod only)
- HTTPRoute agent-api on envoy-internal only, <bot>-api.${SECRET_DOMAIN},
  labelled public (bearer clients; Hermes checks the key)
- CiliumNetworkPolicy agent-api: 8642 from envoy-internal proxies only
- ReferenceGrant for pocket-id-exempt

The screen pod is unchanged. The component now includes the sops
component so the app Kustomization can substitute SECRET_DOMAIN.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…y wins (#5684)

Hermes loads $HERMES_HOME/.env with override=True, and the image's
docker/stage2-hook.sh writes a generated API_SERVER_KEY there on any boot
whose env has none. Chief's volume had one from earlier boots, so the API
server checked that key and every 1Password-key client got 401.

The config init container now gets the optional api-server envFrom (so
its stage2 run skips generation instead of writing a new key each boot)
and deletes any ^API_SERVER_KEY= line from /opt/data/.env when that key is
present. docs/hermes.md explains why, and corrects the claim that a bot
with the API off has no listener: Hermes keeps a loopback-only one.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…#5685)

Serves the gpt-image-2 deployment from Azure resource
ryholt-foundry-eastus2-resource on the v1 API. It uses its own key,
AZURE_FOUNDRY_EASTUS2_API_KEY, read from 1Password field
litellm/azure_foundry_eastus2_api_key. model_info.mode is
image_generation; the v1.106.6 cost map prices azure/gpt-image-2.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
)

Both are deployments on Azure resource ryholt-foundry-eastus2-resource,
next to gpt-image-2. The three entries share one anchor for the eastus2
api_base, key, and v1 API version. The v1.106.6 cost map prices both
under their azure/ names.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
carpenike and others added 30 commits November 17, 2026 23:09
… pytest workers (#5918)

On the 8-CPU pod, va-acr-app's tests need 3,200-3,600 CPU-seconds. At 12
workers the pod was throttled in about half of its periods, so the tests
are CPU-bound. Limits go to 16 CPU / 24Gi and PYTEST_XDIST_AUTO_NUM_WORKERS
to 24 (1.5 per CPU). Memory peaked at 8.6 GiB with 12 workers and grows
about 0.3 GiB per worker. Requests go to 8 CPU / 8Gi so concurrent test pods
spread across the nodes; each node has about 47 CPUs unrequested.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Prometheus only selected ServiceMonitors, PodMonitors and rules labeled
release=kube-prometheus-stack, the chart default. Rook's rook-ceph-mgr
and rook-ceph-exporter monitors carry team=rook, so the TSDB had no
ceph_* series. 18 ServiceMonitors and 2 PodMonitors were skipped in all,
among them Cilium, cert-manager, external-secrets, flux-operator,
volsync, tuppr, the DNS and tunnel exporters, CNPG and reloader.

- kube-prometheus-stack: set the five *SelectorNilUsesHelmValues to
  false so Prometheus selects every monitor, rule, probe and
  ScrapeConfig in every namespace.
- Rook's operator-owned monitors relabel cluster=rook-ceph, which wins
  over the cluster=k8s-holthome external label. An alert relabel rule
  puts Ceph alerts back on k8s-holthome before they reach Forge.
- rook-ceph-cluster: createPrometheusRules (chart default false), which
  renders prometheus-ceph-rules (16 groups, 89 rules). The Kustomization
  now waits on prometheus-operator-crds.
- New monitors for the Flux controllers, the Envoy Gateway controller
  and the Envoy proxies. They sit in kube-prometheus-stack so that
  flux-instance and envoy-gateway do not wait on the Prometheus CRDs.
- docs/architecture.md records the selection policy.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
)

GrafanaOnCallServiceDown matched up{job="grafana-oncall-engine"}, but the
ServiceMonitor sets job from app.kubernetes.io/name, so targets carry
job="grafana-oncall". absent() was always true and Forge paged Pushover
critical. Match the Service name instead, which forceRename pins.

The Finance Signal status read drops out in 2-32 minute bursts, and the
listener, report status, attention and weekly fulfilled gauges move with
it. ReportMissed had no for, and the others had 1-2m, so each burst fired
and resolved. Use for: 5m and keep_firing_for: 30m on ListenerUnhealthy,
ReportUnavailable, ReportAttention, ReportMissed and SnapshotUnavailable.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…t fail auth (#5922)

At the default 60 s TTL every active virtual key went back to postgres17
each minute. When the database stalled, key auth failed, the proxy
answered 503, and Hermes read that as "provider overloaded" and fell back
to claude-sonnet. Set general_settings.user_api_key_cache_ttl to 3600 and
document the tradeoff and the readiness-probe gap it leaves.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The volsync chart's ServiceMonitor scrapes /metrics over HTTPS without a
bearer token. The controller's authn/authz filter answers 401, so both
volsync targets have been down since Prometheus began selecting every
ServiceMonitor (#5919).

metrics.disableAuth: true adds --metrics-require-rbac=false to the
controller and drops the chart's metrics-auth ClusterRoles and binding.
Metrics stay on HTTPS :8443, which the existing ServiceMonitor uses.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
sled-1's ens1f0 has dropped about 4.9% of received frames as CRC/alignment
errors since 12 Nov and lost link twice in a day, which stalled Ceph RBD
writes across the cluster for 4-44 s. The chart's NodeNetworkReceiveErrs
never fired: it needs a 2m ratio above 1% for an hour, and the 2m ratio
dipped often enough to restart the timer.

NodeNicErrors takes the RX+TX error ratio over 1h on physical ports
(> 0.1% and > 1000 errors, for 15m, keep_firing_for 1h). NodeNicLinkFlapped
fires on any carrier change in 30m, skipping nodes up for less than 45m
so a reboot's counter reset does not count. Backtested over 9 days:
NodeNicErrors matches only sled-1 ens1f0 (lowest 1h ratio 0.58%), and
NodeNicLinkFlapped matches the 11-18 02:53 sled-1 outage and an 11-17
08:03 drop on all four nodes.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…mpty the Service (#5924)

/health/readiness answers 503 while the database is unreachable. With one
replica that removed LiteLLM's only endpoint at 11-15 01:05 and 11-17 03:49
UTC, the starts of two Hermes fallback bursts, although cached keys could
still be served. /health/liveliness still answers 503 during graceful
shutdown, so rollouts drain as before. Docs and ADR 0006 updated.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…5925)

hindsight-api was OOMKilled (exit 137) seven times, three of them between
11:00 and 11:10Z on 2026-11-18. During a Ceph alert storm, Hank woke once per
CephPGsUnclean group and recalled each time. Four recalls started within 2s,
each ran the in-process cross-encoder, and the container passed its 4Gi
limit within seconds of a ~1.3GiB baseline.

Raise the limit to 8Gi (request stays 2Gi). Limit the local reranker to two
concurrent passes (HINDSIGHT_API_RERANKER_LOCAL_MAX_CONCURRENT, default 4)
over at most 150 candidates (HINDSIGHT_API_RERANKER_MAX_CANDIDATES, default
300). Both are read by hindsight-api 1.2.0 config.py.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…domains (#5926)

* docs(mail): ryanholt.net and stefanieholt.com as alias domains

Accounts stay in holthome.net because OIDC sign-ins append
usernameDomain; the other domains are aliases. Records the edge rules
for both domains and the per-domain cutover from Microsoft 365.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(mail): devialea.com from Cloudflare Email Routing

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(mail): agent mailboxes and the agent-auth-results script

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(mail): chief@holthome.net agent mailbox

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
EMAIL_ENABLED gives a bot Hermes' bundled email platform on its own
Stalwart mailbox: an ExternalSecret with the app password and fixed
settings (mx host, IMAP 993, SMTP 465, authserv-id pin), egress to the
Stalwart pods on those ports, and a matching ingress rule on Stalwart.
Chief gets chief@holthome.net, answering only Ryan's addresses.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Without a home channel Hermes mails a "No home channel is set for
Email" notice when an email conversation starts.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Headlamp ran every request as its ServiceAccount, which was bound to
view, so Secrets and other panes were empty and nothing could be edited.

Headlamp now has its own Pocket ID client and passes the user's ID token
to the API server. The control planes trust id.holthome.net for audience
headlamp (talos/control-plane/20-authentication.yaml), mapping usernames
and groups to the oidc: prefix. Group oidc:admins is bound to
cluster-admin. The ServiceAccount no longer has a ClusterRoleBinding.

The route opts out of the gateway's Pocket ID login
(gateway.holthome.net/auth: native, ReferenceGrant headlamp-gateway-auth),
so a user logs in once. The client secret comes from 1Password item
headlamp through ExternalSecret headlamp-oidc.

The Talos config is already applied to sled-0, sled-1 and sled-2.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Add withings_get_measurements and withings_trend to the
/mcp/agent/grokbot view's tool list. Both are read-only; Withings has
no write tools.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…on (#5931)

Add every mfp_* tool (the same 33 Grokbot has) to /mcp/steffi. The view
is credentials per_user, so each call uses mfp.steffi from
homelab-mcp-user-credentials, never Ryan's shared jar; a missing or
expired file is an error, not a fall-back. Refresh the comments that
still said no view uses per_user and that mfp has only one cookie jar.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…solidation (#5932)

Meetings, People, Projects, Research, Reviews, Voice, HHS and TASKS.md
moved under Work/ on 2026-11-19, so their roots are dropped. The roots
left are tooling that stays at the vault root (Claude's memory,
Templates, the Excalidraw and ExcaliBrain plugin folders). No client's
access changes.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…5933)

Every Obsidian community plugin was removed on 2026-11-20 and the
Templates, Excalidraw and Excalibrain folders were archived, so the work
domain is now Work, memory and CLAUDE.md.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…laude, not Ryan (#5934)

Skill incident-response 2.1.0: a new alert group joins an open note about
the same component from the last six hours (group_keys), at most one board
task per incident, and Needs Ryan only for a decision or hands-on work that
is Ryan's.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… research (#5935)

Personal/Research, personal sensitivity, owned by the Grok Chief of Staff;
grokbot reads and writes it. Plus a Research project under Personal on the
OpenProject board. Guardrails live in the vault's Personal/Research/Plan.md.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Vault tools only: read and write Work/, read Personal/Research. New vault
client copilot (internal, this client only) and a Claude-only assistant
domain for memory/ and CLAUDE.md, split out of work.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat(replog): deploy the workout log on the internal gateway

SQLite on a Ceph RBD volume, one replica, VolSync snapshot backup.
The image tag is a placeholder until the first v* release exists.

Co-authored-by: Ryan Holt <ryan@ryanholt.net>

* feat(replog): leave MCP and OIDC commented until a later enable

The process serves /api/mcp only when the Pocket ID env is set. The
internal route is already native, with a commented /api/mcp rule to
uncomment alongside that env.

Co-authored-by: Ryan Holt <ryan@ryanholt.net>

* feat(replog): match Forge's public site and leave MCP to homelab-mcp

The hostname stays replog.holthome.net on both gateways, Pocket ID
login stays, and the Deployment starts at zero replicas. The website
route does not single out /api/mcp. A fresh database gets a new
secret key. The Pocket ID client secret is copied from Forge.

Co-authored-by: Ryan Holt <ryan@ryanholt.net>

* feat(replog): deny /api/mcp on the public route and add replog-k8s

The website rule is a prefix of /, which would forward /api/mcp. A longer
prefix returns 404 from Envoy and does not reach the pod. The runner
scale set follows va-acr-app-k8s and stays at minRunners 0. The app image
remains the v0.0.0 placeholder at 0 replicas.

Co-authored-by: Ryan Holt <ryan@ryanholt.net>

* revert(replog): drop the replog-k8s runner scale set

carpenike/replog is public. GitHub-hosted runners are free for it, and a
privileged Docker runner will not be registered. The /api/mcp deny stays.

Co-authored-by: Ryan Holt <ryan@ryanholt.net>

* feat(replog): pin the published image and run one replica

ghcr.io/carpenike/replog is published. Pin sha-f448ffd and its digest,
and set replicas to 1. Merging still does not move the name off Forge.

Co-authored-by: Ryan Holt <ryan@ryanholt.net>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Main published sha-9e33a7f after the earlier pin. replicas stays 1.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(homelab-mcp): add Ryan's /mcp/replog view

Pin homelab-mcp to the image published for 450891b, add the view
from mcp#169, reference the coach token from 1Password, and allow
port 8080 between homelab-mcp and replog.

* feat(replog): map the coach service token from 1Password

ExternalSecret replog writes REPLOG_SERVICE_TOKEN from item replog,
field service_token. The bootstrap admin is a coach, so
REPLOG_SERVICE_USER is not set. No token value is committed.

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Pin ghcr.io/carpenike/replog to the image published for replog
252ae15d9e3297a2eef7e881ad13f6d0f090c59a (#54). Digest confirmed
from GHCR. Replicas, ExternalSecret, network policy, ndots, and
homelab-mcp are unchanged.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
The /mcp/replog view filters to scope:atrium-personal-replog, but the
replog module was never in HOMELAB_MCP_TOOL_CATEGORIES, so it never
registered and tools/list returned nothing.
Pin ghcr.io/carpenike/replog to the image published for replog
beb93c1c224cc7a51aad0f9ceed5d4491e636c8f (#55). Digest confirmed
from GHCR. Replicas, ExternalSecret, network policy, ndots, and
homelab-mcp are unchanged.

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* feat(replog): per-user MCP credential and image pin

Set /mcp/replog to credentials per_user so the bearer is replog.ryan
from item homelab-mcp-user-credentials. Leave homelab-mcp/replog_token
mapped to HOMELAB_MCP_REPLOG_TOKEN. Pin replog to main 147ec5a by the
tag's Docker-Content-Digest.

* homelab-mcp: pin image to carpenike/mcp#170 merge (f3bb75e8)

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant