Skip to content

build: expire the rolling majestic tarballs a day after they were fetched - #2352

Merged
widgetii merged 1 commit into
masterfrom
dl-expire
Sep 2, 2026
Merged

build: expire the rolling majestic tarballs a day after they were fetched#2352
widgetii merged 1 commit into
masterfrom
dl-expire

Conversation

@widgetii

@widgetii widgetii commented Sep 1, 2026

Copy link
Copy Markdown
Member

Problem

Field report from a from-source build (gk7205v200): the SSH claim door works, EULA and all,
but the browser door dead-ends —

majestic[925]: Not found: /setup.html
majestic[925]: Empty password for root is not allowed

while cameras.html from the same directory serves fine. That "Not found" line is
send_file_to_user's access() failure: /var/www/setup.html genuinely is not in the image.
The majestic-webui-dist.tar.gz download is a rolling release asset — fixed filename, no
hash — so buildroot's dl cache keeps the first copy forever. A cache primed between
cameras.html (≤ Aug 21) and setup.html (Aug 30) reproduces the report exactly: a fresh
majestic that expects the setup page, paired with a webui from before the page existed. The
majestic.*.master.tar.bz2 tarballs have the same trap. (Both Empty password lines are
device_unclaimed()'s debug heartbeat — the camera and the gate are fine.)

What this does

prepare (runs before every build) deletes cached copies of the two rolling assets when
they are older than a day. Fresh copies are kept, so offline rebuilds inside that window
still work; CI downloads into an empty cache every run and never reaches the branch.

Hardware tested on

Not a camera-behavior change in itself — it changes which bytes a stale-cache build fetches.
The failure it prevents was reproduced end to end today on a hi3516ev300 lab camera: a local
build with an Aug-21 cached dist produced exactly Not found: /setup.html on an unclaimed
camera; the same camera with the current dist serves setup.html 200 while unclaimed.

Evidence

Before (stale cache, local build, unclaimed camera):

# curl -s -o /dev/null -w '%{http_code}' http://camera/setup.html
404        (majestic log: Not found: /setup.html)

After (expiry applied to a scratch dl dir with one 2-day-old copy of each asset and one
fresh majestic tarball):

$ find dltest -name '*.tar*'          # after `make BOARD=... prepare`
dltest/majestic/majestic.gk7205v200.lite.master.tar.bz2   # fresh copy kept

and with the current dist on the same camera, unclaimed:

setup.html: 200 bytes=15403
/: 302 -> http://10.216.128.68/setup.html

ci-matrix.py --self-test passes.

Immediate workaround for anyone already bitten, no rebuild of the cache needed:
rm output/buildroot-*/dl/majestic*/majestic* and rebuild.

Scope

  • No kernel patches under general/package/all-patches/linux/ (those go to OpenIPC/linux)
  • No files specific to a single retail camera model (those go to OpenIPC/builder)
  • No probing or bring-up tooling (that goes to OpenIPC/ipctool)
  • Nothing under general/overlay/ or in a shared load_<vendor> script hardcodes a value specific to my board
  • Package sources come from an OpenIPC repository, and any version bump keeps at least the specificity of the pin it replaces (a new package should pin a full 40-character SHA)
  • No LD_PRELOAD, and no binaries that cannot be rebuilt from source
  • New code is selected by a defconfig, so CI actually builds it

@qodo-free-for-open-source-projects

Copy link
Copy Markdown

PR Summary by Qodo

Expire stale rolling Majestic downloads before builds

🐞 Bug fix ⚙️ Configuration changes 🕐 Less than 10 minutes

Grey Divider

AI Description

• Expires Majestic and WebUI rolling downloads after 24 hours during build preparation.
• Prevents stale binaries and WebUI assets from producing missing setup pages.
• Preserves cached artifacts for one day to support short-term offline rebuilds.
Diagram

graph TD
  A["Build command"] --> B["Prepare target"] --> C["Download cache"] --> D{"Older than day?"}
  D -- Yes --> E["Delete rolling assets"] --> F["Fetch current assets"] --> G["Firmware build"]
  D -- No --> H["Keep cached assets"] --> G
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Pin immutable versioned artifacts
  • ➕ Guarantees reproducible builds with explicit versions and hashes
  • ➕ Eliminates time-based cache invalidation
  • ➕ Prevents binary and WebUI compatibility drift when versions are coordinated
  • ➖ Requires upstream to publish immutable artifacts
  • ➖ Requires package metadata updates for every release
  • ➖ May delay consumption of rolling releases
2. Always purge rolling artifacts
  • ➕ Ensures every connected build retrieves the newest assets
  • ➕ Requires no timestamp policy
  • ➖ Breaks offline rebuilds
  • ➖ Adds unnecessary downloads to repeated local builds
  • ➖ Increases exposure to transient network or upstream failures

Recommendation: The 24-hour expiry is the best scoped immediate mitigation because it fixes indefinitely stale caches while retaining limited offline reuse. Immutable, hash-pinned releases would provide stronger reproducibility and should be preferred long term if upstream release coordination is feasible.

Files changed (1) +11 / -0

Bug fix (1) +11 / -0
MakefileExpire stale Majestic rolling artifacts during preparation +11/-0

Expire stale Majestic rolling artifacts during preparation

• The 'prepare' target now searches the configured or default Buildroot download cache for Majestic binary and WebUI rolling archives older than 1,440 minutes and deletes them. Missing directories or cleanup failures remain non-fatal, while fresh cached artifacts are preserved.

Makefile

@qodo-free-for-open-source-projects

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Grey Divider

Tip of the day
💡 Did you know, you can describe a rule in plain language on the Rules page and Qodo drafts it for you

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

…ched

The majestic and majestic-webui downloads are rolling release assets: a
fixed filename, no hash, refreshed upstream on every publish. Buildroot's
dl cache keeps the first copy forever, so a from-source build with an old
cache pairs a majestic that expects the setup page with a webui from before
the page existed -- majestic then logs Not found: /setup.html on an
unclaimed camera while the SSH door works, which is exactly how it was
reported. Fresh copies are kept, offline rebuilds inside the day still
work, and CI downloads into an empty cache every run and never gets here.
@widgetii
widgetii merged commit 88c2d43 into master Sep 2, 2026
118 checks passed
@widgetii
widgetii deleted the dl-expire branch September 2, 2026 00:20
widgetii added a commit that referenced this pull request Sep 2, 2026
build.yml and build-one.yml cache `output/dl` and prune moving-ref tarballs
from it. Buildroot never wrote there: DL_DIR defaults to $(TOPDIR)/dl, and
TOPDIR is the source tree the Makefile hands to -C, so downloads landed in
output/buildroot-$(BR_VER)/dl.

  $ make -C output/buildroot-2024.02.10 O=$PWD/output -p | grep '^DL_DIR'
  DL_DIR := .../output/buildroot-2024.02.10/dl

output/dl therefore never existed. actions/cache only archives paths that
exist, so the cache saved nothing and restored nothing, and the refresh find
matched nothing -- both silently, the find because of its own
`2>/dev/null || true`. The two cancelled to "every board re-downloads every
tarball, every run", which is also why CI escaped the stale rolling tarball
that #2352 fixed for local builds. Confirmed on this PR's own run:
`Cache not found for input keys: dl-09, dl-`, while the per-board ccache beside
it restored normally.

Setting BR2_DL_DIR in the environment moves buildroot to the directory the
cache already keys on, rather than teaching three more call sites to spell
output/buildroot-$(BR_VER)/dl and keep it in step with BR_VER. Buildroot reads
it ahead of .config by design, and the Makefile's expiry from #2352 already
prefers $(BR2_DL_DIR).

Turning the cache on makes the refresh live for the first time, which is the
half that keeps it safe -- neither should be restored without the other. Its
regex covers the moving-ref class this tree produces: 24 packages pin
_VERSION = HEAD and download as <pkg>-HEAD.tar.gz, majestic-webui as
-dist.tar.gz, majestic's S3 tarball as .master.tar.bz2. In a real 201-file dl,
9 files are moving and 192 are semver- or SHA-named and immutable.

build-one.yml becomes restore-only. `dl-<month>` is one key shared with the
whole matrix and actions/cache saves only on a key miss, so a single-board
manual dispatch could otherwise freeze that month's snapshot for everyone. The
matrix keeps the shared key deliberately: the repo already holds 197 active
caches totalling 15.3 GB because ccache is keyed per board, and a second
per-board series would evict the entries that buy the most build time.

The suppression goes with it. A cold cache is legitimate and now says so;
anything else is an error that reaches the log.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant