Manifest release hardening

We should harden the manifest release process to make it difficult to publish a manifest update that breaks users installations.

This is my promised follow-up to my comment on #250, about what guarantees we want to provide about midenup releases and the published manifest, as well as how we should go about checking that we don't break those promises.

There are really three primary guarantees we're interested in here:

Once I install midenup, published manifest changes won't break my installation. A change to the manifest which requires me to update midenup must either be published as a new schema version, or as a change to the minimum-supported midenup version. Both should result in midenup telling me that I need to upgrade. However, I should be able to continue to use and manage my local installation without upgrading - the upgrade is only required to obtain a fresh copy of the upstream manifest.
If I upgrade midenup, it should upgrade my local state OR tell me why it can't attempt the upgrade - it should never attempt an upgrade that cannot succeed. We do not support rollbacks, so once an install is upgraded, the old state is no longer recoverable. The user should be told that they can perform the upgrade anyway by performing a clean installation.
Toolchains are self-consistent, i.e. all components are known to work with each other. It should not be possible for a manifest change to upgrade/downgrade a component and end up breaking any use of the toolchain due to the component being incompatible in some way with other components of the toolchain.

This implies the following properties of the midenup client and manifest:

Each schema version has a minimum supported client version. This is used to determine when an older client must upgrade to use a new manifest change that didn't require a breaking change to the schema (e.g. due to a bug in midenup itself, or due to a change in schema semantics without a change in structure).
New clients must support all older manifests, to be able to upgrade them (or at a minimum, specify that an upgrade of the old installation cannot be performed, and a clean install is needed).
Old clients must check that they meet the minimum client version, in addition to the schema version, and let the user know they need to upgrade their client when they get a chance. This check must not get in the way of commands that do not strictly require something from the upstream manifest (e.g. miden commands, or midenup commands that operate purely locally).

A release of midenup and/or manifest must pass the following gates in order to proceed:

Gate	What it should prove
Structural and transition validation	Every manifest resolves; published selectors and migration paths remain usable; component removals, acquisition changes, and downgrades receive explicit treatment.
Released-client compatibility	Explicitly supported released binaries can install the candidate manifest and run its commands. Start with the oldest supported release and latest; expand where behavior differs.
Upgrade compatibility	Install from the previous published manifest, preserve that home, switch to the candidate, and update. Cover network followers, pinned versions, explicit component selections, and migration origins; verify data and selection preservation.
Candidate-client compatibility	The proposed midenup release reads the live manifest and upgrades homes created by supported older clients.
Artifact and functional checks	On both shipped targets—Linux x86-64 and macOS ARM64—verify artifact integrity, installation, successful command exits, and a small compile/run workload that exercises component compatibility. Test prebuilt acquisition and Cargo fallback separately.
Publication checks	Deploy the exact tested bytes, record their digest and baseline, serialize publication, and verify the served manifest afterward.

As for when these checks should be performed - it is primarily important that we gate releases; but it would also be useful to know when a change is going to force client upgrades. Sometimes that will be obvious, but having a CI job that checks for these and tags PRs that will break existing clients as such, will aid in review. Further, some of these gates can be performed on any change to channel-manifest.json - and ideally we could be precise about what we test about those changes, e.g. if a component version is updated for a single toolchain, then we run checks for just that one toolchain.

Lastly, we may want to specify a process for evicting legacy toolchains from the manifest. We have several toolchains in there today that aren't actually properly functional, and should be removed - and are almost certainly unused. Going forward though, we probably need to define our criteria and process for toolchain removal.

I'll follow up with more implementation/technical details in a later comment - but I wanted to leave this comment to get the discussion started. If there are other things you feel this outline failed to cover, definitely let me know.

Writing some other points that were mentioned:

The manifest should be an immutable, versioned release artifact. The existing release tool should allow publishing it under its own tag, either alone or together with a midenup release. This way, a manifest change that needs a new client should not take effect before binaries are published.
The minimum client version specified in the manifest would apply to the whole manifest, not to individual toolchains. It would be bumped if the manifest changes without a new schema version (the recent addition of the %version expression would have been an example).
The CI should test upgrading from the minimum supported client to the current ref on every change.
Each component should have test scenarios, perhaps written by the team that owns it since it would be difficult to maintain the whole suite here. They should be scripted against a toolchain version and run only for the components a manifest change affects. Manifest changes would ideally be approved or rejected automatically, once the testing above covers them.

The manifest should be an immutable, versioned release artifact. The existing release tool should allow publishing it under its own tag, either alone or together with a midenup release. This way, a manifest change that needs a new client should not take effect before binaries are published.

I've changed my mind on the details around this. What I really want is the ability to optionally prevent the manifest from being published until the next release of midenup is finalized. This might warrant a new release unit type/configuration in miden-release-tool, I'm not 100% sure yet. Basically, the existing publish workflow would need to check the release configuration to determine whether it is gated on a release, and if so, it would skip automatic publishing, and expect itself to be dispatched by the release workflow post-finalize.

The issue with the workflow as it is today, is that a breaking schema change gets published before the midenup release is available that can read it - a user that happens to hit the window between the two artifacts being published will attempt to upgrade, and find that there is no new release. We need to be able to control when and in what order those items get published.

Each component should have test scenarios, perhaps written by the team that owns it since it would be difficult to maintain the whole suite here. They should be scripted against a toolchain version and run only for the components a manifest change affects. Manifest changes would ideally be approved or rejected automatically, once the testing above covers them.

I would emphasize that teams are responsible for their components - when we publish a new toolchain, components are not carried forward until their teams open the requisite PR (or one of us does it for them voluntarily). They would be responsible for ensuring that the tests are implemented and work before that PR gets merged.

My thinking on how tests would be expressed/defined, would be driven off a configuration file like the following:

# .release/acceptance.toml

# A scenario that verifies behavior of the 'vm', 'core', and 'midenc' components
# by executing a script that either passes or fails based on its exit code
[[scenarios]]
title = "Can compile and prove a basic MASM program with midenc and the VM"
toolchain = "^0.17.0"
requires = ["vm", "core", "midenc"]
kind = "script"
script = '''
#!/usr/bin/env bash

set -euo pipefail

cat <<EOF
begin
    push.1 push.2 add
EOF > source.masm

miden build source.masm -lmasp:static=core --exe -o source.masp
miden vm prove source.masp
'''

# A scenario that executes a lit/FileCheck test suite with a given name
# The scenario fails if any test in the suite fails. Lit would be invoked with variables
# defined for the toolchain and component versions, so that tests could be
# ignored/applied conditionally
[[scenarios]]
title = "Can debug programs produced by midenc"
toolchain = "^0.17.0"
requires = ["core", "midenc", "miden-debug"]
kind = "lit"
suite = "debug-midenc-produced"

Scenarios would be run any time a change affects a required component in a matching toolchain. A toolchain component for which there is no matching acceptance test would be rejected (i.e. every component must have at least one scenario that matches it in order for the change to be considered acceptable).

The different kinds of scenarios, and how they are configured is yet to be determined - what I sketched out here is just food for thought, but I think we can cover basically everything of interest with the following:

Shell scripts, where the test relies on the script executing successfully. Code review would be used to ensure shell scripts adhere to certain common rules, to avoid scripts exiting successfully when they should have failed. This type of test is useful for simple composition of components where the output doesn't matter, only the exit status, such as the example shown.
lit/FileCheck tests. These are similar to scripts, but allow us to verify things about the output of commands that are run. This is especially important for debugger tests, where we need to script debugger actions, and make sure that certain debug info was actually available. litcheck provides built-in facilities for conditionally applying tests based on environment and configuration, which allows us to sidestep implementing that in our own test tooling.

The test harness would handle setting up the test environment for each scenario, including the midenup installation, shell environment, etc.

The output would look something like:

Verifying revision deadbeef..

Pre-requisites:

* Manifest is valid .. PASS
* All toolchain components have acceptance tests .. PASS

Affected toolchains (and components):

* 0.17.0 (vm, core, midenc, miden-debug)

Running matching acceptance tests for toolchain 0.17.0..

* Can compile and prove a basic MASM program with midenc and the VM .. PASS
* Can debug programs produced by midenc .. FAIL

<error output here>

* Can build a basic wallet and create an account via MockChain .. SKIPPED

Revision deadbeef should be REJECTED due to one or more test failures, see output for details

Obviously we could make it look nicer in an actual terminal, but that's the gist of how we could render the test scenarios being executed.

@TomasArrachea

@TomasArrachea The following is the design document that came out of working through the implications of all the guarantees we want to check. I have an implementation plan for this version as well, but we should refine this spec first - once we're happy with it, then the finalized implementation plan can be shared as well.

1. Purpose and scope

Midenup releases and the upstream manifest evolve independently of users' installations. Release
safety must preserve three properties:

G1 — Local continuity: a manifest update cannot disable installed tools or local management.
A client upgrade may be required to obtain fresh upstream information, but not to keep using
the existing installation.
G2 — Safe client upgrades: a new client migrates a supported home (the local installation directory)
or explains why conversion
is unsupported before changing it. Committed upgrades are one-way and recover forward after
interruption. A refusal explains the clean-install alternative.
G3 — Compatible toolchains: components work together and updates preserve the intended
combination. This work establishes structural validity and snapshot consistency. Component
interoperability tests are a separate project.

The implementation covers manifest compatibility, local-state migration, artifact integrity,
installation consistency, and release/publication gates. It reuses midenup's resolver and
installation publisher, with a private Rust verifier coordinating the checks. The existing
release-tool retains responsibility for package publication.

Structural and integrity checks cannot establish that components interoperate. They also do not
provide application database migration, custom-build certification, local-state rollback, or
protection against a compromised manifest host.

2. Manifest compatibility and local continuity

Three independent version boundaries

Version	Responsibility
`manifest_version`	Structural schema compatibility, following semantic versioning. Breaking structural changes require a major increment; compatible additions use a minor increment.
`min_client_version`	The oldest client that correctly interprets the manifest's fields, even when older clients can deserialize them.
Home format	Compatibility with local state, directory layout, receipts, and recovery journals.

Changing a field's type or removing a required part of the schema requires an appropriate schema
version change. Raising min_client_version cannot substitute for that change. When the structure
stays compatible but a field's meaning changes, raising the minimum identifies the first client
that implements the required interpretation. A change affecting both structure and meaning must
satisfy both boundaries.

The existing v3 parser requires min_client_version, deserializes the document, and compares the
minimum with its embedded package version using SemVer precedence. Equal or newer clients are
accepted; older clients receive an upgrade-required error. Missing or malformed minima are invalid,
and build metadata does not affect precedence. This ordering is sufficient: the minimum must be
enforced before using the manifest, but need not be checked before deserialization. Schema-version
dispatch remains responsible for rejecting unsupported structures.

Baseline and endpoints

B is the first release using the new manifest endpoint. All safety work described here lands
before B, and its exact staged binaries must pass qualification before publication. The initial
manifest's min_client_version equals B's release version; the development placeholder does not
select that version. After publication, record B's version, binary hashes, home format, and fixtures
as the fixed compatibility baseline. Later minima cannot fall below B on this endpoint.

The repository already separates the manifests, served under https://0xmiden.github.io/midenup/:

Readers	Repository file	Endpoint
Pre-B clients	`manifest/channel-manifest.json`	`/channel-manifest.json`
B and later	`manifest/v3/manifest.json`	`/v3/manifest.json`

Freeze the legacy file byte-for-byte and include it in every Pages artifact. Old clients ignore
unknown minimum fields, so keeping their endpoint unchanged prevents new manifest changes from
breaking them. Never redirect it or implicitly fall back to it from the new endpoint. This cannot
repair existing client bugs or guarantee continued availability of external artifacts.

Refresh and local operations

Refresh distinguishes unsupported schema, unsatisfied minimum, malformed content, and transport
failure. Upgrade diagnostics identify the installed/required version or schema, endpoint, and
whether cached information is being used. They direct users to the documented installer or
cargo install --locked --force midenup; midenup has no self-update command.

Only validated bytes may atomically replace the cache. Rejected content and interrupted writes
leave the last usable copy intact. Commands that permit cached results may continue with an accurate
warning; an explicit refresh cannot report fresh success when using cached data. Upstream-dependent
work without a usable cache fails clearly.

Installed command execution, show list, selecting an installed toolchain, uninstall, and garbage
collection operate entirely from local state. They need neither upstream access nor a manifest
cache. In contrast, list describes upstream availability, and installing a missing toolchain
requires upstream information. Optional annotations must not delay or prevent local results.

A pre-B cache lacking min_client_version is not usable by the new parser. It may remain as historical
provenance, but no minimum is fabricated for it. Home migration and local use still work without it.

3. Safe home upgrades

One format guard and a concurrency boundary

Add a small home-format.json envelope. Its integer format covers state, layout, receipts, and
journals; its optional migration reference identifies a committed conversion. state_version can
still select a state decoder, but cannot alone protect unversioned layout changes. B introduces
format 1; later incompatible local interpretations advance the encompassing format.

Every entry point checks this envelope before legacy migration, state decoding, recovery, shim
setup, or format-sensitive mutation. Unknown or malformed formats refuse without changing the
installation. A missing envelope means either a proven empty home or a recognized legacy layout
requiring preflight—not permission to overwrite existing state.

A separate format lease prevents a converter from changing metadata beneath another command:

Ordinary commands take a shared lease, reread the envelope, and retain the lease through a
supervised component's exit. Writers then take HomeLock and recheck state.
Conversion releases any shared lease, takes an exclusive lease, then HomeLock, and preflights
again. Locks always follow this order; shared ownership is never upgraded in place.
Lease acquisition uses bounded waits or explanatory refusal. Running tools are not killed to
allow conversion.

Healthy dispatch avoids HomeLock, so an ordinary install does not block it. The format lease
is the synchronization cost of safe conversion. Installed publication and user-data paths remain
stable; this design does not track detached application lifetimes or relocate live databases.

Preflight and forward recovery

Under exclusive format ownership and HomeLock, construct the complete conversion plan, including
state, layout, receipts, links, and any pending recovery. Unsupported later steps, unknown journals,
conflicting destinations, unreadable required records, or ambiguous ownership reject the whole plan
before authoritative mutation. Even recognized installation recovery waits until preflight succeeds.

Migration preserves installed versions, commands, selections, defaults, custom recipes, and user
data without fetching a manifest or rebuilding tools. Missing acquisition provenance may prevent
future verified reuse, but must not disable an otherwise usable local installation.

Stage and validate the result, then durably prepare a private journal. Atomically replacing and
synchronizing the envelope with the target format and journal reference is the one-way commit
point. Only afterward may authoritative state/layout change. Older clients then refuse the new
format, while a supporting client resumes the recorded local plan without consulting upstream.

An unreferenced prepared journal is not authoritative: older clients ignore it rather than guessing
how to execute or delete it. A referenced missing or unsupported journal causes unchanged refusal.
Precommit staging may be discarded; after completion, obsolete metadata and recovery material are
removed. There is no supported rollback copy. Creating/acquiring lock files is permitted during
refusal; installation records, toolchain files, cache, configuration, links, and user data remain intact.

Preflight cannot predict hardware failure. It rejects determinable incompatibilities before commit
and provides forward recovery for interrupted supported conversions. Refusal names the problem and
explains how to make a clean installation in a separate home, without automatically deleting data.

Historical support

Support ordinary released v1.0.0–v1.2.0 homes through tested adoption adapters. Required unsupported
alpha-era conversions receive nonmutating clean-install guidance. An inactive legacy file beside
valid state is not a reason to overwrite that state or reject the home. Any conversion needed by a
supported stable fixture must have a safe adapter; startup must not invoke old converters speculatively.

The reliable older-client format guard starts at B. Pre-B executables cannot be made to honor it
and are unsupported against converted homes; continued pre-B use requires a separate home.

4. Preserving component snapshots

An official snapshot identifies one channel's component definitions and immutable acquisition
inputs for each target: versions, sources, features, locked dependencies, Rust toolchain, artifact
hashes, installation dependencies, commands, and profiles. Date or network-pointer changes alone
do not change its identity. Computing the identity requires no mutable network lookup.

Every install, activation, update, channel migration, and metadata-only reconciliation resolves the
complete desired selection against one snapshot. Validate dependency closure, target support, and
name/destination conflicts before changing the installation. Carry forward a component only when
its definition and verified acquisition identity match. A conflicting held-back component causes
refusal; it does not silently lose its pin or get relabeled with a newer definition.

Acquire and verify changed outputs, then publish the selection atomically through the existing
installation publisher. The user can retain the old snapshot or update the required components
together. Selecting or executing an unchanged installation remains a local operation.

Explicit custom/path-patched rebuilds remain available as custom installations. Automatic upstream
reconciliation must not assemble an unverified mixture around their modifications or assign them
an official snapshot identity.

Use controlled components and small source fixtures to test manager behavior: command routing,
selected definitions, acquired bytes, partial selections, promotions, removals, metadata-only
updates, and preservation of user-data paths. These checks do not establish application compatibility.

Future interaction gate

A later project will supply configured component interactions and scenarios, then automate the
affected scenarios as an additional release gate. A version bump selects interactions involving
that component; other behavior-changing inputs must also be accounted for. requires expresses
installation dependencies, not the complete set of interactions.

The release pipeline retains exact candidate/predecessor manifests and component identities for
this integration. Configuration, selection rules, scenario representation, runner, coverage policy,
and CLI adapters belong to the later design. B's qualification does not depend on them, and current
reports must not claim that absent interaction tests passed.

5. Immutable inputs and artifact integrity

Production definitions use exact crate versions or Git commits, usable committed/published lockfiles,
exact Rust toolchains and features, and pinned external build/container inputs. Validate source
availability and these identities without silently rewriting the candidate. A pinned digest fixes
expected bytes even at a mutable URL; it does not guarantee their future availability.

Use digest for target-independent SHA-256 and digests for a target-to-SHA-256 map. Exactly one
form is permitted, with a digest for every resolved target. Verify archive transport bytes before
extraction and plain transfers before publication. A mismatch is terminal and cannot trigger source
fallback. Ordinary unavailability may use an advertised fallback with its own pinned recipe.

Receipts record source/lockfile identity, recipe, acquisition path, and output hashes. Verify cache
hits and carried-forward outputs as well as new downloads. Controlled build fixtures exercise
locked builds and fallback behavior on each supported platform; successful fallback cannot satisfy
a case requiring the prebuilt path.

These checks establish integrity relative to the accepted manifest. They do not prove every
component builds or interoperates, promise reproducible source outputs, or authenticate an unsigned
manifest from a compromised host.

6. Verification and release evidence

A private tools/release-safety Rust package reuses midenup's resolver/planner and runs actual
released or staged binaries in isolated homes. .release/safety.toml records B, the frozen legacy
hash, retained channels, targets, client/home catalogs, fixture versions, input requirements, and
explicit transitions or retirements. Initial native targets are x86_64-unknown-linux-gnu and
aarch64-apple-darwin.

Checks cover:

G1: compatible, raised-minimum, unsupported-schema, malformed, and unavailable manifests,
with valid, absent, and unusable caches; local commands with network access prohibited. Minimum
rejection uses structurally compatible documents, while structural breaks use the appropriate
schema version. Missing/malformed minima and SemVer boundary cases are tested separately.
G2: every cataloged home/layout/journal variant, whole-plan refusal, interruption at commit
boundaries, concurrent readers/writers, and two installed client binaries. B qualification uses
labeled derived future-format homes; later format-changing candidates must also produce actual
homes that real released B and other older readers refuse unchanged.
Integrity and consistency: structural resolution, immutable inputs, corrupt transfers,
fallback/cache/reuse behavior, and complete-snapshot installation through controlled fixtures.

Catalog stable releases from v1.0.0, including pre-B characterization cases. Identify binaries by
hash and release provenance. Record fixture origins and transformations explicitly; instrumented
fault-injection builds and derived corrupt/interrupted homes are not untouched release artifacts.

Generate a deterministic case inventory before running tests. Reports bind source commit,
candidate/predecessor hashes, client binaries, targets, fixture/policy/verifier identities, expected
and executed case IDs, observations, and logs. Missing, duplicate, skipped, cancelled, mismatched,
or infrastructure-failed required cases block the gate. An empty required suite cannot pass.
Reports describe structural, integrity, and lifecycle evidence without claiming interoperability.

PRs use the checked-in client catalog. Release/publication gates and a daily check compare it with
live stable-release metadata; ordinary PRs do not depend on that lookup. Qualification may use the
exact prospective release entry, with binary hashes in generated evidence. Successful finalization
opens or updates a catalog PR with published provenance. Until reconciled, further releases and
publications block; unrelated PRs continue. Failure to create that PR is a follow-up failure, not
an unpublished client release.

7. CI integration

Trigger	Required checks
Every PR / merge group	Offline policy/catalog consistency, static manifest resolution, and safety fixtures.
Manifest changes	Applicable historical-client cases, changed input/snapshot validation, and manager transitions on both targets.
Client/runtime changes	Exact candidate binaries, historical homes, recovery/concurrency, integrity, and coherence fixtures. Unknown impact selects the full applicable suite.
Client release	Live catalog completeness and qualification of exact staged assets against relevant manifests and homes.
Manifest publication	Exact predecessor/candidate checks, complete applicable evidence, frozen legacy bytes, and serialized deployment.
Daily schedule	Catalog drift, served hashes, and selected availability checks.

Cheap G1 cases include every B+ stable release. Installation fixtures cover the oldest/latest
accepting clients and each distinct installer/dispatcher implementation; grouping requires reviewed
implementation identity. Published inputs are verified without adding an interaction-test matrix.

Publication reruns required behavior from the reviewed main commit rather than trusting arbitrary
PR reports. Build caches do not replace test evidence. Use bounded parallelism/timeouts and retain
complete reports. Measure cold/warm runtime, transfer volume, and runner cost during rehearsal;
5–15 minutes is an initial, unmeasured offline PR budget. Resource limits cannot silently remove
required cases. Interaction-test costs will be assessed separately.

8. Publication and recovery

Manifest publication becomes a main-tip workflow_dispatch run with a protected environment and
reviewed declaration binding candidate path/hash to an expected predecessor. Merging does not
publish. Approval covers the concrete candidate and its evidence.

Under one publication lock, recheck main tip, catalog completeness, candidate/report identity, and
the deployed predecessor. Deploy the exact tested Pages artifact, including frozen legacy bytes
and a receipt binding source and manifest hashes. Hold the lock through confirmation. Changed
inputs require updated evidence rather than silently retargeting the operation.

Use hosting deployment records and their bound receipt to establish the predecessor; a cached CDN
response alone is insufficient. Record deployment ID and artifact identity. Poll public URLs and
the receipt for expected bytes, using bounded retries and cache-busting/no-cache requests. Allow
an initial 20-minute propagation window; those requests cannot guarantee bypassing every cache.

Failure before deployment leaves production unchanged. A successful hosting deployment with stale
or unavailable served bytes is deployed, unconfirmed. Reconcile its identity before another
attempt. Unknown identity blocks publication; a known artifact with persistent serving failure can
receive an explicitly reviewed forward repair through the full gate. An already confirmed exact
candidate is an idempotent success. There is no automatic rollback.

Intentional component downgrades require reviewed source/destination declarations and manager
transition checks. This validates selection/publication mechanics, not application compatibility;
local home-format rollback remains unsupported.

9. Retention and rollout

Retain channels 0.9.0, 0.10.0, 0.11.5, 0.12.4, 0.13.3, 0.14.0, 0.15.0, 0.16.0, and 0.17.0.
Audit their structure and immutable inputs early: existing definitions need artifact digests,
fixed source/toolchain identities, and lockfile verification. Repairs must be reviewed; an
unrepairable input needs an explicit retention amendment. Retirement removes future upstream
availability, never installed tools or data, and includes selector/migration-origin behavior tests.

Implement the runtime boundaries and verifier, repair retained inputs, then rehearse the complete
publication process in a nonproduction environment. Interaction scenarios are not a prerequisite.

For the first release:

Set min_client_version to B's actual version and qualify the exact staged B assets with that
v3 manifest and the frozen legacy file. Any served placeholder is an ordinary predecessor.
Deploy and confirm the qualified v3 manifest and unchanged legacy bytes, then publish those
exact B binaries. B must not become available while its endpoint is unconfirmed or unqualified.
Finalize B's catalog entry and preserve its fixtures for subsequent releases. If client
publication fails or partially succeeds, reconcile immutable package/tag state before retrying;
preserve the confirmed manifest and never lower its minimum to bypass the failure.

This first publication may rely on prospective B evidence because no released client yet uses the
new endpoint. Afterward, a newly required client must be published before the live minimum is raised
to it. Existing users then have both an available upgrade path and continued local use.

Thanks for writing this up. I think it sounds good overall, leaving a few questions/notes on each section.

Sections 1 and 4: interaction tests

Components are hard to test alone, so scenarios will likely end up exercising some degree of integration as well (node and client, faucet and node, etc). Anyway, I think the testing harness may deserve its own separate issue and discussion.

Section 2: compatibility

The spec says all safety work lands before "B", the first midenup release that reads /v3/manifest.json. Technically I think we only need to ship first the binary-side pieces (the home-format guard, the cache fixes and the digest verification), so we can start there. The verifier, CI and publication gates run outside the binary, so they can follow.
Agree with freezing the legacy file, though frozen as is, pre-B users would never learn they need to upgrade. We could make one last legacy edit before freezing. One option is a channel with a component kind 1.2.x doesn't know, which fails on resolve with "not supported by this version of midenup; upgrade midenup or deselect it". The other is bumping the legacy manifest_version, so every fetch warns with OutdatedMidenup and falls back to the cache.

Section 3: home upgrades

I am not sure if adding a conversion journal is strictly needed, since today's conversions can safely be re-run from any point that could have been interrupted. But following the existing publication journal design could also make sense.

Section 4: snapshots

Makes sense. Carry-forward a component based on the plan key already exists, so the new piece is the snapshot identity, which would be a hash of the channel definition. "Verified" carry-forward depends on Section 5 digest verification. Until then the identity only shows the definition matches, not the bytes.

Section 5: integrity

The manifest schema should reject mutable sources outright, allowing only registry versions and git revisions through a manifest-specific authority type (patches keep the current Authority), so a branch, tag or path fails to parse in any manifest. Patches on the miden-project.toml already cover moving sources. Old clients accept a superset, so no schema bump is needed, and v3 is already all-registry. This also makes Section 4's snapshot identity meaningful.
Per-target digests could live in the existing targets.<target> entries instead of a separate digests map. Today the target-specific artifact form has a single digest shared by all targets, but each target downloads a different file, so it can only match one of them. A parallel map would also have to stay in sync with targets.
New requirements should apply to the whole manifest, not only new channels. Every artifact needs a digest, so retained channels get backfilled.
Receipts would also need output hashes and the acquisition path (prebuilt or fallback) to verify cache hits and carried-forward outputs.

Section 6: verification

I am not sure if I see what is the failure that the evidence tooling (case inventory, hash-bound reports, missing or skipped cases blocking) would address. Maybe a simpler solution would be to avoid a separate tools/release-safety with its own policy file, and keep this logic in the workflows, integration tests and update-manifest.
A checked-in client catalog forces a commit or PR on main after every release, and blocks releases until it merges. The release and publication gates already fetch live release metadata to check the catalog. We could have them use it directly instead, and avoid the manual step.

Section 7

Looks good.

Section 8: publication

Once all the checks are in place, we could ideally aim for automatic publication on merge, without requiring manual approval.

Section 9: rollout

I'm not sure how would a non-production environment rollout would work here.

In general, I think a some points here can be discussed separately in dedicated issues.

Components are hard to test alone, so scenarios will likely end up exercising some degree of integration as well (node and client, faucet and node, etc). Anyway, I think the testing harness may deserve its own separate issue and discussion.

Definitely. I think for purposes of release hardening, we need only have a sense of how we'd invoke the test harness, but we wouldn't need any component tests to start with (though naturally we'll have some essentially right away). We can treat that as largely a separate concern.

The spec says all safety work lands before "B", the first midenup release that reads /v3/manifest.json. Technically I think we only need to ship first the binary-side pieces (the home-format guard, the cache fixes and the digest verification), so we can start there. The verifier, CI and publication gates run outside the binary, so they can follow.

Agreed, it over states the timing there - as you noted, it is really referring to the safety work that needs to be implemented in midenup itself.

Agree with freezing the legacy file, though frozen as is, pre-B users would never learn they need to upgrade. We could make one last legacy edit before freezing. One option is a channel with a component kind 1.2.x doesn't know, which fails on resolve with "not supported by this version of midenup; upgrade midenup or deselect it". The other is bumping the legacy manifest_version, so every fetch warns with OutdatedMidenup and falls back to the cache.

When we're ready to cut over the legacy file, I think bumping the manifest_version is the simplest approach, and then we can remove the legacy manifest after some deprecation period. We'll want to test to make sure that doesn't cause any issues with the upgrade, but I think that should suffice.

I am not sure if adding a conversion journal is strictly needed, since today's conversions can safely be re-run from any point that could have been interrupted. But following the existing publication journal design could also make sense.

I think the benefit of the conversion journal is that we wouldn't have to guess about how to resume after an interruption - we'd have a clear record of what was done. I'm not entirely convinced we don't need it, but it might be premature to introduce it until we expect to have conversions that would really need it, or review of the final implementation makes it clear the journal solves a real gap around recovery.

The manifest schema should reject mutable sources outright, allowing only registry versions and git revisions through a manifest-specific authority type (patches keep the current Authority), so a branch, tag or path fails to parse in any manifest. Patches on the miden-project.toml already cover moving sources. Old clients accept a superset, so no schema bump is needed, and v3 is already all-registry. This also makes Section 4's snapshot identity meaningful.

I don't necessarily agree on this - for local development purposes, being able to patch in a component by referencing a given branch or tag is convenient. I don't have an issue with preventing the production channel manifest from containing mutable refs in it though, as a release gate.

Per-target digests could live in the existing targets.<target> entries instead of a separate digests map. Today the target-specific artifact form has a single digest shared by all targets, but each target downloads a different file, so it can only match one of them. A parallel map would also have to stay in sync with targets.

Agreed

New requirements should apply to the whole manifest, not only new channels. Every artifact needs a digest, so retained channels get backfilled.

Agreed - insofar as such requirements are applicable to older channels (and AFAIK that's certainly the case for now). In cases where a requirement cannot be applied to an older channel, we'll likely need some kind of explicit allowlist for those exceptions (should the time come).

Receipts would also need output hashes and the acquisition path (prebuilt or fallback) to verify cache hits and carried-forward outputs.

Yes, receipts are specified to contain both.

I am not sure if I see what is the failure that the evidence tooling (case inventory, hash-bound reports, missing or skipped cases blocking) would address. Maybe a simpler solution would be to avoid a separate tools/release-safety with its own policy file, and keep this logic in the workflows, integration tests and update-manifest.

That tool is really the test runner - and those items are what it operates on and produces.

I definitely don't want to encode any logic/rules in YAML/workflows if at all possible - ideally all of this stuff is easily run locally for testing/verification. There are limits to what we can test with GitHub Actions short of doing stuff for real - so I prefer to avoid encoding anything aside from job orchestration and basic conditional execution/concurrency rules there. Test matrices are likely to get messy as well, so trying to handle those complexities in workflow files would get ugly.

Integration tests are not really the place for the component testing stuff either IMO - that suite should be focused specifically on testing midenup's functionality without any dependency on a specific channel or set of components (we have one integration test which is an exception to this, but we will be able to remove it once we have component testing in place as a release gate).

I'm not opposed to combining the functionality of release-safety and update-manifest into something like manifest-utils or whatever, and having it handle tasks for modifying the manifest as well as verifying it and running test cases against it - but I do see those as separate tasks within the tool.

A checked-in client catalog forces a commit or PR on main after every release, and blocks releases until it merges. The release and publication gates already fetch live release metadata to check the catalog. We could have them use it directly instead, and avoid the manual step.

We can just treat it as another release artifact, that way it doesn't need to be checked-in - we just fetch it from the release rather than git.

Once all the checks are in place, we could ideally aim for automatic publication on merge, without requiring manual approval.

Potentially - at a minimum to enable this we'd require a reviewer from a specific group to sign off on any PR that would trigger a release (technically, that's already the case, but I've been signing off on my own releases as repo admin), and the reviewer would have to be someone other than the PR author.

The goal for now is to make it so signing off on a release is a painless and safe process, which should make turnaround time much faster, and from there we can properly explore making it fully automated. I think the concern will always be that automatic releases involving what amounts to arbitrary execution on a user's machine is a disaster waiting to happen, so having human manual review is a necessary check on something slipping through that shouldn't.

I'm not sure how would a non-production environment rollout would work here.

I can trigger a release that I never finalize (i.e. a staging release), so that would be sufficient for us to test that the release process works as expected, but I need to pin down the exact process for manifest-only releases so that they get gated the same way. Assume for now anyway that we'll exercise the test rollout as a staging release.

In general, I think a some points here can be discussed separately in dedicated issues.

Agreed

Navigation Menu

Repository navigation

Manifest release hardening #248

1. Purpose and scope

2. Manifest compatibility and local continuity

Three independent version boundaries

Baseline and endpoints

Refresh and local operations

3. Safe home upgrades

One format guard and a concurrency boundary

Preflight and forward recovery

Historical support

4. Preserving component snapshots

Future interaction gate

5. Immutable inputs and artifact integrity

6. Verification and release evidence

7. CI integration

8. Publication and recovery

9. Retention and rollout

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Manifest release hardening #248

Description

Activity

bitwalker commented on Oct 4, 2026

TomasArrachea commented on Oct 6, 2026

bitwalker commented on Oct 8, 2026

bitwalker commented on Oct 21, 2026

1. Purpose and scope

2. Manifest compatibility and local continuity

Three independent version boundaries

Baseline and endpoints

Refresh and local operations

3. Safe home upgrades

One format guard and a concurrency boundary

Preflight and forward recovery

Historical support

4. Preserving component snapshots

Future interaction gate

5. Immutable inputs and artifact integrity

6. Verification and release evidence

7. CI integration

8. Publication and recovery

9. Retention and rollout

TomasArrachea commented on Oct 23, 2026

bitwalker commented on Oct 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions