airgap-bundler

airgap-bundler collects software from the internet on a connected machine, packages it into a single archive, and gives you scripts to install that archive on the airgapped side.

If you are new to this repo, the important idea is simple:

Build side: a machine with internet access downloads artifacts and creates a bundle
Install side: a machine inside the restricted environment unpacks that bundle and runs the included scripts

This tool does not build your final application images. It prepares the raw materials needed on the far side of the boundary.

Start Here

There are two common workflows.

Routine HNCD bundle (most days)

Use this when you only need the standard HNCD artifacts.

python3 build.py validate profiles/hncd.toml
python3 build.py build profiles/hncd.toml

HNCD + Spectro Vertex bundle (occasional)

Use this when you also need to seed a Spectro Cloud VerteX airgap binary.

python3 build.py validate profiles/vertex.toml
python3 build.py build profiles/vertex.toml

profiles/vertex.toml is a superset of profiles/hncd.toml. It includes the normal HNCD content plus the extra tool support needed for the Vertex wrapper flow.

What Problem This Tool Solves

Airgapped environments have a predictable problem: the software you need lives on the internet, but the machines that must run it do not.

This tool solves that by creating a portable .tar.gz bundle containing:

container images
RPM packages
OCI artifacts such as packs or charts
static helper tools such as skopeo
installation scripts
checksums and provenance files

You build the bundle once on the connected side, move it across the boundary, and install from the bundle on the restricted side.

The Two Machines

The workflow is easier to follow if you picture two separate systems.

1. Connected machine

This machine can reach the public internet.

It runs:

python3 build.py validate ...
python3 build.py build ...
optionally python3 build.py build ... --upload ...

Its job is to download everything listed in a profile and package it.

2. Airgapped target machine

This machine cannot reach the public internet.

It receives the finished bundle, unpacks it, and runs:

./install.sh for the standard HNCD install flow
./wrapper.sh first, if this is a Vertex bundle day

Its job is to load and push the bundled content into the destination environment.

High-Level Flow

At a high level:

Choose a profile
Build a bundle on the connected side
Transfer the bundle into the restricted environment
Unpack it
Configure install.env
Run the included scripts

The bundle is self-describing. If S3 upload is enabled, the destination S3 path is calculated before the build starts and written into the bundle itself.

Which Profile Should I Use?

This repo currently ships two practical profiles.

`profiles/hncd.toml`

Use this for routine builds.

It contains the standard HNCD content and keeps bundle size smaller.

`profiles/vertex.toml`

Use this when you also need to seed a Spectro Vertex airgap binary.

It contains:

everything in hncd.toml
oras support for the Vertex wrapper flow
placeholders for Vertex-specific OCI artifacts when needed

If you are unsure and do not need Spectro Vertex that day, use hncd.toml.

What Goes Into a Bundle

Four main artifact types are supported.

Container images

Examples: base images such as alpine, packer, terraform, gitlab-runner

RPMs

Examples: gitlab-runner, terraform, packer

OCI artifacts

Examples: Helm charts, Spectro packs, other OCI-distributed content

Static tools

Examples: skopeo, and on Vertex days, oras

Static tools matter because airgapped hosts often do not have the right package repositories available. Shipping the needed binaries with the bundle makes the install flow more reliable.

Example bundled tool:

tools/skopeo

The Profile File

A profile is a TOML file that declares exactly what should be collected.

If you need to add or update content, the profile is usually the first place to look.

Example: container image

[[images]]
image = "hashicorp/packer:1.14.2"
archive = "packer-1.14.2-base.tar"
dest_repo = "packer"

Example: RPM

[[rpms]]
url = "https://rpm.releases.hashicorp.com/.../packer-1.14.2-1.x86_64.rpm"
filename = "packer-1.14.2-1.x86_64.rpm"
sha256 = "..."

Example: static tool

[[tools]]
name = "skopeo"
url = "https://github.com/lework/skopeo-binary/releases/download/v1.17.0/skopeo-linux-amd64"
filename = "skopeo"
sha256 = "e31bc3c68909b51112fd91beec651acaddc1ad0d53806aa0e791d0a9ebc4293a"

Example: OCI artifact

[[oci_artifacts]]
ref = "ghcr.io/spectrocloud/packs/nginx:1.25.0"
archive = "spectro-pack-nginx-1.25.0.tar"
dest_repo = "spectro/packs/nginx"
kind = "pack"

Requirements

Build-side requirements

These are needed on the connected machine.

Tool	Why it is needed	Required?
Python 3.8+	Runs the bundler	Yes
`skopeo`	Preferred image / OCI artifact handling	Strongly recommended
`docker`	Fallback if `skopeo` is absent	Fallback only
`curl`	Downloads RPMs and other files	Yes
`tar`	Creates archives	Yes
`aws` CLI	Upload to S3	Only if using `--upload`
`terraform`	Provider mirroring	Only if mirroring Terraform providers

If you are on Python earlier than 3.11, you may also need tomli.

Target-side requirements

These are needed on the airgapped target.

Tool	Why it is needed
`bash`	Runs `install.sh` and `wrapper.sh`
`docker`	Loads images; required for the Spectro binary flow
`dnf`	Installs RPMs
`aws` CLI	Gets ECR login tokens when using ECR
`sha256sum` or `shasum`	Verifies checksums

Notes:

The bundle can ship helper tools such as skopeo and oras.
On Vertex bundle days, wrapper.sh expects oras 1.0.0. The Vertex profile includes the bundled tarball for that.

Building a Bundle

1. Validate the profile

This checks syntax and structure without downloading artifacts.

python3 build.py validate profiles/hncd.toml

or:

python3 build.py validate profiles/vertex.toml

2. Build the bundle

Routine HNCD bundle:

python3 build.py build profiles/hncd.toml

Vertex bundle:

python3 build.py build profiles/vertex.toml

3. Optionally upload to S3

python3 build.py build profiles/hncd.toml --upload --bucket my-transfer-bucket

Output lands in ./output/.

Example:

output/hncd-20260310-144018.tar.gz
output/bundle.env

bundle.env contains the bundle name, checksum, and pre-calculated S3 URLs.

Example:

BUNDLE_NAME=hncd-20260310-144018.tar.gz
BUNDLE_URL=s3://hncd-artifacts/airgap-bundles/hncd/hncd-20260310-144018.tar.gz
BUNDLE_SIDECAR_URL=s3://hncd-artifacts/airgap-bundles/hncd/hncd-20260310-144018.tar.gz.sha256
BUNDLE_SHA256=9c645978303a60226a91ff4e96c383a121d556b8404da8d85f11c1012bbca4f5

bundle.env is also written during --dry-run so you can inspect the would-be values before uploading.

Installing on the Target Side

1. Unpack the bundle

tar xzf hncd-20260310-144018.tar.gz
cd hncd-20260310-144018/

2. Create `install.env`

cp install.env.example install.env
vi install.env

For many environments, the minimum useful values are:

AWS_ACCOUNT_ID=123456789012
AWS_REGION=us-gov-west-1

If you are pushing to a non-ECR registry instead, you may use:

REGISTRY=registry.internal.example.com:5000
REGISTRY_AUTH=none

install.env sits beside install.sh and wrapper.sh and is read automatically.

Do not commit install.env. Use install.env.example as the template.

3. Run the installer

sudo ./install.sh

The installer prints a pre-flight summary before doing any real work. If a required tool is missing, it tells you what was skipped and why.

Useful one-off overrides

CLI flags override install.env for that run only.

sudo ./install.sh --skip-rpms
sudo ./install.sh --skip-rpms --skip-terraform --skip-oci
sudo ./install.sh --registry registry.other.example.com

Vertex Workflow

Use this section only when you built with profiles/vertex.toml and need to seed a Spectro Cloud VerteX airgap binary.

What `wrapper.sh` does

wrapper.sh is included in the bundle and handles the Spectro pre-seeding flow.

In simplified terms, it:

verifies the .bin file exists
optionally verifies a .sha256 sidecar
logs in to private ECR using aws, oras, and docker
exports the ECR_* variables the Spectro scripts expect
extracts the binary with --noexec --target <dir>
runs legacy compatibility steps if needed
runs any custom hook scripts bundled in hooks/wrapper.d/
cleans up the extracted working directory

Running the Vertex wrapper

After unpacking the bundle and filling in install.env:

sudo ./wrapper.sh airgap-fips-v4.8.39.bin

Then continue with the normal install flow if needed:

sudo ./install.sh

Vertex configuration

The Vertex section in install.env.example controls this flow.

Typical values:

AWS_ACCOUNT_ID=102015676567
AWS_REGION=us-gov-west-1
ECR_DNS_SUFFIX=amazonaws.com
VERTEX_IMAGE_BASE=spectro-images
VERTEX_PACK_BASE=spectro-packs

install.sh uses the same AWS_REGION + ECR_DNS_SUFFIX derivation for normal ECR pushes. REGISTRY_REGION remains available only as a legacy override for ECR auth/derivation when you need it.

In the current GovCloud flow, both the image and pack registries default to the same private ECR host derived from:

<AWS_ACCOUNT_ID>.dkr.ecr.<AWS_REGION>.<ECR_DNS_SUFFIX>

Typical values:

AWS/GovCloud: ECR_DNS_SUFFIX=amazonaws.com
SC2S: ECR_DNS_SUFFIX=sc2s.sgov.gov

Custom Vertex hook scripts

If you need a one-off fix for a specific Spectro release, the preferred method is:

add an executable .sh file under:
```
include/hooks/wrapper.d/
```
rebuild the bundle
run wrapper.sh normally

Hooks are copied into the final bundle under:

hooks/wrapper.d/

Each hook receives the extracted airgap directory as argument 1.

A sample no-op hook is included:

hooks/wrapper.d/00-example-hook.sh

Legacy compatibility steps such as patching bin/functions.sh or running apply_pack.sh / apply_patch.sh are still supported, but they are treated as compatibility shims rather than the main extension mechanism.

Testing a Bundle (HNCD Engineers)

If you received a bundle from Recro and need to validate it in your environment before using it, see:

docs/bundle-install-test-handoff.md

That guide covers:

moving the bundle into your S3 location
triggering the bundle-install-test CI job
reading pre-flight output
retrieving logs from failed runs

GitLab CI Pipeline

The .gitlab-ci.yml automates the build → transfer → install sequence.

In plain language:

the build job runs on a connected runner
it uploads the bundle to S3
it publishes bundle.env as a dotenv artifact
later jobs read BUNDLE_URL and BUNDLE_SHA256 from that artifact

By default, install and verify are manual. Set AUTO_INSTALL=true if you want install to start automatically after build.

Runner requirements

Tag	Where it runs	Needs
(none)	Anywhere	`python3`
`connected`	Internet-routable environment	`python3`, `skopeo`, `curl`, `aws` CLI
`airgapped`	Inside the restricted VPC	`bash`, `dnf`, `aws` CLI, passwordless `sudo`

Passwordless sudo is required on the airgapped runner:

# /etc/sudoers.d/gitlab-runner
gitlab-runner ALL=(ALL) NOPASSWD: ALL

Required CI/CD variables

Variable	What it is for
`AWS_ACCESS_KEY_ID`	Connected-runner AWS credentials
`AWS_SECRET_ACCESS_KEY`	Connected-runner AWS credentials
`AWS_ACCOUNT_ID`	12-digit AWS account ID
`AWS_REGION`	Example: `us-gov-west-1`
`TRANSFER_BUCKET`	S3 bucket used to move bundles
`TRANSFER_PREFIX`	S3 key prefix, such as `airgap-bundles/hncd`

Optional CI/CD variables

Variable	Effect
`SKIP_IMAGES`	`true` to skip image pull
`SKIP_RPMS`	`true` to skip RPM download
`SKIP_OCI`	`true` to skip OCI artifact pull
`AUTO_INSTALL`	`true` to auto-trigger install
`ECR_DNS_SUFFIX`	Override derived ECR hostname suffix, for example `sc2s.sgov.gov`

Where This Tool Stops

This tool gets raw materials across the boundary. It does not produce the final hardened workload images.

For example, the bundler delivers base content such as:

alpine
packer
terraform
gitlab-runner

After those are available in the restricted environment, another tool such as oci-image-builder can build final STIG-hardened images on top of them.

A useful mental model is:

airgap-bundler moves ingredients
downstream image-building tools make the finished product

Bundle Layout in S3

Bundles land at the configured prefix with a checksum sidecar.

Example:

airgap-bundles/hncd/
  hncd-20260310-144018.tar.gz
  hncd-20260310-144018.tar.gz.sha256

The full S3 URL is calculated at build start and written into meta/bundle-source.txt inside the archive.

You can set the destination in the profile:

[s3]
bucket = "my-transfer-bucket"
prefix = "airgap-bundles/hncd"
region = "us-gov-west-1"

You can also override bucket or prefix at build time with CLI flags.

Other Commands

Upload an already-built bundle to S3:

python3 build.py upload output/hncd-20260306-120000.tar.gz --bucket my-bucket

List bundles in S3:

python3 build.py list --bucket my-bucket

If You Only Remember Three Things

Use profiles/hncd.toml for normal days and profiles/vertex.toml for Vertex days.
Build on the connected side, install on the airgapped side.
On Vertex days, run ./wrapper.sh before ./install.sh.

Name		Name	Last commit message	Last commit date
.omegon		.omegon
ai/memory		ai/memory
airgap_bundler		airgap_bundler
docs		docs
include		include
profiles		profiles
test		test
tests		tests
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
CROSS_PLATFORM_ASSESSMENT.md		CROSS_PLATFORM_ASSESSMENT.md
README.md		README.md
build.py		build.py
demo.sh		demo.sh
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation

airgap-bundler

Start Here

Routine HNCD bundle (most days)

HNCD + Spectro Vertex bundle (occasional)

What Problem This Tool Solves

The Two Machines

1. Connected machine

2. Airgapped target machine

High-Level Flow

Which Profile Should I Use?

profiles/hncd.toml

profiles/vertex.toml

What Goes Into a Bundle

Container images

RPMs

OCI artifacts

Static tools

The Profile File

Example: container image

Example: RPM

Example: static tool

Example: OCI artifact

Requirements

Build-side requirements

Target-side requirements

Building a Bundle

1. Validate the profile

2. Build the bundle

3. Optionally upload to S3

Installing on the Target Side

1. Unpack the bundle

2. Create install.env

3. Run the installer

Useful one-off overrides

Vertex Workflow

What wrapper.sh does

Running the Vertex wrapper

Vertex configuration

Custom Vertex hook scripts

Testing a Bundle (HNCD Engineers)

GitLab CI Pipeline

Runner requirements

Required CI/CD variables

Optional CI/CD variables

Where This Tool Stops

Bundle Layout in S3

Other Commands

If You Only Remember Three Things

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages

`profiles/hncd.toml`

`profiles/vertex.toml`

2. Create `install.env`

What `wrapper.sh` does