Failed Star (fs)

A small, self-contained LLM inference engine for Apple Silicon —
built from scratch, in the open, to learn and teach how inference engineering works.

📖 Read the learning site →

A failed star (a brown dwarf) is smaller than a dwarf star: not enough mass to sustain fusion. This project is the smaller sibling of Dwarf Star (ds4), antirez's self-contained inference engine for DeepSeek-V4. Where ds4 targets big MoE models on 96GB+ Macs, Failed Star develops against a tiny model on the author's 64GB M5 MacBook Pro — a development machine, not a minimum hardware requirement — and trades raw capability for something else: every line is meant to be read, understood, and learned from.

Why this exists

The goal is understanding inference, by building it. Reading about attention is one thing; writing the kernel that computes it and watching tokens stream out of your own code is another. This repo is the second thing.

Three sources form its spine, cross-referenced throughout the docs. (The prerequisites point to a wider set of optional brush-up and go-deeper resources — those fill gaps; these three are what the docs lean on.)

1. The concepts — Inference Engineering by Philip Kiely (Baseten, 2026). The "why" and the vocabulary. (Peruse the free interactive guide, or get your own copy from Baseten Books; an optional local Inference Engineering.pdf is ignored by Git.)
2. A real implementation — ds4, cloned into reference/ds4/. The "how a pro does it." Working code doesn't lie.
3. Architecture context — Sebastian Raschka's free articles: the architecture comparison, gallery, and workflow for understanding LLMs. (His book is a good optional extra, not a dependency — see the prerequisites.)

How it's built

• Host language: Rust. Model loading, tokenizer, orchestration, sampling, KV cache — all Rust.
• GPU kernels: MSL (Metal Shading Language) — hand-written, one operation per file, just like ds4's metal/ shaders.
• Metal via raw FFI / the Objective-C runtime — no convenience wrapper crate. We send messages to Metal ourselves so nothing is hidden. Tight, like ds4.
• First model: Qwen3-0.6B — a tiny dense model with GQA, RoPE, SwiGLU, and RMSNorm; small enough to inspect and debug while still looking like a real modern LLM.
• Correctness via golden vectors: match logits from the model's official implementation. (Python appears only as a one-shot oracle, never as a second engine.)
• Runtime target: modern Apple Silicon/macOS only. The CPU reference may run elsewhere incidentally for learning and orb checks; Linux is not a supported product platform.

Repo layout

fs/
├── README.md                  ← you are here
├── PLAN.md                    ← the milestone curriculum (M0 … M7 + optional experiments)
├── PROGRESS.md                ← running session log; start here each session
├── Inference Engineering.pdf  ← local copy of the book (ignored; bring your own)
├── src/                       ← Rust engine + thin CLI
├── scripts/                   ← uv-managed Python oracle/data scripts
├── tests/golden/              ← committed golden fixtures for verification
├── tools/                     ← site/sync helper scripts
├── docs/                       ← the learning site + notes (served at /fs via Pages)
│   ├── index.html             ← learning-site landing page (rich HTML)
│   ├── prerequisites.md       ← what to know before diving in (read this first)
│   ├── 00-map.md              ← THE BIG PICTURE of an inference engine
│   ├── m0-tokenizer.md        ← M0 writeup (.md + rich .html version)
│   ├── dev-loop.md            ← how to resume work after a break
│   ├── testing.md             ← verification strategy and golden-vector plan
│   ├── diagrams.html          ← shared diagram gallery
│   ├── RESOURCES.md           ← cross-reference index (book §§, ds4 files, Raschka)
│   ├── learnings/             ← bite-sized notes on what we figured out & why
│   └── assets/                ← logo + site assets
├── reference/ds4/             ← antirez's ds4 — pinned git submodule (read-only ref)
└── models/                    ← downloaded model assets (ignored; generated locally)

Where to start

Returning after a break? Start with Coming back to Failed Star (HTML reading route). It separates required reading from optional detours and explains exactly what works, what the tests establish, and the next complete teaching arc.

1. Read docs/prerequisites.md — the honest "what to know before you dive in" (no training or backpropagation), with brush-up resources and a knowledge-map.
2. Read docs/00-map.md — the end-to-end picture of an inference engine, with an "abstraction ladder" so you can stop digging at whatever depth interests you.
3. Skim PLAN.md — the milestones.
4. Each session, open PROGRESS.md to see what's next.
5. If resuming development, use docs/dev-loop.md and docs/testing.md for the local checks and verification strategy.

Status

🌱 M0 — Tokenizer: ✅ done. M1 — Load the weights: ✅ done. M2 — Forward pass: ✅ done. M3 — Deterministic generation: in progress. fs logits "The capital of France is" runs all 28 Qwen3-0.6B layers in clear CPU Rust; all four complete official fp32 checkpoints match, including every one of the 151,936 logits. fs inspect models/qwen3-0.6b loads config.json + model.safetensors, derives the expected tensor set from the config, cross-checks the file against it, and prints a shape-first legend + tensor table + verdict — the real model checks clean (311 tensors, 596M logical params; see docs/m1-weights.md). The weights are mmap'd zero-copy via raw POSIX FFI and widened for the clear f32 CPU oracle. M3 is the current planning target; no generation loop is implemented yet. It starts with greedy parity before sampling.

Milestones (the full curriculum, with cross-links, lives in PLAN.md):

• M0 — Tokenizer — text ↔ token IDs, verified against the real vocab
• M1 — Load the weights — fs inspect; tensor set cross-checked vs the config
• M2 — Forward pass → logits — fs logits; full official fp32 parity
• M3 — Deterministic generation, then sampling ← current
• M4 — KV cache
• M5 — Metal bring-up + end-to-end GPU execution
• M6 — Profile-driven Metal optimization/fusion
• M7 — Quantization go/no-go

Default CI is model-free on macOS Apple Silicon (fmt, build, test, clippy). Asset-backed model checks and future Metal correctness/performance checks are explicit local-Mac runs; standard GitHub runners are not promised to run Metal, and orb results are useful but not authoritative.

This is a slow, multi-session learning project. It is not (yet) fast, capable, or finished — that's the point. Local models keep getting better; the bet is that a clean, well-documented small engine becomes more useful to more people over time.