Skip to content

About

The operations layer for your local LLM stack

Topics

Resources

Stars

30 stars

Watchers

0 watching

Forks

Latest commit

 

History

889 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

 llama-dash

llama-dash is a dashboard and API gateway for your self-hosted llama-swap + llama.cpp stack. Manage models, inspect requests, and control client access from one place.

Dark - Dashboard

Quick start · Manual setup · Client setup · Documentation

OpenAI SDK / Claude Code / Continue / Open WebUI
                    │
                    ▼
              llama-dash :3000
      dashboard · auth · logs · routing · metrics
             │                     │
             ▼                     ▼
      llama-swap :8080         direct /v1 upstreams
  llama.cpp models · peers      OpenAI · Anthropic

What it does

  • Monitor traffic, token usage, cost estimates, and CPU/GPU health.
  • Inspect request history with filters, sorting, shareable time ranges, payloads, and timing details. Usage rollups outlive raw request logs.
  • Load and unload models, inspect capabilities and history, pause inference (llama-swap v268+), and edit validated configuration.
  • Control access with dashboard login, API keys, rate limits, model permissions, and MCP relay allow-lists.
  • Route requests to local models, peers, or direct upstreams. Rewrite model names, reject requests, inject credentials or system prompts, and limit request sizes.
  • Test chat, image, speech, transcription, and realtime voice models in the playground.
Dashboard
Live traffic, tokens, model residency with per-request ticks, upstream and GPU health
Playground
Chat against local endpoints with request/response inspection
Request detail
Routing, attribution, latency, tokens, and payload metadata
Logs
Raw llama-swap, proxy, and upstream streams in one viewer
Dark - Dashboard Dark - Playground Dark - Request Details Dark - Logs
Model detail
Load history, stats, recent requests, weights size, and config context
Session History
View detailed session history
Policies
Aliases, routing rules, passthrough auth, and request limits
Requests
Searchable history with filters, sorting, and histogram
Dark - Model Details Dark - Session History Dark - Policies Dark - Requests

Quick start (Docker Compose)

The bundled stack runs llama-dash and llama-swap, with the dashboard and client API at http://localhost:3000. Choose AMD/Vulkan or NVIDIA/CUDA below. Both use the published ghcr.io/ndom91/llama-dash:latest image.

1. Copy the configuration

git clone https://github.com/ndom91/llama-dash.git
cd llama-dash
cp .env.example .env
cp config/config.example.yaml config/config.yaml

2. Set the secret and model

  • Set BETTER_AUTH_SECRET in .env to the output of openssl rand -base64 33. Compose loads .env and won't start without this secret.
  • Place a GGUF model in ./models/ and update its path and model ID in config/config.yaml. The example expects ./models/llama-3.gguf with the ID llama3.
  • If you plan to store upstream provider credentials, also set CREDENTIAL_ENCRYPTION_KEY in .env.

The example enables logTimeFormat: rfc3339nano so the Logs viewer can read original wall-clock timestamps from llama-swap's own messages, including replayed history. Existing configs can add this top-level option. This setting does not timestamp raw model output such as llama.cpp's process-uptime logs. Without an embedded timestamp, each source's initial batch shows — and subsequent live batches show browser receipt time, identified in the timestamp tooltip. Since llama-swap does not mark the history boundary, a source with no history also leaves its first live batch untimed. Reconnecting reloads history and resets this tracking.

3. Start the stack

AMD / Vulkan

docker compose -f docker-compose.amd.yaml pull
docker compose -f docker-compose.amd.yaml up -d

Uses llama-swap's unified-vulkan image. The stack passes /dev/dri to both services for inference and dashboard GPU stats.

NVIDIA / CUDA

Requires the NVIDIA Container Toolkit on the host.

docker compose -f docker-compose.nvidia.yaml pull
docker compose -f docker-compose.nvidia.yaml up -d

Uses llama-swap's unified-cuda13 image with gpus: all. It supports Turing (RTX 20xx, GTX 16xx) and newer GPUs. For Pascal cards such as the GTX 10xx or P40, change the llama-swap image to ghcr.io/mostlygeek/llama-swap:unified-cuda.

4. Open the dashboard

Open http://localhost:3000, create the first dashboard user, then load your model from Models. llama-dash runs database migrations automatically at startup.

Image versions, local builds, and config reloads

Set LLAMA_DASH_IMAGE_TAG in .env to a published image tag, such as a commit SHA tag, to pin a specific build.

To build locally, add -f docker-compose.build.yaml and --build to either vendor's up command:

docker compose -f docker-compose.amd.yaml -f docker-compose.build.yaml up -d --build

Both setups use llama-swap's unified containers and share the ./config/ directory between services. This lets llama-dash save config.yaml atomically and llama-swap reload it with -watch-config. Model files are mounted from ./models/.

Manual setup

For an existing llama-swap instance, install Node 24+ and pnpm, then run these commands from the repository:

cp .env.example .env   # edit INFERENCE_BASE_URL to point at your instance
pnpm install
pnpm db:migrate        # creates data/dash.db
pnpm dev               # http://localhost:5173

See the environment variables reference for all variables, defaults, and deployment notes.

Client setup

The Endpoints page has connection examples for curl, Python, TypeScript, Home Assistant, Claude Code, opencode, Continue, and Open WebUI.

See the Claude Code & Anthropic passthrough guide for proxy setup.

Acknowledgements

This project was developed with significant assistance from LLMs. Architecture decisions, implementation, and documentation were all shaped through human-AI collaboration.

License

MIT

About

The operations layer for your local LLM stack

Topics

Resources

Stars

30 stars

Watchers

0 watching

Forks

Contributors

Languages