Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion docs/cli/datasets-and-training.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,9 @@ To list training containers and find container IDs, including your organization'
viam train containers list --org-id=<org-id>
```

The `--org-id` flag is optional.
Without it, the command lists only Viam-managed containers.

## Manage datasets

### Create a dataset
Expand Down Expand Up @@ -270,7 +273,9 @@ On ARM Macs, Docker runs them through Rosetta emulation, which may be slower.
viam training-script test-local \
--dataset-root=./my-dataset/ \
--training-script-directory=./my-training-script/ \
--container-id=<container-id>
--dataset-file=dataset.jsonl \
--container-id=<container-id> \
--model-output-directory=./output/
```

## Manage training containers
Expand Down
4 changes: 2 additions & 2 deletions docs/cli/reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -3144,7 +3144,7 @@ Manage training scripts for [custom ML training](/train/custom-training-scripts/
```sh {class="command-line" data-prompt="$"}
viam training-script upload --framework=<framework> --org-id=<org-id> --path=<path-to-script> --script-name=<script-name> --type=<type>
viam training-script update --org-id=<org-id> --script-name=<script-name> --visibility=<visibility>
viam training-script test-local --training-script-directory=<script-directory> --dataset-root=<dataset-root> --container-id=<container-id> [...named args]
viam training-script test-local --training-script-directory=<script-directory> --dataset-root=<dataset-root> --container-id=<container-id> [--dataset-file=<dataset-file>] [--model-output-directory=<output-dir>] [--custom-args=<key=value>]
```

### `training-script upload`
Expand Down Expand Up @@ -3193,7 +3193,7 @@ Run an ML training script locally with Docker, in the same container a cloud tra
See [Test locally with Docker](/train/custom-training-scripts/#test-locally-with-docker).

```sh {class="command-line" data-prompt="$"}
viam training-script test-local --training-script-directory=my-training/ --dataset-root=./my-dataset/ --container-id=abc123 --model-output-directory=./output/
viam training-script test-local --training-script-directory=my-training/ --dataset-root=./my-dataset/ --dataset-file=dataset.jsonl --container-id=abc123 --model-output-directory=./output/
```

<!-- prettier-ignore -->
Expand Down
21 changes: 5 additions & 16 deletions docs/train/custom-containers.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,8 @@ When you start a [custom training job](/train/custom-training-scripts/), you can
Viam-managed containers come with Python and TensorFlow.
Any other dependency your training script lists in `setup.py` gets installed with pip at the start of every job, on billed GPU time.
For large dependency stacks, such as the libraries for fine-tuning a vision-language-action (VLA) model, that install can take a long time.
A custom container has those dependencies built in, so each job starts training right away.
A custom container has those dependencies built in.
As long as the container includes every dependency your training script needs, jobs don't spend time installing dependencies with pip and start training right away.

The container holds only the environment.
Your training script stays in the registry and works without changes: the same `setup.py` and `model/training.py` structure runs in a custom container just as it does in a Viam-managed one.
Expand Down Expand Up @@ -48,20 +49,8 @@ Keep your training code in your training script, which stays private to your org

## Build and push an image

Start from an NVIDIA CUDA base image so your dependencies can use the GPU that training jobs run on.
Then install Python, pip, and the packages your training script needs.

For example, this `Dockerfile` installs PyTorch and Hugging Face Transformers:

```dockerfile {class="line-numbers linkable-line-numbers"}
FROM nvidia/cuda:12.4.1-cudnn-runtime-ubuntu22.04

RUN apt-get update && \
apt-get install -y --no-install-recommends python3 python3-pip && \
rm -rf /var/lib/apt/lists/*

RUN pip3 install --no-cache-dir torch transformers
```
Build an image that meets the [requirements](#requirements) and includes the packages your training script needs.
You can start from any base image.

Build the image for `linux/amd64` and push it to Docker Hub:

Expand All @@ -73,7 +62,6 @@ docker buildx build --platform linux/amd64 \
Make sure the repository is public on Docker Hub.

You can use any tool that produces a standard Docker image.
For example, [Cog](https://cog.run/) can generate a GPU-ready image from a short configuration file.

## Register the container

Expand Down Expand Up @@ -139,6 +127,7 @@ See [Test locally with Docker](/train/custom-training-scripts/#test-locally-with
```sh {class="command-line" data-prompt="$"}
viam training-script test-local \
--training-script-directory=my-training/ \
--dataset-file=dataset.jsonl \
--dataset-root=<destination> \
--container-id=<container-id> \
--model-output-directory=<output-dir>
Expand Down
Loading