Skip to content

Latest commit

 

History

History
150 lines (123 loc) · 9.82 KB

environment-variables.md

File metadata and controls

150 lines (123 loc) · 9.82 KB

Environment variables

Every environment variable KempnerForge reads, grouped by source. Authoritative locations in the tree: kempnerforge/distributed/setup.py, kempnerforge/resilience/elastic.py, kempnerforge/metrics/logger.py, and the helper launch scripts under scripts/slurm/.

Rank / world-size (torchrun or SLURM)

Read by get_world_info() in kempnerforge/distributed/setup.py. Each torchrun-style var falls back to the SLURM equivalent, so the same entry point works under both launchers.

Variable Fallback Purpose
RANK SLURM_PROCID Global rank (0..world_size-1)
LOCAL_RANK SLURM_LOCALID Rank within a node (0..gpus_per_node-1)
WORLD_SIZE SLURM_NTASKS Total number of ranks

get_world_info() reads whichever is set and then calls os.environ.setdefault(...) on all three, so downstream code (PyTorch, WandB, logging) sees a torchrun-shaped environment even on srun-direct launches.

Rendezvous (MASTER_ADDR / MASTER_PORT)

Variable Set by Notes
MASTER_ADDR user / launch script / init_distributed() If unset, init_distributed() runs `scontrol show hostnames $SLURM_JOB_NODELIST
MASTER_PORT user / launch script / init_distributed() If unset, derived from SLURM_JOB_ID via a seeded RNG; otherwise picked by binding an ephemeral socket

The SLURM launch helpers (scripts/slurm/multinode.sh, _run_training.sh) export both before srun so every rank agrees. init_distributed() only fills in missing values — it never overwrites.

NCCL / Gloo (auto-set by _set_nccl_env())

init_distributed() calls _set_nccl_env(), which detects the IB interface (ls /sys/class/net | grep '^ib') and populates defaults:

Variable Default Purpose
NCCL_SOCKET_IFNAME detected IB interface (e.g. ib0) NCCL bootstrap transport
GLOO_SOCKET_IFNAME same as above Gloo (used by DCP async checkpoint coordination)
NCCL_IB_DISABLE 0 Keep InfiniBand RDMA enabled
NCCL_NET_GDR_LEVEL 2 GPUDirect RDMA level — enables GPU→NIC DMA

All four use setdefault, so anything you export before launch wins. The launch scripts also set NCCL_IB_GID_INDEX=3, which NCCL reads and the Python code does not. There is no NCCL_TIMEOUT setting: neither NCCL nor PyTorch reads that variable. distributed.nccl_timeout_sec sets the default process group's timeout; the groups init_device_mesh creates for other mesh dimensions keep PyTorch's 30-minute default.

SLURM metadata (resilience)

Read by get_slurm_context() and running_under_slurm() in kempnerforge/resilience/elastic.py. Purely informational — used to populate logs and decide whether a job is a restart.

Variable Purpose
SLURM_JOB_ID Job identifier; also used to derive MASTER_PORT
SLURM_JOB_NAME Job name (logged)
SLURM_JOB_NODELIST Nodelist; parsed by scontrol show hostnames to find MASTER_ADDR
SLURM_NNODES Node count (logged)
SLURM_NTASKS_PER_NODE Tasks per node (logged)
SLURM_RESTART_COUNT Non-zero means this is a requeue (used to detect restarts)
SLURM_JOB_PARTITION Partition name (logged)
SLURM_ARRAY_TASK_ID Array index when applicable (logged)

Logging

Variable Read in Purpose
NO_COLOR kempnerforge/metrics/logger.py Disables ANSI color codes in logs when set to any truthy value
RANK same Used to prefix log lines with the current rank

Experiment tracking (MLflow / Databricks)

Read by the metrics config (kempnerforge/config/metrics.py, at load), the MLflow backend (kempnerforge/metrics/mlflow/backend.py), the eval publisher (kempnerforge/metrics/mlflow/publish.py), MLflow itself, or the Databricks SDK; the backend and the publisher prepare the Databricks variables through one helper in kempnerforge/metrics/mlflow/auth.py. KempnerForge only fills these when they are unset (the token alias and the profile export also fill an empty DATABRICKS_TOKEN or DATABRICKS_CONFIG_PROFILE), so a non-empty value you export always wins. scripts/check_env.py reads the auth variables too, to report which route a shell has. Setup and auth routes: MLflow backend.

Variable Read by Purpose
MLFLOW_TRACKING_URI config, backend Tracking URI when metrics.mlflow_tracking_uri is unset (config, then this, then databricks)
MLFLOW_EXPERIMENT backend Experiment when metrics.mlflow_experiment is unset; an absolute workspace path on Databricks
DATABRICKS_HOST, DATABRICKS_TOKEN SDK The environment auth route; override the values of a profile selected by a databricks://<profile> URI or DATABRICKS_CONFIG_PROFILE, and with none selected, setting either one makes the SDK ignore ~/.databrickscfg
DATABRICKS_API_TOKEN backend, publisher Alias copied into an unset or empty DATABRICKS_TOKEN for a Databricks URI
DATABRICKS_CONFIG_PROFILE SDK ~/.databrickscfg profile; set from a databricks://<profile> tracking URI by the backend and the publisher when unset or empty. A value naming another profile is kept with a warning: MLflow's tracking calls and the experiment lookup then use the URI's profile, and SDK clients built from the environment alone, MLflow's run-artifact uploads among them, use this one
DATABRICKS_AUTH_TYPE SDK Never set by KempnerForge; pat is recommended in job scripts using token or profile auth
MLFLOW_HTTP_REQUEST_TIMEOUT MLflow Set to 30 by the backend before importing mlflow
MLFLOW_HTTP_REQUEST_MAX_RETRIES MLflow Set to 3 by the backend before importing mlflow
MLFLOW_DATABRICKS_ENDPOINT_HTTP_RETRY_TIMEOUT MLflow Set to 60 by the backend before importing mlflow
MLFLOW_DISABLE_TELEMETRY MLflow Set to true by the backend before importing mlflow, so MLflow's usage telemetry never draws from Python's random after a resume restored it; export false to keep the telemetry (MLflow § Usage telemetry)
MLFLOW_ACTIVE_MODEL_ID MLflow, publisher MLflow's process-wide active model (mlflow.set_active_model sets one too); never set by KempnerForge. While one is set, the eval publisher refuses to publish metrics that have no model to link to, since log_batch would silently link them to it

User-facing launch-script variables

Read only by the SLURM helpers under scripts/slurm/, not by any Python module:

Variable Default Purpose
KEMPNERFORGE_LOG_DIR ${SLURM_SUBMIT_DIR:-$PWD}/logs/distributed Per-rank log destination for _run_training.sh, _run_distributed_tests.sh, _dcgm_monitor.sh
IB_IFNAME ib0 (fallback in interactive.sh) IB interface for the NCCL/Gloo exports in the launch scripts
PYTORCH_CUDA_ALLOC_CONF (unset) Passed through in scripts/slurm/interactive.sh and the MoE-EP benchmark scripts as expandable_segments:True

Who sets what

Source Variables it populates
User / job script KEMPNERFORGE_LOG_DIR, IB_IFNAME, PYTORCH_CUDA_ALLOC_CONF, optionally NCCL_* overrides
torchrun RANK, LOCAL_RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT
srun (SLURM) SLURM_* (incl. SLURM_PROCID, SLURM_LOCALID, SLURM_NTASKS, SLURM_JOB_*)
scripts/slurm/_run_training.sh Exports RANK, LOCAL_RANK, WORLD_SIZE, MASTER_ADDR, MASTER_PORT, NCCL_*, GLOO_* before calling the trainer
init_distributed() Fills missing RANK/LOCAL_RANK/WORLD_SIZE from SLURM_*, missing MASTER_ADDR/MASTER_PORT, and all four NCCL_*/GLOO_* defaults
MLflow backend, eval publisher DATABRICKS_CONFIG_PROFILE (from a databricks://<profile> URI) and DATABRICKS_TOKEN (from DATABRICKS_API_TOKEN) when unset or empty; the backend also fills the three MLFLOW_* HTTP defaults and MLFLOW_DISABLE_TELEMETRY above

See also