You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the Compose deployment, the port each service binds inside its container is user-configurable, which it shouldn't be.
SPIDER_STORAGE_PORT currently drives three things at once:
configs/storage.yaml → port:, i.e. what spider-storage binds inside the container.
compose.yaml → ports[].target, the container side of the host mapping.
configs/scheduler.yaml → storage_endpoint.port and configs/worker.yaml → storage.port, i.e. how the other services reach storage over the Compose network.
None of those are things a deployer needs to change: they are internal to the deployment's own network. The knob a deployer does need — where storage is reachable on the host — already exists separately as SPIDER_STORAGE_PUBLISHED_IP / SPIDER_STORAGE_PUBLISHED_PORT. SPIDER_SCHEDULER_PORT has the same shape and is arguably clearer still, since spider-scheduler is only on the internal network and is never published at all.
Fixing the container-side port would make the deployment easier to consume:
One concept, one knob. Today a user who wants to move Spider off a busy host port can plausibly reach for either variable, and only SPIDER_STORAGE_PUBLISHED_PORT actually does what they want.
Downstream integrators don't have to decide what their own port field means. In CLP's package we surface a single spider.port in clp-config.yaml; we map it to SPIDER_STORAGE_PUBLISHED_PORT and deliberately leave SPIDER_STORAGE_PORT at Spider's default, so the config CLP mounts into compression-coordinator points at spider-storage:50051. That decision only had to be made because both ports are exposed (see feat(clp-package): Select and configure Spider as the scheduler through clp-config.yaml. clp#2491).
It matches the usual Compose convention, where the container port is fixed and only published: is parameterized.
Possible implementation
Hardcode 50051 in configs/storage.yaml, in compose.yaml's ports[].target and healthcheck, and in the storage_endpoint / storage blocks of configs/scheduler.yaml and configs/worker.yaml.
Hardcode 50052 for the scheduler equivalents in configs/scheduler.yaml, configs/worker.yaml, and the scheduler healthcheck.
Keep SPIDER_STORAGE_PUBLISHED_IP / SPIDER_STORAGE_PUBLISHED_PORT as they are, and drop SPIDER_STORAGE_PORT / SPIDER_SCHEDULER_PORT from .env.example.
If there's a case for keeping them configurable that I'm missing, documenting them in .env.example as internal-only would also resolve the ambiguity.
Request
In the Compose deployment, the port each service binds inside its container is user-configurable, which it shouldn't be.
SPIDER_STORAGE_PORTcurrently drives three things at once:configs/storage.yaml→port:, i.e. whatspider-storagebinds inside the container.compose.yaml→ports[].target, the container side of the host mapping.configs/scheduler.yaml→storage_endpoint.portandconfigs/worker.yaml→storage.port, i.e. how the other services reach storage over the Compose network.None of those are things a deployer needs to change: they are internal to the deployment's own network. The knob a deployer does need — where storage is reachable on the host — already exists separately as
SPIDER_STORAGE_PUBLISHED_IP/SPIDER_STORAGE_PUBLISHED_PORT.SPIDER_SCHEDULER_PORThas the same shape and is arguably clearer still, sincespider-scheduleris only on the internal network and is never published at all.Fixing the container-side port would make the deployment easier to consume:
SPIDER_STORAGE_PUBLISHED_PORTactually does what they want.portfield means. In CLP's package we surface a singlespider.portinclp-config.yaml; we map it toSPIDER_STORAGE_PUBLISHED_PORTand deliberately leaveSPIDER_STORAGE_PORTat Spider's default, so the config CLP mounts intocompression-coordinatorpoints atspider-storage:50051. That decision only had to be made because both ports are exposed (see feat(clp-package): Select and configure Spider as the scheduler throughclp-config.yaml. clp#2491).published:is parameterized.Possible implementation
50051inconfigs/storage.yaml, incompose.yaml'sports[].targetand healthcheck, and in thestorage_endpoint/storageblocks ofconfigs/scheduler.yamlandconfigs/worker.yaml.50052for the scheduler equivalents inconfigs/scheduler.yaml,configs/worker.yaml, and the scheduler healthcheck.SPIDER_STORAGE_PUBLISHED_IP/SPIDER_STORAGE_PUBLISHED_PORTas they are, and dropSPIDER_STORAGE_PORT/SPIDER_SCHEDULER_PORTfrom.env.example.If there's a case for keeping them configurable that I'm missing, documenting them in
.env.exampleas internal-only would also resolve the ambiguity.