Skip to content

feat(docker-compose): Fix the container-side service ports instead of exposing them as SPIDER_STORAGE_PORT / SPIDER_SCHEDULER_PORT #457

Description

@20001020ycx

Request

In the Compose deployment, the port each service binds inside its container is user-configurable, which it shouldn't be.

SPIDER_STORAGE_PORT currently drives three things at once:

  • configs/storage.yaml → port:, i.e. what spider-storage binds inside the container.
  • compose.yaml → ports[].target, the container side of the host mapping.
  • configs/scheduler.yaml → storage_endpoint.port and configs/worker.yaml → storage.port, i.e. how the other services reach storage over the Compose network.

None of those are things a deployer needs to change: they are internal to the deployment's own network. The knob a deployer does need — where storage is reachable on the host — already exists separately as SPIDER_STORAGE_PUBLISHED_IP / SPIDER_STORAGE_PUBLISHED_PORT. SPIDER_SCHEDULER_PORT has the same shape and is arguably clearer still, since spider-scheduler is only on the internal network and is never published at all.

Fixing the container-side port would make the deployment easier to consume:

  • One concept, one knob. Today a user who wants to move Spider off a busy host port can plausibly reach for either variable, and only SPIDER_STORAGE_PUBLISHED_PORT actually does what they want.
  • Downstream integrators don't have to decide what their own port field means. In CLP's package we surface a single spider.port in clp-config.yaml; we map it to SPIDER_STORAGE_PUBLISHED_PORT and deliberately leave SPIDER_STORAGE_PORT at Spider's default, so the config CLP mounts into compression-coordinator points at spider-storage:50051. That decision only had to be made because both ports are exposed (see feat(clp-package): Select and configure Spider as the scheduler through clp-config.yaml. clp#2491).
  • It matches the usual Compose convention, where the container port is fixed and only published: is parameterized.

Possible implementation

  • Hardcode 50051 in configs/storage.yaml, in compose.yaml's ports[].target and healthcheck, and in the storage_endpoint / storage blocks of configs/scheduler.yaml and configs/worker.yaml.
  • Hardcode 50052 for the scheduler equivalents in configs/scheduler.yaml, configs/worker.yaml, and the scheduler healthcheck.
  • Keep SPIDER_STORAGE_PUBLISHED_IP / SPIDER_STORAGE_PUBLISHED_PORT as they are, and drop SPIDER_STORAGE_PORT / SPIDER_SCHEDULER_PORT from .env.example.

If there's a case for keeping them configurable that I'm missing, documenting them in .env.example as internal-only would also resolve the ambiguity.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions