Skip to content

fix(orchestrator): envd error responses truncated to 100 chars, hiding root cause in logs and error chain #3480

Description

@AdaAibaby

Problem

In packages/orchestrator/pkg/sandbox/envd.go, all envd error-response bodies are truncated to 100 characters via utils.Truncate(string(body), 100) before being embedded in fmt.Errorf strings and log fields:

```go
// line 182 — callEnvdCollapse
fmt.Errorf("collapse returned %d: %s", resp.StatusCode, utils.Truncate(string(body), 100))

// line 208 — postEnvd (covers /init, /pause, /resume, …)
fmt.Errorf("%s returned %d: %s", path, resp.StatusCode, utils.Truncate(string(body), 100))

// line 287 — CallEnvdUpgrade
fmt.Errorf("upgrade returned %d: %s", resp.StatusCode, utils.Truncate(string(body), 100))

// line 496 — initEnvd log field
zap.String("response_body", utils.Truncate(string(body), 100))
```

Because lines 182, 208, and 287 inject the truncated string directly into the error value, the truncation propagates all the way up the error chain and surfaces to callers as:

```
failed to create sandbox: failed to wait for sandbox start: failed to init new envd: /init returned 400: failed to mount "7e22c38a-5584-4730-8c31-449f5058ea19": failed to chroot into "/data1/orchestrator/vo
```

The message is cut mid-path, losing the actual root cause.

Impact

NFS volume mount failures are a concrete example. The full error body from envd is ~300 characters:

```
failed to mount "7e22c38a-5584-4730-8c31-449f5058ea19": failed to chroot into "/data1/orchestrator/volumes/team-f0796066-fb0b-4ea7-a9fe-613e9b8831bb/vol-885b64ff-686d-4f5c-b16d-eb4d5f9cd577": failed to bind mount "/data1/orchestrator/volumes/team-f0796066-fb0b-4ea7-a9fe-613e9b8831bb/vol-885b64ff-686d-4f5c-b16d-eb4d5f9cd577": no such file or directory
```

At 100 characters the message is truncated to:

```
failed to mount "7e22c38a-5584-4730-8c31-449f5058ea19": failed to chroot into "/data1/orchestrator/vo
```

The operator cannot tell from orchestrator logs or the returned error whether the failure is a missing directory, a permission issue, an NFS timeout, or something else. Diagnosing the root cause requires cross-referencing a completely separate service's logs (the NFS proxy).

Fix

Remove utils.Truncate from the four error-path call sites. envd error response bodies are short diagnostic strings generated by the server, not arbitrary user data, so there is no log-explosion risk.

PR: #3481

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions