Uploads with Expect: 100-continue stall on inspected endpoints until the client stops waiting for the 100 #5053

User Story

I'm using OpenShell as part of a sandbox that runs on mac laptops for engineers at my company to use. When an agent inside OpenShell tries to upload to S3 with the AWS CLI or boto3, every upload with a body hangs when it goes through an inspected endpoint.

Problem Statement

If a client sends Expect: 100-continue through an endpoint OpenShell inspects, the upstream's 100 Continue does not reach the client until the client has sent the body anyway. A client that waits for the 100 before sending the body stalls until its own expect timeout runs out. This happens with protocol: rest endpoints and with endpoints that leave protocol and tlsunset (automatic inspection). It does not happen with tls: skip.

From reading the code, the plain relay path in l7/rest.rs forwards Expect upstream and then relays the request body before it calls relay_response. The upstream's 100 sits unread until the body is finished. The SigV4 path and the middleware buffering path answer the 100 themselves, so they do not hit this.

For botocore, this is a hang rather than a delay. botocore waits up to one second for the socket to become readable and sends the body if it does not. Over TLS 1.3, though, the client's socket becomes readable right after the request headers go out. This is a post-handshake record with no HTTP data in it, most likely the proxy's session ticket. botocore takes that as the start of a response and blocks reading a status line that never comes. Over TLS 1.2, the socket stays quiet, botocore's one-second fallback sends the body, and the upload succeeds.

Measured through an inspected S3 endpoint on v0.1.2. A Python probe that copies botocore's logic sends the headers with Expect: 100-continue, waits up to one second for the socket to become readable, then reads:

aws s3api put-object with a 300 KB body and --cli-read-timeout 8 failed after 18.3 s by default, and succeeded in 2.6 s with OPENSSL_CONF set to MaxProtocol = TLSv1.2.

Impact / Why This Matters

• botocore sends Expect: 100-continue on every S3 PutObject that has a body. Through an inspected endpoint, aws s3api put-object and boto3 uploads wait out the read timeout, retry, and fail with "Connection was closed before we received a valid response". Conditional writes with a body (--if-match, --if-none-match) are unusable.
• curl gives up after its default one-second expect timeout and sends the body, so curl uploads work but each one gets an extra second.
• Workarounds, each with a cost:
  • tls: skip removes inspection and policy enforcement for the host.
  • The AWS CLI's CRT transfer client sends no Expect, but it covers only aws s3 cp, sync, and mv, not s3api or boto3.
  • Capping the client at TLS 1.2 (an OpenSSL config with MaxProtocol = TLSv1.2, through OPENSSL_CONF) lets botocore's fallback fire. It has to be scoped to the AWS tools so it doesn't downgrade every other client in the sandbox. Every upload still pays a one-second wait.

Acceptance Criteria

• A PUT with Expect: 100-continue through a protocol: rest endpoint completes without waiting for the client's expect timeout. In the repro below, the run with the header takes about as long as the run without it.
• The same holds for an endpoint with automatic inspection (no protocol or tls set).
• aws s3api put-object --body <file> through an inspected endpoint completes over the default TLS version.
• The client gets the interim and final responses in order, with no stray 100 Continue after the final response.

Reproduction Steps

1. Create a sandbox from an image that has curl (the default image has none; we added curl to the default image) with this policy:

version: 1
filesystem_policy:
  include_workdir: true
  read_only: [/bin, /usr, /lib, /proc, /dev/urandom, /app, /etc, /var/log]
  read_write: [/sandbox, /tmp, /dev/null]
landlock:
  compatibility: best_effort
network_policies:
  s3:
    name: s3
    endpoints:
      - host: s3.us-east-1.amazonaws.com     # L7 REST inspection
        port: 443
        protocol: rest
        enforcement: enforce
        access: full
      - host: s3.amazonaws.com               # automatic inspection
        port: 443
      - host: s3-external-1.amazonaws.com    # raw tunnel
        port: 443
        tls: skip
    binaries:
      - { path: /usr/bin/curl }

openshell sandbox create --name expect-repro --from <image-with-curl> --policy policy.yaml --no-auto-providers --detach

2. Outside the sandbox, make a presigned S3 PUT URL for each of the three hosts. Any HTTPS endpoint that answers Expect: 100-continue works.
3. In the sandbox, upload to each URL with and without the header. --expect100-timeout 20 makes the stall easy to see:

head -c 300000 /dev/urandom > /tmp/body.bin
curl -sS -o /dev/null --expect100-timeout 20 -H 'Expect: 100-continue' -T /tmp/body.bin -w '%{http_code} %{time_total}s\n' "$URL"
curl -sS -o /dev/null --expect100-timeout 20 -H 'Expect:'              -T /tmp/body.bin -w '%{http_code} %{time_total}s\n' "$URL"

Results on v0.1.6:

Suggested UX (if applicable)

No response

Environment

• OpenShell: 0.1.6
• OS: Ubuntu 24.04 (Lima VM, aarch64) on macOS 27.0
• Deployment: local gateway, Docker driver (Docker 29.9.2), image nvcr.io/nvidia/base/ubuntu:24.04 plus curl
• The AWS CLI and botocore measurements below are from our v0.1.2 deployment (AWS CLI 2.39.1). The relay code involved is unchanged in v0.1.6 and on main as of 2026-10-29.

Logs

Excerpts from curl -v with Expect: 100-continue. [...] marks omitted lines, and the bucket, key and query string are redacted. Through the automatically inspected endpoint, S3's 100 arrives only after curl stops waiting and sends the body:

> PUT /<bucket>/<key>?<redacted> HTTP/1.1
[...]
> Expect: 100-continue
[...]
* Done waiting for 100-continue
[...]
< HTTP/1.1 100 Continue
[...]
< HTTP/1.1 200 OK

Through the tls: skip endpoint, the 100 arrives right away:

> PUT /<bucket>/<key>?<redacted> HTTP/1.1
[...]
> Expect: 100-continue
[...]
< HTTP/1.1 100 Continue
[...]
< HTTP/1.1 200 OK

aws s3api put-object with a body, through an inspected endpoint (AWS CLI 2.39.1, OpenShell v0.1.2, bucket and key redacted):

aws: [ERROR]: Connection was closed before we received a valid response from endpoint URL: "https://s3.us-east-1.amazonaws.com/<bucket>/<key>".