Skip to content

Fix repack, OTLP export and container image issues from the release review - #81

Merged
wu-sheng merged 1 commit into
mainfrom
fix/export-repack-review
Oct 18, 2026
Merged

wu-sheng merged 1 commit into
mainfrom
fix/export-repack-review

Conversation

@wu-sheng

Copy link
Copy Markdown
Member

Fixes from the release review of 0.6.0: repack, the OTLP export, invalid UTF-8 in landed data, and the container image.

Repack

  • The idle clock. A merged file kept its first source file's collected time, and files were numbered slot by slot,
    so the newest file of a repacked session could carry a time weeks old. Retention reads that header, so it could
    remove or retire a session that was still active. Each new file now takes the time of the newest file its records
    came from, and files are numbered in the order their records landed; the source and the destination are equally
    idle.
  • The budget. Repack counted only record lines, not the header and closing line, so a file could exceed
    max_delta_bytes with ordinary records. It now cuts with sessiondata.Cut, as collectors do.
  • Order. Two files merge only when they are next to each other and their headers match, so a header that changes
    and changes back keeps its records in order.
  • Cursors. Only <prefix>.cursor was carried, so a second recorder root restarted from offset zero. Every cursor
    is carried, its last_seq mapped to the file holding its own source's last record, and a relative origin, a
    recorder inside the root, gets the origin and cursor name a collector into the destination gives it.

Invalid UTF-8

A tool input holding a byte that is not valid UTF-8 landed as it was, and the OTLP export can never send such a file:
protobuf strings must be UTF-8. Every batch holding it failed on every pass, and the file after it was skipped. Now
such data is left out at landing, its size and SHA-256 kept, the part omitted (or too_large when also nested too
deep), with a dropped entry saying why; a change or execution record is written again with U+FFFD. The export
encodes each file alone, reports a file it cannot send under its own name, and never skips the next. CLAUDE.md's
"payload bytes are the source bytes" invariant names the exception.

The OTLP client

A 2xx answer whose body did not arrive whole, or was longer than 1 MiB, was taken as success and the batch recorded
sent. It is now an error, and the batch is sent again.

The container image

The image runs as uid 65532 and declared /asz/data without creating it, so a new named or anonymous volume was
root's and the first collection failed with permission denied. The image now creates it, owned by 65532.

Checked

  • make check passes, tests/chain alone with a longer timeout after it timed out under load; the earlier JSON engine
    job (GOEXPERIMENT=nojsonv2) passes.
  • New tests cover each fix, and each guard was proven by reverting it. The image was checked with Docker 28.4.0 on a
    new named and an anonymous volume.

…eview

asz repack moved the idle clock that retention reads from the header of
the file with the highest sequence. A file it cut kept the time of the
first file its records came from, and files were numbered stream by
stream, so the highest sequence need not be the newest file. Retention
could remove or retire a session in the new root that the source root
kept. Each new file now has the header of the newest file its records came
from, and files are numbered in the order their records landed.

asz repack counted only record lines against the budget, so a file of
several records could be larger than max_delta_bytes. It now cuts with
sessiondata.Cut, header and closing line included, through the new
Cut.TakeRaw.

asz repack carried only <kind>.cursor. A second source of a stream, such
as a second recorder directory, keeps <kind>-<hash>.cursor, so collection
into the new root read it again from its start. Every cursor is carried
now. Each names the new file holding the last record its own source
landed, and an origin relative to the storage root, a recorder inside it,
is written again with its cursor's name as a collector into the new root
computes them; the rule for both moves to storage, where the change
collector and repack share it. Records of two files are cut into one only
when the files are next to each other and their headers are the same apart
from seq and at, so every file names its own source and no record moves
past another.

A data part holding a byte that is not valid UTF-8 landed as it was, and
the OTLP export could never send its file, since a protocol buffer string
must be valid UTF-8. The writer now takes such data out: the part says
omitted, or truncated while its text is left, a change or execution record
is written again with U+FFFD, and the record's dropped list says why, so
it is not taken for a payload too large to land. The export reports a file
that cannot be encoded on its own and keeps it out of every request, and a
failed request no longer leaves out of the pass the file being batched
when it failed.

The OTLP HTTP client read a 2xx answer whose body was cut off as a full
success, and an answer longer than the 1 MiB it reads as its first MiB,
and recorded every file of the request as sent. Both are errors now, and
the request goes again.

The container image declared /asz/data without making it, so Docker made a
new named or anonymous volume owned by root, and the first collection
failed with permission denied. The image now holds the directory, owned by
the user it runs as, 65532.
@wu-sheng wu-sheng added this to the 0.6.0 milestone Oct 18, 2026
@wu-sheng
wu-sheng merged commit da56fb1 into main Oct 18, 2026
24 checks passed
@wu-sheng
wu-sheng deleted the fix/export-repack-review branch October 18, 2026 13:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant