Repository navigation
Fix repack, OTLP export and container image issues from the release review - #81
Merged
Merged
Conversation
…eview asz repack moved the idle clock that retention reads from the header of the file with the highest sequence. A file it cut kept the time of the first file its records came from, and files were numbered stream by stream, so the highest sequence need not be the newest file. Retention could remove or retire a session in the new root that the source root kept. Each new file now has the header of the newest file its records came from, and files are numbered in the order their records landed. asz repack counted only record lines against the budget, so a file of several records could be larger than max_delta_bytes. It now cuts with sessiondata.Cut, header and closing line included, through the new Cut.TakeRaw. asz repack carried only <kind>.cursor. A second source of a stream, such as a second recorder directory, keeps <kind>-<hash>.cursor, so collection into the new root read it again from its start. Every cursor is carried now. Each names the new file holding the last record its own source landed, and an origin relative to the storage root, a recorder inside it, is written again with its cursor's name as a collector into the new root computes them; the rule for both moves to storage, where the change collector and repack share it. Records of two files are cut into one only when the files are next to each other and their headers are the same apart from seq and at, so every file names its own source and no record moves past another. A data part holding a byte that is not valid UTF-8 landed as it was, and the OTLP export could never send its file, since a protocol buffer string must be valid UTF-8. The writer now takes such data out: the part says omitted, or truncated while its text is left, a change or execution record is written again with U+FFFD, and the record's dropped list says why, so it is not taken for a payload too large to land. The export reports a file that cannot be encoded on its own and keeps it out of every request, and a failed request no longer leaves out of the pass the file being batched when it failed. The OTLP HTTP client read a 2xx answer whose body was cut off as a full success, and an answer longer than the 1 MiB it reads as its first MiB, and recorded every file of the request as sent. Both are errors now, and the request goes again. The container image declared /asz/data without making it, so Docker made a new named or anonymous volume owned by root, and the first collection failed with permission denied. The image now holds the directory, owned by the user it runs as, 65532.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes from the release review of 0.6.0: repack, the OTLP export, invalid UTF-8 in landed data, and the container image.
Repack
so the newest file of a repacked session could carry a time weeks old. Retention reads that header, so it could
remove or retire a session that was still active. Each new file now takes the time of the newest file its records
came from, and files are numbered in the order their records landed; the source and the destination are equally
idle.
max_delta_byteswith ordinary records. It now cuts withsessiondata.Cut, as collectors do.and changes back keeps its records in order.
<prefix>.cursorwas carried, so a second recorder root restarted from offset zero. Every cursoris carried, its
last_seqmapped to the file holding its own source's last record, and a relative origin, arecorder inside the root, gets the origin and cursor name a collector into the destination gives it.
Invalid UTF-8
A tool input holding a byte that is not valid UTF-8 landed as it was, and the OTLP export can never send such a file:
protobuf strings must be UTF-8. Every batch holding it failed on every pass, and the file after it was skipped. Now
such data is left out at landing, its size and SHA-256 kept, the part
omitted(ortoo_largewhen also nested toodeep), with a
droppedentry saying why; a change or execution record is written again with U+FFFD. The exportencodes each file alone, reports a file it cannot send under its own name, and never skips the next. CLAUDE.md's
"payload bytes are the source bytes" invariant names the exception.
The OTLP client
A 2xx answer whose body did not arrive whole, or was longer than 1 MiB, was taken as success and the batch recorded
sent. It is now an error, and the batch is sent again.
The container image
The image runs as uid 65532 and declared
/asz/datawithout creating it, so a new named or anonymous volume wasroot's and the first collection failed with
permission denied. The image now creates it, owned by 65532.Checked
make checkpasses,tests/chainalone with a longer timeout after it timed out under load; the earlier JSON enginejob (
GOEXPERIMENT=nojsonv2) passes.new named and an anonymous volume.