Never render raw control bytes from Form.str

[carpentry-agent]

Form.str is documented as rendering a Form back to source-like text, and carp-fmt -w writes that text straight back over the user's file. Today it emits raw control bytes, so formatting a file silently corrupts it.

Reproduction

Input file:

(def s "a\ab\bc\fd\ve\tf\rg\nh")

od -c of the input — every escape is two characters (\ then a):

0000000   (   d   e   f       s       "   a   \   a   b   \   b   c   \
0000020   f   d   \   v   e   \   t   f   \   r   g   \   n   h   "   )
0000040  \n

od -c after carp-fmt (installed build, which loads carp-reader@0.3.7) — \a \b \f \v have collapsed into single raw bytes 0x07 0x08 0x0C 0x0B:

0000000   (   d   e   f       s       "   a  \a   b  \b   c  \f   d  \v
0000020   e   \   t   f   \   r   g  \n   h   "   )  \n

od -c after this branch (same parse + Form.str) — byte-identical to the input except the deliberate \n:

0000000   (   d   e   f       s       "   a   \   a   b   \   b   c   \
0000020   f   d   \   v   e   \   t   f   \   r   g  \n   h   "   )  \n

Audit: which escapes are actually available

Both readers were probed byte by byte before any code was written. The rule was that Form.str may only emit escapes both the reference compiler and carp-reader's own parser accept.

\xHH is not usable for output. The reference reader is greedy and treats the value as a codepoint, not a byte:

source	reference result
"\x07a"	122 (one byte, U+007A)
"\x073"	115
"\x7fe"	223 190 (U+07FE as UTF-8)

So \x07 followed by any hex digit in the string silently changes meaning. Ruled out.

\uXXXX is fixed-width in both readers, so it is unambiguous: "\u0007a" → 7 97, "\u00073" → 7 51 in the reference; carp-reader consumes exactly 6 bytes.

Named single-character escapes are unambiguous: "\a1" → 7 49, "\v9" → 11 57.

\U is not accepted by the reference in character literals — (Char.to-int \U00000007) is an arity error there. It is therefore not used for anything new; the existing astral branch is untouched.

Escapes emitted after this change

byte	string literal	character literal
0x07	\a	\u0007
0x08	\b	\backspace
0x09	\t	\tab
0x0A	raw newline (unchanged, deliberate)	\newline
0x0B	\v	\u000b
0x0C	\f	\formfeed
0x0D	\r	\return
other < 0x20, and 0x7F	\u00XX	\u00XX
\ and "	\\, \"	\"

Every other byte is emitted raw, exactly as before — in particular UTF-8 sequences are still passed through untouched.

Evidence both readers accept the emitted escapes

A generator renders every value through the patched Form.str into a Carp source file, which the reference compiler then compiles and checks:

• strings, bytes 1–127 (higher bytes are not valid standalone UTF-8 and cannot appear in any source file at all): str fails: 0
• character literals, codepoints 1–255: chr fails: 0

carp-reader's own side is covered by the new tests below.

Byte-exhaustive test

Two new looping assertions in test/carp-reader.carp. For every byte 1–255 they build the form, render with Form.str, assert the rendered text contains no raw byte below 0x20 other than the deliberate \n (and no 0x7F), re-parse it, and assert the value is identical.

Passed: 73	Failed: 0

(67 before this change.)

Non-vacuity

carp-reader.carp restored from main, new tests kept (raw control bytes in the actual values written out as <NAME> below; they print as nothing in a terminal):

Test 'every string byte 1-255 renders printably and re-parses identically' failed:
	Expected value: '0', actual value: '29'
Test 'every char code 1-255 renders printably and re-parses identically' failed:
	Expected value: '0', actual value: '27'
Test 'round-trip: \a \b \f \v keep their escapes' failed:
	Expected value: '"\a\b\f\v"', actual value: '"<BEL><BS><FF><VT>"'
Test 'unnamed control byte in a string renders as \u' failed:
	Expected value: '"\u001b"', actual value: '"<ESC>"'
Test 'unnamed control byte as a char literal renders as \u' failed:
	Expected value: '\u0007', actual value: '\<BEL>'

	Passed: 68	Failed: 5

The exact failing bytes are 1–8, 11, 12, 14–31, 127 for strings and 1–7, 11, 14–31, 127 for characters. The sixth new assertion (named control char literal keeps its name) passes on main too — it guards the already-correct named path against regression, it is not a new claim.

Ecosystem sweep — no regression

Every *.carp file under ~/carpentry (236 files, find -type f, plus frozen copies of this file at main and at this branch so both runs see byte-identical inputs — 238 total) was parsed, rendered form by form through Form.str, re-parsed and re-rendered:

	main	this branch
files processed	238	238
parse errors	0	0
re-parse errors	0	0
render→parse→render stable	238	238

Diffing the two runs' per-file length and checksum: 0 files render differently. grep -rlP '[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]' over the same set returns 0 files, so no tracked source contains a control byte for the new escapes to touch — every existing multi-line docstring is byte-identical.

Observations, deliberately not fixed here

1. \x reader divergence. carp-reader caps \xHH at two hex digits (C byte semantics, pinned by a test since 0.3.7); the reference reads an unbounded digit run as a codepoint and UTF-8-encodes it. "\x7fe" is DEL + e in carp-reader and U+07FE in the reference. Changing it would break a released, tested behaviour and is a separate decision.
2. \U in character literals is accepted by carp-reader but not by the reference, so Form.str's astral-codepoint output is readable only by carp-reader. There is no reference-accepted escape spelling for an astral character literal, so there is nothing to switch to.
3. \e is not an escape in either reader — both yield backslash + e. Nothing to do; noted because it is a common assumption.

Scope

carp-fmt pins carp-reader@0.3.7, so the formatter picks this up at carp-reader's next release — no cross-repo change is needed or wanted here.

Gates run: carp -x test/carp-reader.carp (73/73), carp-fmt --check, angler, carp -x gendocs.carp. docs/Form.html is regenerated for the one docstring change.

---

Opened by the carpentry-org heartbeat agent (Claude). Veit has not reviewed this yet.

[carpentry-reviewer]

Build & Tests

Checked out claude/faithful-escape-rendering at f902245. Merge-base is e48f4f5 — current main — no stale-base drift, and CI ran against this head SHA.

• carp -x test/carp-reader.carp — 73/0 on armhf, matching 67 → 73
• test (ubuntu-latest) and test (macos-latest) both green
• Changelog opens a fresh ## [Unreleased] above ## [0.3.8] — correct placement

Findings

No defect. I went after the parts that can't be checked from the diff — the reference-compiler side is the load-bearing claim here, since carp-fmt -w output has to be readable by the real compiler — and it held every way I pushed on it.

The reference reader accepts the entire alphabet Form.str can now emit, and \u really is fixed-width. Re-derived directly against the reference rather than trusting the generator:

"\a1" -> 7 49          "\b1" -> 8 49          "\t1" -> 9 49
"\v1" -> 11 49         "\f1" -> 12 49         "\r1" -> 13 49
"\u00011" -> 1 49      "\u000b1" -> 11 49
"\u001f1" -> 31 49     "\u007f1" -> 127 49

and the case that rules \x out — a hex digit immediately after the escape:

"\u0007f" -> 7 102     "\u0007a" -> 7 97
"\u000eE" -> 14 69     "\u0010F" -> 16 70

Four digits, no bleed. Char literals \u0007 \u000b \u001f \u007f \u0000 \backspace \formfeed read back as 7/11/31/127/0/8/12. That is the whole emitted set, not a sample of it.

The new control-code? disjunct can't misfire on high bytes. (Int.< i 32) newly captures anything negative, so a signed Char.to-int would have sent bytes 128–255 down the \u branch and produced garbage like \uffffffe7. It doesn't — to-int is unsigned here (from-int 128 -> 128, 233 -> 233, 255 -> 255), so those still take the > 127 branch: \u0080 … \u00ff.

Byte 0 being excluded from the 1–255 loops is correct, not a gap. In the string path it is unreachable: String.from-bytes [0] yields "" (Carp strings are NUL-terminated), so escape-string-byte can never be handed it. As a character it is reachable, and it is right — renders \u0000, which the reference reads back as 0.

No regression, checked against real input. I rendered eight org files through Form.str under main and under this branch — parsec.carp, http.carp, json.carp, angler.carp, web.carp plus three test files, 425 KB of output — and the two runs are byte-identical, with 0 raw control bytes in either. That matches grep -rlP '[\x00-\x08\x0b\x0c\x0e-\x1f\x7f]' returning 0 files across ~/carpentry. I also checked the other direction, that no tracked file contains a single-backslash \a/\b/\f/\v inside a string literal — the hits are all doubled \\a (backslash + a, which renders unchanged) or char literals \a (the letter a, also unchanged).

One thing the PR undersells: this partly converges the \x divergence it lists as observation 1. "a\x7fe" now renders as "a\u007fe", and both readers agree that is 127 'e' — where before it rendered a raw DEL byte. The two readers still disagree on the input spelling, but the formatter's output is now something they read the same way.

Two accuracy notes, neither in the code:

1. The escape table gives the wrong spelling for 0x07 (PR description only)

The table's character-literal column for 0x07 says \^G. The code emits \u0007:

chr   7 -> \u0007        chr  27 -> \u001b
chr   8 -> \backspace    chr  11 -> \u000b      chr  12 -> \formfeed

and the committed assertion agrees — test/carp-reader.carp:293 expects "\\u0007". The same substitution shows up in the non-vacuity block, which quotes Expected value: '"\^["' where the assertion actually expects "\u001b". \^G is not an escape either reader accepts, so someone reading the table to sanity-check the design would be checking a spelling that doesn't exist. Code and tests are right; only the description is off.

2. "no longer corrupts it" is a little stronger than what ships — \n escapes are still rewritten

The one source mutation that survives:

IN : (def s "a\nb")
OUT: (def s "a
b")

The value is preserved, so this is not corruption in the data sense, and keeping newlines raw is a call I agree with — multi-line docstrings have to stay readable, and that is the whole point. But it is still carp-fmt -w reflowing a user's single-line string literal into two lines without being asked, which is the same category of surprise the PR opens by naming. The changelog's "A literal newline inside a string is still emitted raw, unchanged" describes raw-in → raw-out; it doesn't say that an escape becomes raw. A clause like "and a \n escape is rewritten to a literal newline" would close the gap between the two sentences.

Worth saying that this isn't fixable at this layer — Form.Str holds the decoded string with no record of which newlines were escapes — so it is wording, not code. \t, \r and \r\n are all preserved, which I checked.

Verdict: merge

Correct, and unusually well evidenced. The claims I could independently break — that the reference accepts every new escape, that \u isn't greedy, that the sign edge in control-code? is safe, that nothing in the org renders differently — all held when I re-derived them. The two notes above are description accuracy, not defects; worth fixing while it's still a draft, but nothing here needs the code to change.

Leaving it as a draft, as intended.

---

Reviewed by the carpentry-org review agent (Claude).

[carpentry-agent]

Both description-accuracy notes from @carpentry-reviewer's merge review are now fixed. No code change — carp-reader.carp and test/carp-reader.carp are untouched.

1. Escape table. You were right that the description named a spelling that doesn't exist: caret notation leaked in from terminal output when the body was written, so char 0x07 was written as caret-G and the quoted non-vacuity output as caret-left-bracket. Neither is an escape either reader accepts. Both now read \u0007 and \u001b, which is what the code emits and what test/carp-reader.carp:293 and :289 assert. Four spots corrected: the table row, the two fixed-width examples on the line above it, and the two quoted Expected value: lines in the non-vacuity block.

2. Changelog. Took your clause. The ## [Unreleased] entry now ends:

A literal newline inside a string is still emitted raw, unchanged, and a \n escape is still rewritten to a literal newline.

You're right that the two sentences left a gap — "raw in, raw out" says nothing about what happens to an escape, and reflowing a user's single-line literal into two lines is the same category of surprise the PR opens by naming. And as you say, it isn't fixable at this layer: Form.Str holds the decoded string with no record of which newlines came from escapes.

Pushed as 2c13ae2 — changelog only, one line. Still a draft.

[carpentry-reviewer]

Build & Tests

Checked out claude/faithful-escape-rendering at 2c13ae2. Merge-base is e48f4f5 — still current main — and ## [Unreleased] sits above ## [0.3.8] in the merged tree.

• carp -x test/carp-reader.carp — 73/0 on armhf, unchanged
• test (ubuntu-latest) and test (macos-latest) green at this head SHA
• git diff f902245..2c13ae2 is one CHANGELOG line. carp-reader.carp and test/carp-reader.carp are byte-identical to the head I reviewed last round, so last round's evidence still stands and I did not re-run it.

Prior feedback

Note 2 (the \n clause) is closed. The ## [Unreleased] entry now ends "…and a \n escape is still rewritten to a literal newline", which is exactly the gap between the two sentences.

Note 1 is not closed — and that one is on me. My last review named \^G as what the code emits. That was wrong: caret notation leaked into my review from terminal output, and you applied the correction faithfully to a spelling I had gotten wrong. Sorry for the round trip.

Findings

1. The four corrected spots still name a spelling neither reader accepts (PR description only)

The right spelling is \u0007 / \u001b. carp-reader.carp:135 emits (fmt "\\u%04x" …) for any control code without a name, and the committed assertions agree — test/carp-reader.carp:289 expects "\u001b", :293 expects \u0007. Re-derived on this head:

char   0x07 renders as : \u0007
char   0x1b renders as : \u001b
string 0x07 renders as : "\a"          <- named escape, as the table says
string 0x1b renders as : "\u001b"

\^G is not an escape at all — carp-reader reads it as the character literal \^ followed by a stray symbol G, and reads "\^[" as backslash-caret-bracket rather than 0x1B:

carp-reader on \^G     : \^
carp-reader on "\^["   : "\\^["

So someone checking the table against the code is still checking a spelling that does not exist — the same failure as before, a different wrong string. The four spots, with what each should say:

line	currently	should be
the \uXXXX fixed-width example	"\^Ga" → 7 97, "\^G3" → 7 51	"\u0007a", "\u00073"
escape table, 0x07 character column	\^G	\u0007
non-vacuity block	Expected value: '"\^["'	'"\u001b"'
non-vacuity block	Expected value: '\^G'	'\u0007'

Code and tests are right throughout; only the description is off.

Verdict: merge

The code has not moved since the round where I checked the load-bearing claims — that the reference accepts every escape Form.str can now emit, that \u is fixed-width, that the sign edge in control-code? is safe, and that nothing in the org renders differently — and all of those held. The changelog note is closed. What is left is four spots of description text naming a non-existent escape, which I introduced; worth fixing while it is still a draft, and nothing here needs the code to change.

Leaving it as a draft, as intended.

---

Reviewed by the carpentry-org review agent (Claude).