Skip to content

Add UTF-8 unit tests for apteryx_set, apteryx_get, apteryx_set_tree, apteryx_get_tree - #456

Merged
sparlane merged 2 commits into
masterfrom
copilot/research-utf8-unicode-handling
Apr 23, 2026
Merged

sparlane merged 2 commits into
masterfrom
copilot/research-utf8-unicode-handling

Conversation

Copilot AI commented Apr 16, 2026

Copy link
Copy Markdown
Contributor

Adds a dedicated "UTF-8" CUnit suite (19 tests, 132 assertions) covering both valid multi-byte Unicode and known edge-case / invalid-input behaviour across all four core data APIs.

Happy-path tests (10)

Verify correct round-trip storage and retrieval — including byte-length assertions to catch silent character-to-byte collapsing:

Coverage APIs
2-byte Latin Extended (é, ö) set/get
3-byte CJK (日本語) set/get
4-byte emoji (😀) set/get
2-byte Arabic + Cyrillic set/get
Mixed widths (café 日本 😀, 17 bytes / 9 chars) set/get
Multi-byte char in path component set/get
UTF-8 leaf values via tree set_tree/get
UTF-8 values returned in tree nodes set_string/get_tree
UTF-8 node keys set_tree/get and set/get_tree

Edge-case / invalid-input tests (9)

Document two known limitations of the current implementation:

Class A – Silent NUL-byte truncation.
rpc_msg_encode_string() uses strlen(), so any embedded 0x00 terminates the data at that point; bytes after it are silently dropped.

char value[] = { 'c','a','f', 0xC3,0xA9, 0x00, 'm','o','n','d','e', 0x00 };
apteryx_set(path, value);
result = apteryx_get(path);
// strlen(result) == 5  — only "café" survived; "monde" was lost

Covers: value truncation (set/get), path truncation (write redirected to wrong key), and tree-leaf value truncation (set_tree/get).

Class B – No UTF-8 validation.
validate_path() and the RPC layer perform no byte-content inspection. Invalid sequences are accepted and returned unchanged:

Sequence Why invalid
\x80 Lone continuation byte
\xC1\x81 Overlong encoding of U+0041
\xE2\x80 Truncated 3-byte sequence (EM DASH missing 3rd byte)
\xFF Never valid in UTF-8
\xED\xA0\x80 UTF-8 encoding of surrogate U+D800
\xC3\xA9\x80 Valid 2-byte prefix + lone continuation byte (via set_tree/get_tree)

Copilot AI and others added 2 commits April 16, 2026 20:56
@sparlane
sparlane merged commit cff9a57 into master Apr 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants