ingest: stop routing plain .md/.txt through MIME extraction (fixes #1)

[owrede]

Fixes #1.

Root cause

_preprocess_file routes every discovered file through _extract_email_text. Anything that isn't a markdown-exported email falls into the raw-MIME branch, where compat32 get_payload(decode=True) on a str payload re-encodes via raw-unicode-escape and then decodes with utf-8, errors="ignore". Net effect on plain text:

• every char in U+0080–U+00FF is silently deleted: Köln → Kln, Straße → Strae
• every char above U+00FF becomes a literal \uXXXX sequence in the indexed text

For any non-English corpus this silently corrupts the entire index — the source files are untouched, so it's easy to miss.

Fix

1. Gate MIME extraction behind a new _looks_like_email(path, raw) helper: only .eml files, markdown-exported emails (# Email: / **From:**), or files opening with RFC-5322 headers go through _extract_email_text; everything else is ingested verbatim.
2. Read files as explicit UTF-8 (read_text(encoding="utf-8", errors="ignore")) instead of the locale default.

Downstream behavior is unchanged: is_email still derives from email_meta keys, and the non-email branch returns {"source": str(path)} — truthy, same as _extract_email_text's fallback, so email_meta or auto_extractor.extract(...) resolves identically.

Tests

• New tests/test_ingest_unicode.py: gating unit tests + an end-to-end regression test that runs _preprocess_file on a .md containing umlauts/CJK/emoji and asserts the decoded token stream still contains them.
• All existing tests in tests/test_email_calendar_ingestion.py pass.

Suggested follow-up (not in this PR)

The gate removes the trigger, but the lossy round-trip inside _extract_email_text remains reachable for real 8bit-encoded .eml files. Root fix: parse real emails from bytes — email.message_from_bytes(raw.encode("utf-8"), policy=email.policy.default) — so get_payload(decode=True) returns properly CTE-decoded bytes. Happy to send that as a second PR if wanted.

🤖 Generated with Claude Code