Decode ACE filenames as CP850

[bitplane]

If I make an archive with café.txt under ACE32.EXE in Wine, it gets encoded using CP437 and acefile gives me a caf�.txt as it's decoding with UTF-8.

The ACE 1.2 spec says filenames are stored in the creator's OEM character set, but the format doesn't record which OEM code page was used. This makes it pretty rough to get the right filenames out.

However, Western European machines mostly used CP850 (excluding the Baltics and Portugal) while US ones used CP437, and these are quite similar where it matters. Here's a list of the differences - they're not the sort of thing you're likely to see in file names:

Byte	CP850	CP437
9B	ø	¢
9D	Ø	¥
9E	×	₧
A9	®	⌐
B5	Á	╡
B6	Â	╢
B7	À	╖
B8	©	╕
BD	¢	╜
BE	¥	╛
C6	ã	╞
C7	Ã	╟
CF	¤	╧
D0	ð	╨
D1	Ð	╤
D2	Ê	╥
D3	Ë	╙
D4	È	╘
D5	ı	╒
D6	Í	╓
D7	Î	╫
D8	Ï	╪
DD	¦	▌
DE	Ì	▐
E0	Ó	α
E2	Ô	Γ
E3	Ò	π
E4	õ	Σ
E5	Õ	σ
E7	þ	τ
E8	Þ	Φ
E9	Ú	Θ
EA	Û	Ω
EB	Ù	δ
EC	ý	∞
ED	Ý	φ
EE	¯	ε
EF	´	∩
F0	soft hyphen	≡
F2	‗	≥
F3	¾	≤
F4	¶	⌠
F5	§	⌡
F7	¸	≈
F9	¨	∙
FB	¹	√
FC	³	ⁿ

So this patch changes the default encoding from UTF-8 to CP850, and allows an override for anything else. This makes acefile Just Work with more non-ASCII filenames by default.

It does mean that Portuguese and Baltic archives get their names partly right with some characters silently mangled. The mangling is consistent with history, a class of bug that people from those areas are aware of, and unlike the replacement character at it's not lossy so can be unmangled later. Everyone else has to pass an --encoding arg.

[droe]

Thanks for the PR. Making the encoding configurable definitely makes sense. Have to think about the ramifications of changing the default.

Would you by any chance have ACE archives with non-ASCII characters in paths that you could share for testing, either for my private test corpus, or even better for the public one?

[bitplane]

Sure, here's a minimal file (had to gzip it so it'd attach)

ace32-cp437-cafe-filename.ace.gz

I made it with this version of ACE, under Wine: https://www.sac.sk/download/pack/ace26.exe

I also have this fork with a writer: https://github.com/bitplane/acefile/tree/writer

I'm using it to create test data for an archival project that I'm working on. It's missing most of the features at the moment, but it does a basic ACE 2.0 LZ77 archive. I was gonna work on it some more and send a PR, but feel free to pull/use if you think it's useful in its current state. It's good enough for me at the moment (I don't care so much about speed or compression, just that it creates an archive that works for testing)

[bitplane]

Oh also, for ACE files in the wild - I couldn't actually find any that weren't ASCII in sembiance's collection or discmaster. I just wanted to fix the UTF8 thing in my fork before using the writer as a testdata creator.

[droe]

Thanks, those collections are good additions to my private test corpus.

Merged this, but I'll change the default back to utf-8 after merging; on a green field I concur that cp850 would have been a very reasonable default for the reasons you gave. Since the best default depends on the provenance of the archive, I'm sticking to utf-8 as an equally wrong choice for everyone. That has the advantage of not violating POLA for anyone who might have done filename conversion from utf-8 to some codepage on top of the library or the CLI.

Thank you for contributing this enhancement/fix!