Skip to content

fix(px,shapefile): values on the DATA= line; .cst and undeclared DBF encodings (#50) - #51

Merged
ivbeg merged 1 commit into
mainfrom
claude/px-data-line-and-dbf-encoding
Nov 8, 2026
Merged

ivbeg merged 1 commit into
mainfrom
claude/px-data-line-and-dbf-encoding

Conversation

@ivbeg

@ivbeg ivbeg commented Nov 8, 2026

Copy link
Copy Markdown
Collaborator

Closes #50.

  • PX: values may start on the DATA= line, as INE writes them (DATA=70.0 68.0 ...). The first value was "DATA=70.0". The keyword is now dropped from the data text. A closing ; followed by a space or a line break no longer stays on the last value.

  • Shapefile: when no .cpg exists, the DBF encoding is taken from:

    1. the ESRI .cst code page file (Statistics Finland ships ISO-8859-1 in one);
    2. else the language driver id in DBF header byte 29 (guess_dbf_encoding: 0x57/0x03 = cp1252, 0xC9 = cp1251, 0x7B = cp932, …);
    3. else UTF-8 when the records decode as UTF-8, otherwise cp1252.

    Such files failed with dbfFileException: Could not decode field name ... b'Alaj\xe4rvi'.

Tests:

  • New: test_px_values_on_the_data_line, test_dbf_encoding_without_cpg (4 cases), test_guess_dbf_encoding.
  • Full suite: 3082 passed, 585 skipped.
  • ruff check, ruff format --check, radon cc --min D, pydocstyle, vulture: clean.
  • mypy errors in the changed files are below main (px 9 vs 15, shapefile 7 vs 13).
  • tests/test_shapefile.py keeps its CRLF line endings.

On the metacrafter-testdata files: the Finnish municipalities zip now opens (Alajärvi, Äänekoski), and the INE Madrid PX gives VALUE 70.0 instead of "DATA=70.0".

🤖 Generated with Claude Code

…encodings (#50)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ivbeg
ivbeg merged commit ec355f1 into main Nov 8, 2026
54 of 55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PX: first value keeps the DATA= prefix; Shapefile: .cst and undeclared DBF encodings

1 participant