Repository navigation
Support Chinese numeral dates in zh locales - #1569
stellarissss wants to merge 1 commit into
Conversation
|
Convert Chinese numerals in year, month and day positions to Arabic digits before the numeric date simplifications, scoped to date contexts and applied to zh, zh-Hans and zh-Hant. Closes scrapinghub#363
e08be19 to
eaa36a7
Compare
|
Apologies for the late reply — I was away for an important exam and only saw You were right about the quadratic lookahead: with On this machine Thanks for the careful review — happy to address anything else CI or review |
Resolves #363.
Summary
Parse Chinese numeral dates such as
二〇一七年三月二十四日(2017-03-24) inthe
zh,zh-Hansandzh-Hantlocales.Changes
simplificationsrules to the three Chinese YAML sources: yearsdigit-by-digit with a bounded lookahead (up to 4 digits) requiring 年 and a
month; days 1–31 scoped to follow 月; month names 一月–十二月 scoped to
follow 年, so standalone month names keep working.
date_translation_dataPython files.tests/test_chinese_numeral_dates.py(23 cases) covering all threelocales, Arabic-numeral dates and standalone month names.
tests/test_chinese_numeral_year_linear_time.py(12 cases), usingtests.linear_time.assert_fastacross the three locales, to keepsimplification linear on long runs of digits.
The year lookahead is bounded to
{0,3}so that each starting positionchecks at most four characters, following the approach of
1164827a(GHSA-q7fp-xwcm-6x9f). Rules only rewrite numerals inside a 年/月/日 date
context, so unrelated text such as
三年前and二十四日is left untouched.This is a language-data fix with no new public API, so README.rst needs no
changes.
Verification
45 passed.
platform-specific and reproduce on an unmodified baseline).
"一" * 50000 + "年"parses in 0.10 s (0.06 s on master); longer runsand full date suffixes stay linear.
ruff check .andruff format --check .pass.