Skip to content

Support Chinese numeral dates in zh locales - #1569

Open
stellarissss wants to merge 1 commit into
scrapinghub:masterfrom
stellarissss:fix-chinese-numeral-dates
Open

stellarissss wants to merge 1 commit into
scrapinghub:masterfrom
stellarissss:fix-chinese-numeral-dates

Conversation

@stellarissss

@stellarissss stellarissss commented Nov 7, 2026 •

Copy link
Copy Markdown

Resolves #363.

Summary

Parse Chinese numeral dates such as 二〇一七年三月二十四日 (2017-03-24) in
the zh, zh-Hans and zh-Hant locales.

Changes

  • Add simplifications rules to the three Chinese YAML sources: years
    digit-by-digit with a bounded lookahead (up to 4 digits) requiring 年 and a
    month; days 1–31 scoped to follow 月; month names 一月–十二月 scoped to
    follow 年, so standalone month names keep working.
  • Regenerate the three date_translation_data Python files.
  • Add tests/test_chinese_numeral_dates.py (23 cases) covering all three
    locales, Arabic-numeral dates and standalone month names.
  • Add tests/test_chinese_numeral_year_linear_time.py (12 cases), using
    tests.linear_time.assert_fast across the three locales, to keep
    simplification linear on long runs of digits.

The year lookahead is bounded to {0,3} so that each starting position
checks at most four characters, following the approach of 1164827a
(GHSA-q7fp-xwcm-6x9f). Rules only rewrite numerals inside a 年/月/日 date
context, so unrelated text such as 三年前 and 二十四日 is left untouched.
This is a language-data fix with no new public API, so README.rst needs no
changes.

Verification

  • New and linear-time tests: 35 passed; existing linear-time guards:
    45 passed.
  • Full suite: 24950 passed, no regressions (4 failures on Windows are
    platform-specific and reproduce on an unmodified baseline).
  • "一" * 50000 + "年" parses in 0.10 s (0.06 s on master); longer runs
    and full date suffixes stay linear.
  • ruff check . and ruff format --check . pass.
  • Generated data matches the YAML sources.

@AdrianAtZyte

Copy link
Copy Markdown
Member

The year lookahead ([…]*年) is quadratic: "一" * 50000 + "年" takes ~19 s with this change vs ~0.09 s on master.

Convert Chinese numerals in year, month and day positions to Arabic digits before the numeric date simplifications, scoped to date contexts and applied to zh, zh-Hans and zh-Hant. Closes scrapinghub#363
@stellarissss
stellarissss force-pushed the fix-chinese-numeral-dates branch from e08be19 to eaa36a7 Compare November 14, 2026 14:44
@stellarissss

Copy link
Copy Markdown
Author

Apologies for the late reply — I was away for an important exam and only saw
your comment recently.

You were right about the quadratic lookahead: with *年, every starting
position scanned to the end of the string. I've bounded it to {0,3} (a
4-digit year has at most 3 digits after the current one), following the
approach of 1164827 (GHSA-q7fp-xwcm-6x9f), and added
tests/test_chinese_numeral_year_linear_time.py using assert_fast for
zh, zh-Hans and zh-Hant.

On this machine "一" * 50000 + "年" is now 0.10 s (14 s before, 0.06 s on
master). I also checked longer runs and full date suffixes such as
年三月一日; all stay linear, and the existing linear-time guards pass.
Full suite: 24950 passed, no regressions. The PR description now reflects
this as well.

Thanks for the careful review — happy to address anything else CI or review
turns up.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

support date like this '二〇一七年三月二十四日' in zh

2 participants