All commit authors signed the Contributor License Agreement.

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

Change has little impact on Python users. Thanks.

This basically changes the Unicode data tables packing from a two-level to a three-level structure. The perf impact should be minimal and offset by smaller data size.

Do we even need to vendor it? It is a tool after all, we can just install it for regeneration?

What are is the size difference of the files, do you have benchmarks?

Do we even need to vendor it? It is a tool after all, we can just install it for regeneration?

Sure. I vendored it such that the artifacts can be exactly reproduced.

What are is the size difference of the files, do you have benchmarks?

The binary sizes are reported in the opening comment. sloc is net shrinkage:

% git diff main.. | diffstat
 Modules/unicodedata.c                     |   21 
 Modules/unicodedata_db.h                  |10323 +++++++++++++++++++++++++++++++-------------------------------------------------------------
 Modules/unicodename_db.h                  | 7845 ++++++++++++++++++++++++++++-----------------------------------------
 Objects/unicodectype.c                    |    7 
 Objects/unicodetype_db.h                  | 3635 +++++++++-----------------------
 Tools/unicode/makeunicodedata.py          |  178 -
 Tools/unicode/packtab/LICENSE             |  201 +
 Tools/unicode/packtab/README.md           |  184 +
 Tools/unicode/packtab/packTab/__init__.py | 1701 +++++++++++++++
 Tools/unicode/packtab/packTab/__main__.py |  270 ++
 10 files changed, 10146 insertions(+), 14219 deletions(-)

do you have benchmarks?

I'll get some benchmarks.

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

do you have benchmarks?

I'll get some benchmarks.

I added a small Python-level unicodedata.category() benchmark in Tools/unicode/benchmark_unicodedata_category.py and compared an optimized non-debug main build (-O3 -DNDEBUG) against this packtab branch.

Median results on my machine:

• Full Unicode space:
  • main: 98.98 ns/lookup
  • packtab: 108.44 ns/lookup
• BMP only:
  • main: 97.44 ns/lookup
  • packtab: 105.01 ns/lookup
• ASCII only:
  • main: 83.80 ns/lookup
  • packtab: 82.53 ns/lookup

So in this Python-level benchmark, the packtab version is slightly faster for ASCII, but slower for BMP/full-Unicode lookups.

My current hypothesis is:

• ASCII benefits from the smaller table footprint, which likely improves cache behavior.
• The broader Unicode cases pay for the extra lookup indirection: the old split-bin scheme was effectively 2 table fetches, while the current packtab layouts here are often 3 more scattered memory fetches.

That said, many real unicodedata workloads have strong codepoint locality, since Unicode scripts are generally encoded in contiguous ranges. So a uniform full-space scan is useful as a stress test, but it is not necessarily representative of typical text-processing access patterns.

So the space win is real, but at least in this benchmark it appears to trade some non-ASCII lookup speed for reduced binary size and somewhat better hot-cache behavior on tiny working sets.

So, we save ~12% in the size of the file, but slow down lookup in some cases by 10%. Additionally, we greatly increase the maintenance burden (I also see the vendored files are already two releases behind?). I'm not convinced this is worth it, it seems to be costing us more than it is saving, -0.5 on this currently.

Thanks for looking.