While investigating a D1 issue, I found that two translation units compile different implementations of kj::_::intHash32(), which causes a bug in src/workerd/api/sql.c++ statement cache.

Disclaimer: I used Claude Opus 4.6 to help me understand all the moving parts and to analyze the workerd binaries. I have checked everything it claims, but I'm not an expert in these sorts of low-level optimizations, so if anything is wrong please correct me so that I can learn along the way!

Description

This PR tries to fix these two issues:

• D1 remote vs local execute divergence workers-sdk#11708
• 🐛 BUG: Import an D1 exported DB cannot be imported in local - HashIndex detected hash table inconsistency #5977

Both issues report that executing large SQL files on Linux and Windows fails with:

$ npx wrangler d1 execute db-name --file=database.sql
# output from wrangler...

kj/table.c++:57: error: HashIndex detected hash table inconsistency. This can happen if you create a kj::Table with a hash index and you modify the rows in the table post-indexing in a way that would change their hash. This is a serious bug which will lead to undefined behavior.
stack: ; kj::getStackTrace() = 7ff6490d0fb7 7ff649ab56f1 7ff649aafb57 7ff649ab3e1e 7ff6491c488b 7ff648ddd9e0 7ff648ddd70f 7ff648ddd4c3 7ff64aa403b9 7ff6aaa87b74 3df00000010 3df00cda16c 9 3df00020000 3df0c09b9a8

However, the same command executes successfully on macOS ARM.

Investigation

After some testing, I found that the error only appears for SQL files with over 1 MB worth of SQL statements. This led me to the SQL_STATEMENT_CACHE_MAX_SIZE constant

Specifically, it is the call to statementCache.map.eraseMatch(oldQuery) that produces the "HashIndex detected hash table inconsistency" error.

The SQL statement cache uses a kj::Table with a kj::HashIndex

1. Insertion via findOrCreate(JsString) calls JsString::hashCode() (defined in jsvalue.c++)
   

   workerd/src/workerd/api/sql.c++

   Lines 44 to 48 in 244803a

   	kj::Rc<CachedStatement>& slot = statementCache.map.findOrCreate(querySql, [&]() {
   	auto result = kj::rc<CachedStatement>(js, *this, db, querySql, js.toString(querySql));
   	statementCache.totalSize += result->statementSize;
   	return result;
   	});

2. Eviction uses HashableV8Ref::hashCode() cached in the CachedStatement constructor (defined in sql.c++)
   

   workerd/src/workerd/api/sql.c++

   Lines 87 to 90 in 244803a

   	auto oldQuery = jsg::JsString(toRemove.query.getHandle(js));
   	statementCache.totalSize -= toRemove.statementSize;
   	statementCache.lru.remove(toRemove);
   	KJ_ASSERT(statementCache.map.eraseMatch(oldQuery));

The problem is a One Definition Rule violation: kj::_::intHash32() is an inline function in kj/hash.h that compiles to different implementations depending on whether CRC32 compiler flags are set:

• sql.c++ is compiled as part of //src/workerd/io, which has -mcrc (ARM) / -msse4.2 (x86) copts, so intHash32() uses hardware CRC32.
• jsvalue.c++ is compiled as part of //src/workerd/jsg:jsg-core, which has no CRC flags, so intHash32() uses the Thomas Wang software fallback.

Both functions take the same input and produce different outputs. Entries are inserted into one bucket (Thomas Wang hash) but looked up during eviction from a different bucket (CRC32 hash).
Thanks to the amazing logging in the kj::Table, the kj::HashIndex detects the mismatch and logs the "hash table inconsistency" error.

I'm not entirely sure, but I think that macOS ARM is unaffected because the default compiler flags already enable __ARM_FEATURE_CRC32 for all targets.

I had Claude disassemble the relevant functions in the workerd binary distributed with the wrangler package, and it confirmed the mismatch in the Linux x86_64 binary:

JsString::hashCode() at 0x31d1860 -- Thomas Wang software fallback:

31d1886: imull  $0x809, %ecx, %ecx    # Thomas Wang constant (2057)

CachedStatement constructor at 0x2cae690 -- CRC32 hardware instruction:

2cae6f8: crc32l  %eax, %edx            # SSE4.2 CRC32 instruction
2cae6fd: movl    %edx, 0x30(%rbx)      # stored as cached identityHash

eraseImpl at 0x2cafb60 loads the cached CRC32 hash and has an explicit call to logHashTableInconsistency at 0x2cafbcc -- confirming this is the crash path.

I believe the compilation flag mismatch was introduced in c7ad54b. It seems the copts were intended for crypto/crc-impl.c++, but that file is compiled under a separate bazel target (//src/workerd/api:crypto-crc-impl) that never received them. The flags ended up applying to sql.c++ and every other file in //src/workerd/io.

Changes

I'm not sure how to best fix the problem. This PR removes the CRC32 compilation options from //src/workerd/io and adds them to the //src/workerd/api:crypto-crc-impl target. However, this means that the CRC32 optimization isn't used anymore in //src/workerd/io, and that's kind of sad.

We could also remove the call to kj::hashCode in the identityHash implementation, but Kenton wrote that it's not clear if V8's GetIdentityHash() returns uniform results.

What's missing from this PR / Questions

1. Is there a way to create a test?
   I wanted to create a test to reproduce this, but the hash table inconsistency is only a log and doesn't crash or throw.

2. What would be the best fix for this problem?
   I'm not used to the codebase, so the change introduced in this PR is certainly not the best one. It does solve the problem, but it would be great to solve the problem and still use the crc32 compilation option for all the codebase.