Thanks for working on this @hoodmane !

A lot of packages are failing to build with the error:

[wasm-validator error in module] unexpected true: Imported global cannot be mutable,

This comes from wasm-opt. If I remove --mvp-features and add --enable-mutable-globals the error goes away. I guess I'll try patching emscripten with this...

Were the issues at build time or at run time?

If I revert scikit-learn, scikit-image, statsmodels setup to the version on main, and then change the URL to a GitHub tag 3e28f4a they all build fine for me locally as part of this PR.

• scikit-image tests also pass
• scikit-learn and statsmodels fail to import which need more investigation.

At build time. I haven't tested whether they build locally until just recently, I was just looking at the CI. My modification to regenerate the statsmodels patches fixed the statsmodels build in CI. They fail with

error: no member named 'tp_print' in 'struct _typeobject'

in Cython-generated files. See here for example:

https://app.circleci.com/pipelines/github/hoodmane/pyodide/718/workflows/32426aa7-7c1c-4799-9d82-8b51f5bc152f/jobs/5937

If I revert scikit-learn, scikit-image, statsmodels setup to the version on main, and then change the URL to a GitHub tag 3e28f4a they all build fine for me locally as part of this PR.

Could you push this here?

On CI just now the Firefox statsmodels import test passes. Did you try to import it in Chrome or Firefox? We marked that test xfail in Chrome.

Tests for statsmodels and scikit-image also passed when installed from GH with no other changes, but only after restarting CI. So I wonder if it means there is a function signature mismatch somewhere and it fails half the times.

there is a function signature mismatch somewhere and it fails half the times.

Is that a characteristic symptom of function signature mismatch? It does seem particularly flaky here.

Well one of possible causes

Actually maybe we should apply those flags globally to all packages.

Actually maybe we should apply those flags globally to all packages.

Sounds logical to me.

a randomly selected 50% of calls will fail

Is the use of the word "random" here accurate? Or does it really mean that some consistent fraction of the calls will fail? I don't really understand the mechanism that would cause the randomness here. Seems to me that there is a chance we are just hitting on typical CI flakiness here.

Okay all tests passed. I think we should proceed with merging this.

Is the use of the word "random" here accurate? Or does it really mean that some consistent fraction of the calls will fail? I don't really understand the mechanism that would cause the randomness here. Seems to me that there is a chance we are just hitting on typical CI flakiness here.

I think such issues might explain in part the CI flakiness. It does look like under some conditions emcripten compiled code fails non deterministically, there is some number of such issues in their issue tracker. Also apparently it's a documented behavior for va_arg,

If there is no next argument, or if type is not compatible with the type of the actual
next argument (as promoted according to the default argument promotions), random errors will occur. 

under some conditions emscripten compiled code fails non deterministically

This is certainly good to know. Sad though...

random errors will occur

Right... when I read these things I usually plug my ears and scream "unpredictable but repeatable". But one thing UB is allowed to do is fail non repeatably. In the va_arg example, presumably it depends what is in the memory on the stack next to the last argument, which will vary from run to run based on who-knows-what.