Ok, so it seems that shared library "packages" no longer get loaded since there's no Python code to trigger the dynlib loading just in time.

@ryanking13 Are shared library "packages" guaranteed to only consist of a single shared library, or can they include other files (especially several shared libraries)? If it's just a single file then loading them wouldn't take much code effort

The other error CI has found is that Chrome limits the size of WASM modules that it compiles synchronously. Since we no longer preload modules, but only (synchronously) load them on demand, some larger dynlibs fail to load on Chrome.

An important aspect of lazy loading is to not overload with shared libs (scipy is a prime suspect) that are not needed by the user and only load them on demand, when the user actually uses some submodule that depends on it.

One could of course keep all existing code but be clever and e.g. preload large shared libraries but not small ones, but that would feel iffy.

I wonder if it would also be possible to patch Emscripten to try to use async compilation with JSPI in its sync path if available, so that even a sync load triggered by Python could make use of the async pathway.

In any case, this isn't an issue if Pyodide is run in a web worker.

So I wonder if we could maybe just expose a preload option, true by default, with which users could opt out of preloading if they want?

The other error CI has found is that Chrome limits the size of WASM modules that it compiles synchronously. Since we no longer preload modules, but only (synchronously) load them on demand, some larger dynlibs fail to load on Chrome.

Yes, we still need to preload shared libraries, as import ... will try to synchronously compile (dlopen) the shared library, but browsers does not support compiling webassembly modules that are larger than 4KB. The limit is different per browser, but all the browsers have the limiation somehow so we need to do it asynchronously.

@ryanking13 Are shared library "packages" guaranteed to only consist of a single shared library, or can they include other files (especially several shared libraries)? If it's just a single file then loading them wouldn't take much code effort

It can contain multiple shared libraries. For example openssl contains both libssl.so and libcrypto.so.

But that limit only applies on the main thread, right? So on a web worker we can rely on synchronous compilation?

From my experiments, a top-level switch also seems quite heavy-handed. In the end, we want to preload shared libraries that are loaded immediately anyways (since the package imports them immediately) but lazy load those that require user interaction. Guessing that with some heuristic seems hard.

So maybe it could be a hint in the package recipe that is only taken into account when the global option is enabled? That way we would get the big savings from e.g. scipy without penalising packages that just have ~1 extension module

Cc #5264

I wonder if it would also be possible to patch Emscripten to try to use async compilation with JSPI in its sync path if available, so that even a sync load triggered by Python could make use of the async pathway.
In any case, this isn't an issue if Pyodide is run in a web worker.
So I wonder if we could maybe just expose a preload option, true by default, with which users could opt out of preloading if they want?

Yes, we could adopt JSPI and experiment with it. I believe it is technically possible, but it would require decent amount of changes and experiment. Overall it would require the following things:

1. Patching the dlopen implementation in Emscripten using JSPI.
2. Implement a reliable way to select the right code path when JSPI is available or not.
3. Benchmark the performance and measure the overhead of enabling JSPI.

But that limit only applies on the main thread, right? So on a web worker we can rely on synchronous compilation?

Right. I think synchronous compilation will simply work in the worker.

So maybe it could be a hint in the package recipe that is only taken into account when the global option is enabled? That way we would get the big savings from e.g. scipy without penalising packages that just have ~1 extension module

I don't think it is a good idea to put it in the recipe. I don't want to add any more information into the recipe as our goal is to get rid of the lockflie and recipes if possible.

Though, yes I think we can try disabling the preloading in the worker and see if it works well. It might not need many changes I guess.