Bug report
Bug description:
When switching a Python project from the default multiprocessing fork mode to forkserver, I've noticed the processes in the pool got significantly slower to start. Specifying preload modules didn't help.
Debugging internals of multiprocessing showed that most of the time (50 ms in each process in my test) is spent in _fixup_main_from_path(data['init_main_from_path']):
|
_fixup_main_from_path(data['init_main_from_path']) |
This seemed surprising because __main__ was mentioned in the preload parameter, however I've noticed that forkserver.py tries to populate the main_path parameter from spawn.get_preparation_data():
|
desired_keys = {'main_path', 'sys_path'} |
|
data = spawn.get_preparation_data('ignore') |
|
main_kws = {x: y for x, y in data.items() if x in desired_keys} |
However, the latter only writes the path to the init_main_from_path parameter:
|
d['init_main_from_path'] = os.path.normpath(main_path) |
The end effect is that the __main__ module wasn't preloaded in practice, and every child process had to re-run the main script. Unless I'm missing something, the logic in forkserver.py needs to get main_path from the value of init_main_from_path?
CPython versions tested on:
3.13
Operating systems tested on:
Linux
Linked PRs
Bug report
Bug description:
When switching a Python project from the default multiprocessing
forkmode toforkserver, I've noticed the processes in the pool got significantly slower to start. Specifyingpreloadmodules didn't help.Debugging internals of
multiprocessingshowed that most of the time (50 ms in each process in my test) is spent in_fixup_main_from_path(data['init_main_from_path']):cpython/Lib/multiprocessing/spawn.py
Line 246 in eae9d7d
This seemed surprising because
__main__was mentioned in thepreloadparameter, however I've noticed thatforkserver.pytries to populate themain_pathparameter fromspawn.get_preparation_data():cpython/Lib/multiprocessing/forkserver.py
Lines 149 to 151 in eae9d7d
However, the latter only writes the path to the
init_main_from_pathparameter:cpython/Lib/multiprocessing/spawn.py
Line 202 in eae9d7d
The end effect is that the
__main__module wasn't preloaded in practice, and every child process had to re-run the main script. Unless I'm missing something, the logic inforkserver.pyneeds to getmain_pathfrom the value ofinit_main_from_path?CPython versions tested on:
3.13
Operating systems tested on:
Linux
Linked PRs