Skip to content

dict_traverse crash during PGO training on Windows with JIT enabled #144970

@savannahostrowski

Description

@savannahostrowski

Bug report

Bug description:

Building CPython at e779777 with PCbuild/build.bat -c Release --pgo --experimental-jit can crash during the PGO training phase. The instrumented interpreter hits an access violation in dict_traverse → visit_decref → _PyObject_IS_GC with ob = 0xffffffffffffffff (-1). The last successful build was at 2f7634c (the prior night's run; I suspect this has to do with #144757).

I caught this as part of the nightly run for the JIT benchmark website (see https://github.com/savannahostrowski/pyperf_bench/actions/runs/22133822860/job/64045676082) on my Windows runner (x86_64, Windows 11). Reproduces under a Windows service account (NETWORK SERVICE); appears to be timing-dependent.

Details 0:000> !analyze -v ******************************************************************************* * * * Exception Analysis * * * *******************************************************************************

KEY_VALUES_STRING: 1

Key  : AV.Type
Value: Read

Key  : Analysis.CPU.mSec
Value: 375

Key  : Analysis.Elapsed.mSec
Value: 577

Key  : Analysis.IO.Other.Mb
Value: 0

Key  : Analysis.IO.Read.Mb
Value: 20

Key  : Analysis.IO.Write.Mb
Value: 56

Key  : Analysis.Init.CPU.mSec
Value: 1718

Key  : Analysis.Init.Elapsed.mSec
Value: 36703

Key  : Analysis.Memory.CommitPeak.Mb
Value: 111

Key  : Analysis.Version.DbgEng
Value: 10.0.29507.1001

Key  : Analysis.Version.Description
Value: 10.2511.5.1 amd64fre

Key  : Analysis.Version.Ext
Value: 1.2511.5.1

Key  : Failure.Bucket
Value: INVALID_POINTER_READ_c0000005_python315.dll!_PyObject_IS_GC

Key  : Failure.Exception.Code
Value: 0xc0000005

Key  : Failure.Exception.IP.Address
Value: 0x7ffc5f3395a0

Key  : Failure.Exception.IP.Module
Value: python315

Key  : Failure.Exception.IP.Offset
Value: 0x395a0

Key  : Failure.Hash
Value: {41ad561c-4f71-20b0-5dcb-a02ca0bc2c99}

Key  : Failure.ProblemClass.Primary
Value: INVALID_POINTER_READ

Key  : Faulting.IP.Type
Value: Paged

Key  : Timeline.OS.Boot.DeltaSec
Value: 405382

Key  : Timeline.Process.Start.DeltaSec
Value: 3

Key  : WER.OS.Branch
Value: ge_release

Key  : WER.OS.Version
Value: 10.0.26100.1

Key  : WER.Process.Version
Value: 3.15.106.1013

FILE_IN_CAB: python.exe.3948.dmp

NTGLOBALFLAG: 0

APPLICATION_VERIFIER_FLAGS: 0

CONTEXT: (.ecxr)
rax=000001b2fdf20130 rbx=ffffffffffffffff rcx=ffffffffffffffff
rdx=000001b2fdf08c00 rsi=0000000000000027 rdi=000001b2fdf08c00
rip=00007ffc5f3395a0 rsp=0000000dd217c490 rbp=000001b2fdf08c00
r8=00007ffc5fbd1a80 r9=ffffffffffffffff r10=00007ffcd2895980
r11=00007ffcd2893870 r12=000001b2fa194d00 r13=00007ffc5fcb4c80
r14=00007ffc5f79c9c0 r15=00007ffc5fce9db8
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206
python315!_Py_TYPE_impl [inlined in python315!_PyObject_IS_GC+0x20]:
00007ffc5f3395a0 4c8b4108 mov r8,qword ptr [rcx+8] ds:0000000000000007=????????????????
Resetting default scope

EXCEPTION_RECORD: (.exr -1)
ExceptionAddress: 00007ffc5f3395a0 (python315!_Py_TYPE_impl)
ExceptionCode: c0000005 (Access violation)
ExceptionFlags: 0000000
NumberParameters: 2
Parameter[0]: 0000000000000000
Parameter[1]: 0000000000000007
Attempt to read from address 0000000000000007

PROCESS_NAME: python.exe

READ_ADDRESS: 0000000000000007

ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.

EXCEPTION_CODE_STR: c0000005

EXCEPTION_PARAMETER1: 0000000000000000

EXCEPTION_PARAMETER2: 0000000000000007

STACK_TEXT:
(Inline Function) ---------------- : ---------------- ---------------- ---------------- ---------------- : python315!_Py_TYPE_impl 0000000dd217c490 00007ffc5f79c9f5 : 00007ffc5fc67980 00007ffc5fbd1a80 000001b2fdf08c00 ffffffffffffffff : python315!_PyObject_IS_GC+0x20 0000000dd217c4d0 00007ffc5f683fcb : 0000000000000003 000000000000000 000000000000000 000000000000000 : python315!visit_decref+0x35 0000000dd217c500 00007ffc5f36f42c : 000001b2fdf08bf 0000000dd217c708 0000000dd217c6f8 0000000dd217c6f8 : python315!dict_traverse+0xab 0000000dd217c570 00007ffc5f389fd5 : 000000000000085f 0000000dd217c648 0000000dd217c788 0000000dd217c708 : python315!subtract_refs+0x7c 0000000dd217c5c0 00007ffc5f443b38 : 0000000dd217c6f8 00007ffc5fcb2fa0 00007ffc5fce9db8 000000000000000 : python315!deduce_unreachable+0x65 0000000dd217c600 00007ffc5f44de15 : 00007ffc5fcb4c80 00007ffc5fce9db8 000000000000000 00007ffc5fcb4c80 : python315!gc_collect_region+0x78 0000000dd217c6c0 00007ffc5f44f0e6 : 000000000000000 000000000000000 00007ffc5fcb4c80 000000000000000 : python315!gc_collect_increment+0x285 0000000dd217c760 00007ffc5f44fd6a : 000000000000000 00007ffc5fce9db8 000000000000000 000170b127ba8994 : python315!_PyGC_Collect+0x1a6 0000000dd217c7f0 00007ffc5f4500c0 : 000000000000000 000000000000000 000000000000000 000001b2f94b7c20 : python315!_Py_RunGC+0x4a 0000000dd217c830 00007ffc5f45028e : 000001b2fe56ec51 000001b2fe9d9100 000001b2fdf20951 000000000000002 : python315!_Py_HandlePending+0x140 0000000dd217c870 00007ffc5f766de9 : 000001b2fe56ec51 000001b2fdf0c7b0 0000000000000008 000000000000000 : python315!check_periodics+0x3e 0000000dd217c8b0 00007ffc5f4504e5 : 000000001ab78c8b 000000000000000 0000000dd217cbf0 000000000000002 : python315!_PyEval_EvalFrameDefault+0x68cd 0000000dd217cb4 00007ffc5f450891 : 0000000dd217cbf0 000001b2f94e9f90 000000000000000 0000000dd217cd68 : python315!_PyEval_EvalFrame+0x35 0000000dd217cb80 00007ffc5f673e28 : 000000000000000 000000000000000 000000000000000 00007ffc5fce9db8 : python315!_PyEval_Vector+0x291 0000000dd217cc80 00007ffc5f4ae45e : 000001b2fdeffcc0 000000000000002 000000000000000 000000000000000 : python315!_PyFunction_Vectorcall+0x88 0000000dd217ccc0 00007ffc5f3ea92b : 0000000000000002 0000000dd217ce30 000000000000000 0000000dd217cd68 : python315!_PyObject_VectorcallTstate+0xbe 0000000dd217cd10 00007ffc5f673cf9 : 000001b2fde7bf40 000000000000000 000000000000000 000000000000000 : python315!object_vacall+0x1ab 0000000dd217cdd0 00007ffc5f441d83 : 000000000000000 00007ffc5fca7760 00007ffc5fcaec48 000001b2fdf20630 : python315!PyObject_CallMethodObjArgs+0x109 0000000dd217ce20 00007ffc5f7a5ce4 : 000000000000000 000000000000000 000000000000000 00007ffc5fc99478 : python315!import_find_and_load+0xf3 0000000dd217ceb0 00007ffc5f4429dd : 00007ffc5fcb2fa0 000001b2fe53e1c0 000001b2fe53e1c0 00000001b11c0c7 : python315!PyImport_ImportModuleLevelObject+0x384 0000000dd217cf20 00007ffc5f442b4d : 00007ffc5fcaec48 000000000000000 000000000000000 000000000000000 : python315!_PyEval_ImportNameWithImport+0xed 0000000dd217cfd0 00007ffc5f448104 : 00007ffc5fcaec48 000000000000000 00007ffc5fce9db8 000001b2fe53e1c0 : python315!_PyEval_ImportName+0xed 0000000dd217d030 00007ffc5f763fc1 : 000001b2fde7b640 00007ffc5fce9db8 00007ffc5fc99400 000001b2fe451f90 : python315!_PyEval_LazyImportName+0x124 0000000dd217d120 00007ffc5f4504e5 : 000000001b16c513 000000000000000 0000000dd217d460 000000000000000 : python315!_PyEval_EvalFrameDefault+0x3aa5 0000000dd217d3b0 00007ffc5f450891 : 0000000dd217d460 000001b2fe27f420 000000000000000 000000000000000 : python315!_PyEval_EvalFrame+0x35 0000000dd217d3f0 00007ffc5f450a79 : 000001b2fe0ef9b0 000001b2fe53e1c0 000001b2fe53e1c0 000001b2fea71170 : python315!_PyEval_Vector+0x291 0000000dd217d4f0 00007ffc5f7591a9 : 000001b2fe53e1c0 000000000000000 000001b2fe53e1c0 00000000000000f : python315!PyEval_EvalCode+0x199 0000000dd217d590 00007ffc5f758dc0 : 000000000000000 000000000000000 000001b2f95e0871 00007ffc5f8b1946 : python315!builtin_exec_impl+0x3a9 0000000dd217d640 00007ffc5f6b00f5 : 0000000000000002 000000000000000 000000000000000 000000000000002 : python315!builtin_exec+0x150 0000000dd217d6f 00007ffc5f3e37c8 : 00007ffc5f6b0030 000001b2fdf20ae0 0000000000000002 00007ffc5f3681c1 : python315!cfunction_vectorcall_FASTCALL_KEYWORDS+0xc5 0000000dd217d730 00007ffc5f495c61 : 000001b2fdf20ae0 000001b2fe9d94c0 000001b2fdf20ae1 000000000000000 : python315!_PyVectorcall_Call+0x148 0000000dd217d790 00007ffc5f496b6c : 0000000000000010 000000000000000 000001b2fdf20ae1 000000000000002 : python315!_PyObject_Call+0x81 0000000dd217d7f0 00007ffc5f766d33 : 000001b2fea540c1 000001b2fdf0c7b0 0000000000000008 000000000000000 : python315!PyObject_Call+0x5c 0000000dd217d830 00007ffc5f4504e5 : 000000001ab78c8b 000001b2fe5700c0 0000000dd217db70 000000000000002 : python315!_PyEval_EvalFrameDefault+0x6817 0000000dd217dac0 00007ffc5f450891 : 0000000dd217db70 000001b2f94e9f90 000000000000000 0000000dd217dce8 : python315!_PyEval_EvalFrame+0x35 0000000dd217db00 00007ffc5f673e28 : 000001b2fdf6f480 000001b2fe5700c0 000000000000000 00007ffc5fce9db8 : python315!_PyEval_Vector+0x291 0000000dd217dc00 00007ffc5f4ae45e : 000001b2fdeffcc0 000000000000002 000001b2fdf6f480 000001b2fe5700c0 : python315!_PyFunction_Vectorcall+0x88 0000000dd217dc40 00007ffc5f3ea92b : 0000000000000002 0000000dd217ddb0 000000000000000 0000000dd217dce8 : python315!_PyObject_VectorcallTstate+0xbe 0000000dd217dc90 00007ffc5f673cf9 : 000001b2fde7bf40 000000000000000 000000000000000 000000000000000 : python315!object_vacall+0x1ab 0000000dd217dd50 00007ffc5f441d83 : 000001b2fe5538b0 00007ffc5fca7760 000001b2fe0ef9b0 000001b2fdf20630 : python315!PyObject_CallMethodObjArgs+0x109 0000000dd217dda0 00007ffc5f7a5ce4 : 000000000000000 000000000000000 000000000000000 00007ffc5fc99478 : python315!import_find_and_load+0xf3 0000000dd217de3 00007ffc5f4429dd : 00007ffc5fcb2fa0 000001b2fe1c5dc0 000001b2fe1c5dc0 00000001b11c0c7 : python315!PyImport_ImportModuleLevelObject+0x384 0000000dd217dea0 00007ffc5f442b4d : 000001b2fe0ef9b0 000001b2fea70a90 000001b2fe5538b0 000001b2fea70bf0 : python315!_PyEval_ImportNameWithImport+0xed 0000000dd217df50 00007ffc5f448104 : 000001b2fe0ef9b0 000000000000000 00007ffc5fce9db8 000001b2fe1c5dc0 : python315!_PyEval_ImportName+0xed 0000000dd217dfb0 00007ffc5f763fc1 : 000001b2fde7b640 00007ffc5fce9db8 00007ffc5fc99400 000001b2f951d670 : python315!_PyEval_LazyImportName+0x124 0000000dd217e0a0 00007ffc5f4504e5 : 000000001b16c513 000000000000000 0000000dd217e3e0 000000000000000 : python315!_PyEval_EvalFrameDefault+0x3aa5 0000000dd217e330 00007ffc5f450891 : 0000000dd217e3e0 000001b2fe1dab40 000000000000000 000000000000000 : python315!_PyEval_EvalFrame+0x35 0000000dd217e370 00007ffc5f450a79 : 000001b2fe1c5470 000001b2fe1c5dc0 000001b2fe1c5dc0 000001b2fe4d2770 : python315!_PyEval_Vector+0x291 0000000dd217e47 00007ffc5f7591a9 : 000001b2fe1c5dc0 000000000000000 000001b2fe1c5dc0 00000000000000f : python315!PyEval_EvalCode+0x199 0000000dd217e510 00007ffc5f758dc0 : 000000000000000 000000000000000 000001b2f95e0871 00007ffc5f8b1946 : python315!builtin_exec_impl+0x3a9 0000000dd217e5c0 00007ffc5f6b00f5 : 0000000000000002 000000000000000 000000000000000 000000000000002 : python315!builtin_exec+0x150 0000000dd217e670 00007ffc5f3e37c8 : 00007ffc5f6b0030 000001b2fdf20ae0 0000000000000002 00007ffc5f3681c1 : python315!cfunction_vectorcall_FASTCALL_KEYWORDS+0xc5 0000000dd217e6b0 00007ffc5f495c61 : 000001b2fdf20ae0 000001b2fe5330c0 000001b2fdf20ae1 000000000000000 : python315!_PyVectorcall_Call+0x148 0000000dd217e710 00007ffc5f496b6c : 0000000000000010 000000000000000 000001b2fdf20ae1 000000000000002 : python315!_PyObject_Call+0x81 0000000dd217e770 00007ffc5f766d33 : 000001b2fe528201 000001b2fdf0c7b0 0000000000000008 000000000000000 : python315!PyObject_Call+0x5c 0000000dd217e7b0 00007ffc5f4504e5 : 000000001ab78c8b 5f5f6e69616d5f5f 0000000dd217eaf0 000000000000002 : python315!_PyEval_EvalFrameDefault+0x6817 0000000dd217ea40 00007ffc5f450891 : 0000000dd217eaf0 000001b2f94e9f90 000000000000000 0000000dd217ec68 : python315!_PyEval_EvalFrame+0x35 0000000dd217ea80 00007ffc5f673e28 : 5c5f5f6568636163 5f5f6e69616d5f5f 000000000000000 00007ffc5fce9db8 : python315!_PyEval_Vector+0x291 0000000dd217eb8 00007ffc5f4ae45e : 000001b2fdeffcc0 000000000000002 5c5f5f6568636163 5f5f6e69616d5f5f : python315!_PyFunction_Vectorcall+0x88 0000000dd217ebc0 00007ffc5f3ea92b : 0000000000000002 0000000dd217ed30 000000000000000 0000000dd217ec68 : python315!_PyObject_VectorcallTstate+0xbe 0000000dd217ec10 00007ffc5f673cf9 : 000001b2fde7bf40 000000000000000 000000000000000 000000000000000 : python315!object_vacall+0x1ab 0000000dd217ecd0 00007ffc5f441d83 : 000000000000000 00007ffc5fca7760 000001b2fe1c5470 000001b2fdf20630 : python315!PyObject_CallMethodObjArgs+0x109 0000000dd217ed20 00007ffc5f7a5ce4 : 000000000000000 000000000000000 000000000000000 00007ffc5fc99478 : python315!import_find_and_load+0xf3 0000000dd217edb0 00007ffc5f4429dd : 00007ffc5fcb2fa0 000001b2fe15e840 000001b2fe15e840 00000001b11c0c7 : python315!PyImport_ImportModuleLevelObject+0x384 0000000dd217ee20 00007ffc5f442b4d : 000001b2fe1c5470 000000000000000 000000000000000 000000000000000 : python315!_PyEval_ImportNameWithImport+0xed 0000000dd217eed0 00007ffc5f448104 : 000001b2fe1c5470 000000000000000 00007ffc5fce9db8 000001b2fe15e840 : python315!_PyEval_ImportName+0xed 0000000dd217ef30 00007ffc5f763fc1 : 000001b2f9620228 000001b2fdea86a0 0000000dd217f130 00007ffc5fce9db8 : python315!_PyEval_LazyImportName+0x124 0000000dd217f020 00007ffc5f4504e5 : 000000001b16c513 5f5f6e69616d5f5f 0000000dd217f360 000000000000000 : python315!_PyEval_EvalFrameDefault+0x3aa5 0000000dd217f2b0 00007ffc5f450891 : 0000000dd217f360 000001b2fe151730 000000000000000 000000000000000 : python315!_PyEval_EvalFrame+0x35 0000000dd217f2f0 00007ffc5f450a79 : 00007ffc5fca6408 000001b2fe15e840 000001b2fe15e840 000001b2fe13b320 : python315!_PyEval_Vector+0x291 0000000dd217f3f0 00007ffc5f7591a9 : 000001b2fe15e840 000000000000000 000001b2fe15e840 000001b2fe113580 : python315!PyEval_EvalCode+0x199 0000000dd217f490 00007ffc5f758dc0 : 000000000000000 000000000000000 000001b2f95e0871 00007ffc5f8b1946 : python315!builtin_exec_impl+0x3a9 0000000dd217f540 00007ffc5f6b00f5 : 0000000000000002 5f5f6e69616d5f5f 000000000000000 8000000000000002 : python315!builtin_exec+0x150 0000000dd217f5f0 00007ffc5f4ae45e : 000001b2fdf20ae0 00007ffc5fce9db8 0000000dd217f738 5f5f6e69616d5f5f : python315!cfunction_vectorcall_FASTCALL_KEYWORDS+0xc5 0000000dd217f630 00007ffc5f3eaa96 : 0000000000000005 000001b2fdf20ae0 000001b2f9620190 000000000000010 : python315!_PyObject_VectorcallTstate+0xbe 0000000dd217f680 00007ffc5f3f136f : 0000000000000001 000001b2fdf20ae0 000000000000000 000000000000000 : python315!PyObject_Vectorcall+0x56 0000000dd217f6d0 00007ffc5f769b92 : 000001b2fe15e841 000001b2f9620180 000000000000000 000001b2fde9d3b4 : python315!_Py_VectorCallInstrumentation_StackRefSteal+0xef 0000000dd217f7f 00007ffc5f4504e5 : 000000001ab78c8b 000001b2fe4d2350 0000000dd217fb30 000000000000002 : python315!_PyEval_EvalFrameDefault+0x9676 0000000dd217fa80 00007ffc5f450891 : 0000000dd217fb30 000001b2fe3af7f0 000000000000000 000001b2`fe1bba20 : python315!_PyEval_EvalFrame+0x35

STACK_COMMAND: ~0s; .ecxr ; kb

IP_IN_PAGED_CODE:
python315!_PyObject_IS_GC+20 [C:\actions-runner_work\pyperf_bench\pyperf_bench\cpython\Include\internal\pycore_object.h @ 834]
00007ffc`5f3395a0 4c8b4108 mov r8,qword ptr [rcx+8]

FAULTING_SOURCE_LINE: C:\actions-runner_work\pyperf_bench\pyperf_bench\cpython\Include\internal\pycore_object.h

FAULTING_SOURCE_FILE: C:\actions-runner_work\pyperf_bench\pyperf_bench\cpython\Include\internal\pycore_object.h

FAULTING_SOURCE_LINE_NUMBER: 834

FAULTING_SOURCE_CODE:
309:
310: static inline
311: PyTypeObject* _Py_TYPE_impl(PyObject *ob)
312: {

313: return ob->ob_type;
314: }
315:
316: // bpo-39573: The Py_SET_SIZE() function must be used to set an object size.
317: static inline Py_ssize_t
318: _Py_SIZE_impl(PyObject *ob)

SYMBOL_NAME: python315!_PyObject_IS_GC+20

MODULE_NAME: python315

IMAGE_NAME: python315.dll

FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_python315.dll!_PyObject_IS_GC

OS_VERSION: 10.0.26100.1

BUILDLAB_STR: ge_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

IMAGE_VERSION: 3.15.106.1013

FAILURE_ID_HASH: {41ad561c-4f71-20b0-5dcb-a02ca0bc2c99}

Followup: MachineOwner

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

Metadata

Metadata

Labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions