Data race in `sys_set_flag` when `sys.set_int_max_str_digits()` is called concurrently with free-threaded build

[Naserume]

Bug report

Bug description:

sys.set_int_max_str_digits() updates sys.flags via sys_set_flag(), which modifies a shared struct-sequence slot without synchronization:

cpython/Python/sysmodule.c

Lines 3479 to 3487 in ce916dc

	static void
	sys_set_flag(PyObject *flags, Py_ssize_t pos, PyObject *value)
	{
	assert(pos >= 0 && pos < (Py_ssize_t)(Py_ARRAY_LENGTH(flags_fields) - 1));
	PyObject *old_value = PyStructSequence_GET_ITEM(flags, pos);
	PyStructSequence_SET_ITEM(flags, pos, Py_NewRef(value));
	Py_XDECREF(old_value);
	}

Concurrent callers race on the slot and can Py_XDECREF the same old_value, leading to a use-after-free of the old flag object.

Reproducer:

import sys
from threading import Thread

def thread1():
    for i in range (20000):
        sys.set_int_max_str_digits(4300 + (i & 7))

threads = [Thread(target=thread1) for _ in range(4)]
for t in threads: t.start()
for t in threads: t.join()

TSAN Report:

==================
WARNING: ThreadSanitizer: data race (pid=1518581)
  Read of size 8 at 0x7fffb63e01b8 by thread T1:
    #0 PyStructSequence_GetItem /cpython/Objects/structseq.c:112:12  
    #1 sys_set_flag /cpython/./Python/sysmodule.c:3484:27  
    #2 _PySys_SetFlagObj /cpython/./Python/sysmodule.c:3498:5 
    #3 _PySys_SetFlagInt /cpython/./Python/sysmodule.c:3512:15  
    #4 _PySys_SetIntMaxStrDigits /cpython/./Python/sysmodule.c:4671:9 
    #5 sys_set_int_max_str_digits_impl /cpython/./Python/sysmodule.c:1894:9  
    #6 sys_set_int_max_str_digits /cpython/./Python/clinic/sysmodule.c.h:932:20 
    #7 _Py_BuiltinCallFastWithKeywords_StackRef /cpython/Python/ceval.c:839:11  
    #8 _PyEval_EvalFrameDefault /cpython/Python/generated_cases.c.h:2508:35 

  Previous write of size 8 at 0x7fffb63e01b8 by thread T2:
    #0 PyStructSequence_SetItem /cpython/Objects/structseq.c:100:27  
    #1 sys_set_flag /cpython/./Python/sysmodule.c:3485:5  
    #2 _PySys_SetFlagObj /cpython/./Python/sysmodule.c:3498:5 
    #3 _PySys_SetFlagInt /cpython/./Python/sysmodule.c:3512:15  
    #4 _PySys_SetIntMaxStrDigits /cpython/./Python/sysmodule.c:4671:9 
    #5 sys_set_int_max_str_digits_impl /cpython/./Python/sysmodule.c:1894:9  
    #6 sys_set_int_max_str_digits /cpython/./Python/clinic/sysmodule.c.h:932:20 
    #7 cfunction_vectorcall_FASTCALL_KEYWORDS /cpython/Objects/methodobject.c:465:24  
    #8 _PyObject_VectorcallTstate /cpython/./Include/internal/pycore_call.h:144:11  
    #9 PyObject_Vectorcall /cpython/Objects/call.c:327:12
...
SUMMARY: ThreadSanitizer: data race /cpython/Objects/structseq.c:112:12 in PyStructSequence_GetItem
==================
==================
WARNING: ThreadSanitizer: data race (pid=1518581)
  Read of size 4 at 0x7fffc004057c by thread T2:
    #0 _Py_ExplicitMergeRefcount /cpython/Objects/object.c:477:40  
    #1 _Py_brc_queue_object /cpython/Python/brc.c:73:31  
    #2 _Py_DecRefSharedIsDead /cpython/Objects/object.c:413:9  
    #3 _Py_DecRefSharedDebug /cpython/Objects/object.c:425:9 
    #4 _Py_DecRefShared /cpython/Objects/object.c:433:5 
    #5 Py_DECREF /cpython/./Include/refcount.h:385:9  
    #6 Py_XDECREF /cpython/./Include/refcount.h:520:9 
    #7 sys_set_flag /cpython/./Python/sysmodule.c:3486:5 
    #8 _PySys_SetFlagObj /cpython/./Python/sysmodule.c:3498:5 
    #9 _PySys_SetFlagInt /cpython/./Python/sysmodule.c:3512:15  
    #10 _PySys_SetIntMaxStrDigits /cpython/./Python/sysmodule.c:4671:9 
    #11 sys_set_int_max_str_digits_impl /cpython/./Python/sysmodule.c:1894:9  
    #12 sys_set_int_max_str_digits /cpython/./Python/clinic/sysmodule.c.h:932:20 

  Previous atomic write of size 4 at 0x7fffc004057c by thread T3:
    #0 _Py_atomic_store_uint32_relaxed /cpython/./Include/cpython/pyatomic_gcc.h:493:3  
    #1 Py_DECREF /cpython/./Include/refcount.h:379:9 
    #2 Py_XDECREF /cpython/./Include/refcount.h:520:9 
    #3 sys_set_flag /cpython/./Python/sysmodule.c:3486:5 
    #4 _PySys_SetFlagObj /cpython/./Python/sysmodule.c:3498:5 
    #5 _PySys_SetFlagInt /cpython/./Python/sysmodule.c:3512:15  
    #6 _PySys_SetIntMaxStrDigits /cpython/./Python/sysmodule.c:4671:9 
    #7 sys_set_int_max_str_digits_impl /cpython/./Python/sysmodule.c:1894:9  
    #8 sys_set_int_max_str_digits /cpython/./Python/clinic/sysmodule.c.h:932:20 
...
SUMMARY: ThreadSanitizer: data race /cpython/Objects/object.c:477:40 in _Py_ExplicitMergeRefcount
==================
==================
WARNING: ThreadSanitizer: data race (pid=1518581)
  Read of size 8 at 0x7fffc4042720 by thread T3:
    #0 long_dealloc /cpython/Objects/longobject.c:3653:9  
    #1 _Py_Dealloc /cpython/Objects/object.c:3312:5  
    #2 _Py_brc_queue_object /cpython/Python/brc.c  
    #3 _Py_DecRefSharedIsDead /cpython/Objects/object.c:413:9  
    #4 _Py_DecRefSharedDebug /cpython/Objects/object.c:425:9 
    #5 _Py_DecRefShared /cpython/Objects/object.c:433:5 
    #6 Py_DECREF /cpython/./Include/refcount.h:385:9  
    #7 Py_XDECREF /cpython/./Include/refcount.h:520:9 
    #8 sys_set_flag /cpython/./Python/sysmodule.c:3486:5 
    #9 _PySys_SetFlagObj /cpython/./Python/sysmodule.c:3498:5 
    #10 _PySys_SetFlagInt /cpython/./Python/sysmodule.c:3512:15  
    #11 _PySys_SetIntMaxStrDigits /cpython/./Python/sysmodule.c:4671:9 
    #12 sys_set_int_max_str_digits_impl /cpython/./Python/sysmodule.c:1894:9  
    #13 sys_set_int_max_str_digits /cpython/./Python/clinic/sysmodule.c.h:932:20 

  Previous write of size 8 at 0x7fffc4042720 by thread T4:
    #0 _PyLong_SetSignAndDigitCount /cpython/./Include/internal/pycore_long.h:307:27  
    #1 _PyLong_FromMedium /cpython/Objects/longobject.c:267:5 
    #2 PyLong_FromLong /cpython/Objects/longobject.c:409:5  
    #3 _PySys_SetFlagInt /cpython/./Python/sysmodule.c:3507:21  
    #4 _PySys_SetIntMaxStrDigits /cpython/./Python/sysmodule.c:4671:9 
    #5 sys_set_int_max_str_digits_impl /cpython/./Python/sysmodule.c:1894:9  
    #6 sys_set_int_max_str_digits /cpython/./Python/clinic/sysmodule.c.h:932:20 
...
SUMMARY: ThreadSanitizer: data race /cpython/Objects/longobject.c:3653:9 in long_dealloc
==================

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Linked PRs

• gh-151218: Fix data race in sys_set_flag for free-threading #151220

[bhuvi27]

I'd like to work on this.