Skip to content

Comments

gh-145142: Make str.maketrans safe under free-threading#145157

Open
VanshAgarwal24036 wants to merge 10 commits intopython:mainfrom
VanshAgarwal24036:gh-145142-dict-next-critical-section
Open

gh-145142: Make str.maketrans safe under free-threading#145157
VanshAgarwal24036 wants to merge 10 commits intopython:mainfrom
VanshAgarwal24036:gh-145142-dict-next-critical-section

Conversation

@VanshAgarwal24036
Copy link
Contributor

@VanshAgarwal24036 VanshAgarwal24036 commented Feb 23, 2026

Replace unsafe PyDict_Next iteration with snapshot iteration using PyDict_Items to avoid crashes when the input dict is mutated concurrently under free-threading.

Adds a regression test.

Copy link
Contributor

@colesbury colesbury left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments below

}
PyObject *items = PyDict_Items(x);
if(items == NULL) goto err;
Py_ssize_t n = PyList_GET_SIZE(items);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use a critical section here instead of copying the items to a list. You'll need to fix up the goto err. Something like:

{
    ...
    Py_BEGIN_CRITICAL_SECTION(x);
    while (PyDict_Next(x, &i, &key, &value)) {
        ...
    }
    goto done;
err:
    Py_CLEAR(new);
done:
    Py_END_CRITICAL_SECTION();
    return new;
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • It fails the Windows _freeze_module build (syntax errors from macro expansion).
  • A version with early returns builds but causes test_str failures under free-threading on macOS/Linux

Is there a preferred pattern for using Py_BEGIN_CRITICAL_SECTION here, or would you prefer using the dict’s internal locking helpers instead?

Copy link
Member

@picnixz picnixz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest isolating the loop in a separate function so that we do not have those weird labels jumps and confusing names but this should be benchmarked on PGO/LTO tocheck that we do not introduce an overhead (though I am not that worried as str.maketrans() is usually not a hot function).

newkey = PyLong_FromLong(PyUnicode_READ(kind, data, 0));
if (!newkey)
goto err;
goto error;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having the err and error labels is really confusing. We should not have both.

Comment on lines 13187 to 13192
goto done;
error:
Py_CLEAR(new);
done:
Py_END_CRITICAL_SECTION();
return new;
Copy link
Member

@picnixz picnixz Feb 24, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having a "done" while not entirely done (that is, being done inside the function block) is confusing as well. There are too many jumps for it to be clear when reading the code. I would suggest (1) not using labels at all or (2) find better names for the labels (3) or just duplicate the Py_CLEAR(new) and remove the error label (and rename the done label). (4) wrap the logic in a separate helper.

@bedevere-app
Copy link

bedevere-app bot commented Feb 24, 2026

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

Copy link
Contributor

@sharktide sharktide left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bunch of syntax errors you need to fix as well.

I also agree with picnixz’s comments

@VanshAgarwal24036 VanshAgarwal24036 marked this pull request as draft February 24, 2026 12:44
@VanshAgarwal24036 VanshAgarwal24036 force-pushed the gh-145142-dict-next-critical-section branch from d487864 to 05ba3f3 Compare February 24, 2026 15:57
@VanshAgarwal24036 VanshAgarwal24036 marked this pull request as ready for review February 24, 2026 16:37
@VanshAgarwal24036 VanshAgarwal24036 force-pushed the gh-145142-dict-next-critical-section branch from 206f6b2 to 4e715b0 Compare February 24, 2026 17:34
@VanshAgarwal24036
Copy link
Contributor Author

I have made the requested changes; please review again.

@bedevere-app
Copy link

bedevere-app bot commented Feb 24, 2026

Thanks for making the requested changes!

@picnixz: please review the changes made to this pull request.

goto err;
}
}
if(unicode_maketrans_from_dict(x, new) < 0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think @picnixz suggestion was to put the critical section outside the function. Something like:

int err;
Py_BEGIN_CRITICAL_SECTION(x);
err = unicode_maketrans_from_dict(x, new);
Py_END_CRITICAL_SECTION();
if (x < 0) {
    goto err;
}

Then inside unicode_maketrans_from_dict, the error returns can just be return -1 instead of break and no need for the if (!PyErr_Occurred()) check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants