Maybe we should add sone asserts here and there that tok->done is not error to prevent this issue from happening in other places

That's a good idea. Where would you add these asserts though, since most of the time we shouldn't be asserting, but rather check for it?

One of the items I have in my to-do list is to add comments and assert in various places in the tokenizer to make the logic a bit easier to follow. Would it be okay to merge this as-is and then do that in a separate PR?

Would it be okay to merge this as-is and then do that in a separate PR?

Absolutely, let's merge this as is 🚀

Great job as always man!

@lysnikolaou: Do you consider backporting your change to Python 3.11?

See discussion linked in the associated issue, there was a decision made not to backport the initial change that worked for single quoted strings, to avoid breaking compatibility in a patch version release.

See #96670 (comment)

FYI I removed the label until it's more clear what should be backported.

GH-104195 includes a backport of this commit to 3.11.