Python to Drop the Global Lock for Greater Parallelism
In a move to ease use for large-scale data analysis work, the Python Steering Council has elected to drop the lock on Python that restricts a Python user program to using only a single thread. This restriction has become a serious performance impediment for running Python code across multicore CPUs, which is to say most all CPUs sold today.
Specifically, the Python global interpreter lock (“GIL”) prevents “multiple threads from executing Python code at the same time,” according to the proposal ”PEP 703 – Making the Global Interpreter Lock Optional in CPython.”
CPython is the reference implementation of Python.
In a poll of 46 committers, 87% agreed that Python should be made “free-threaded.”
No More GIL!
the Python team has officially accepted the proposal.Congrats @colesbury on his multi-year brilliant effort to remove the GIL, and a heartfelt thanks to the Python Steering Council and Core team for a thoughtful plan to make this a reality.https://t.co/58QK2yctRD
— Soumith Chintala (@soumithchintala) July 30, 2023
Meta, the big data company behind Facebook and Instagram, is one company that is heartily behind the adoption of PEP 703, having committed three engineer years to building out a no-GIL Python.
On July 28, CPython core developer Thomas Wouters wrote that “We intend to accept PEP 703, although we’re still working on the acceptance details.”
Wouters proposed the rollout to happen in three stages. First, there will be an experimental build without the lock. Over time, support will be added for the GIL-less Python. In the final stage, the GIL-less Python will be default, with any vestiges of the GIL stripped away.
This timeline of course, will carry out over a period of years.
The Problem with GIL
“The GIL is a major obstacle to concurrency,” wrote Sam Gross in PEP 702. “The GIL introduces a global bottleneck that can prevent other threads from making progress if they call any Python code.”
This has been a major problem, for instance, in neural network-based AI models, which require many identical operations to run in parallel. Many other languages support this parallel capability, and it can be done within Python as well, through some customization.
GIL has brought other issues as well. Python libraries, for instance, are often made more complicated by GIL, necessitating needless complexity within API design. PyTorch, for instance, provides ways for Python AI models that avoid or work around the GIL entirely, though these hacks come with limitations. It also makes the interface for GPUs more complicated.
“Python’s global interpreter lock makes it difficult to use modern multicore CPUs efficiently for many scientific and numeric computing applications,” Gross summarized
How much work the move will require on existing libraries remains up for debate, as a recent Hacker News discussion shows. C and C++ libraries, or at least their interfaces, may have to be updated.
“The GIL does currently result in robust code by default because data can’t mutate underneath you. Without the GIL the code will appear to work, but it will be trivial for an attacker to use mutations to crash the code. Expect huge numbers of CVEs,” wrote one Python library contributor on the site.
