TNS
VOXPOP
Favorite Social Media Timesink
When you take a break from work, where are you going?
Instagram/Facebook
0%
Discord/Slack
0%
LinkedIn
0%
Video clips on TikTok/YouTube
0%
X, Bluesky, Mastodon et al...
0%
Web surfing
0%
I do not get distracted by petty amusements
0%
Python

Python to Drop the Global Lock for Greater Parallelism

Big data user Meta has pledged three engineer years to creating a version of Python without global locks.
Aug 8th, 2023 6:46am by
Featued image for: Python to Drop the Global Lock for Greater Parallelism
Feature Image by Lynn Greyling from Pixabay.

In a move to ease use for large-scale data analysis work, the Python Steering Council has elected to drop the lock on Python that restricts a Python user program to using only a single thread. This restriction has become a serious performance impediment for running Python code across multicore CPUs, which is to say most all CPUs sold today.

Specifically, the Python global interpreter lock (“GIL”) prevents “multiple threads from executing Python code at the same time,” according to the proposal ”PEP 703 – Making the Global Interpreter Lock Optional in CPython.”

CPython is the reference implementation of Python.

In a poll of 46 committers, 87% agreed that Python should be made “free-threaded.”

Meta, the big data company behind Facebook and Instagram, is one company that is heartily behind the adoption of PEP 703, having committed three engineer years to building out a no-GIL Python.

On July 28, CPython core developer Thomas Wouters wrote that “We intend to accept PEP 703, although we’re still working on the acceptance details.”

Wouters proposed the rollout to happen in three stages. First, there will be an experimental build without the lock. Over time, support will be added for the GIL-less Python. In the final stage, the GIL-less Python will be default, with any vestiges of the GIL stripped away.

This timeline of course, will carry out over a period of years.

The Problem with GIL

“The GIL is a major obstacle to concurrency,” wrote Sam Gross in PEP 702. “The GIL introduces a global bottleneck that can prevent other threads from making progress if they call any Python code.”

This has been a major problem, for instance, in neural network-based AI models, which require many identical operations to run in parallel. Many other languages support this parallel capability, and it can be done within Python as well, through some customization.

GIL has brought other issues as well. Python libraries, for instance, are often made more complicated by GIL, necessitating needless complexity within API design. PyTorch, for instance, provides ways for Python AI models that avoid or work around the GIL entirely, though these hacks come with limitations. It also makes the interface for GPUs more complicated.

Is Python Getting too Complicated for New Users?
Efforts like the Cython project and the removal of the Global Interpreter Lock aim to gear Python for faster, high-throughput data analysis, but also introduce additional complexity.
No way: The improvements are much needed, backward compatible and optional to use.
0%
Yes: Python’s chief value is its simplicity, not its speed. Production can be executed with R, or C++ or some other performant language.
0%
Who cares? Python is too problematic even for new users (blank spaces, runaway libraries, etc.). Newbies, start with JavaScript instead.
0%
Cowboy Neal (h/t, Slashdot and Jack Kerouac)
0%

“Python’s global interpreter lock makes it difficult to use modern multicore CPUs efficiently for many scientific and numeric computing applications,” Gross summarized

How much work the move will require on existing libraries remains up for debate, as a recent Hacker News discussion shows. C and C++ libraries, or at least their interfaces, may have to be updated.

“The GIL does currently result in robust code by default because data can’t mutate underneath you. Without the GIL the code will appear to work, but it will be trivial for an attacker to use mutations to crash the code. Expect huge numbers of CVEs,” wrote one Python library contributor on the site.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: Pragma.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.