(Posted for the whole Steering Council.) As we’ve announced before, the Steering Council has decided to accept PEP 703 (Making the Global Interpreter Lock Optional in CPython) . We want to make it clear why, and under what expectations we’re doing so. It is clear to the Steering Council that theoretically, a no-GIL (or free-threaded) Python would be of great benefit, and the majority of the community seems in agreement. Threads have significant downsides and caveats, but they are widely adopte...
They said multithreading can’t do parallel computing, what part of that is false?
Besides, going to multiprocessing isn’t just “a little overhead” you need to switch from a shared data model to inter process communication, which isn’t always trivial
There is a common dev story in python: Hrmm this is running slow, maybie I can use threads to make it go faster. Weird, not faster, discovers GIL. Maybe I can use multiprocessing. Hrmm this sucks I have to use IPC and serialize things to pass them. Hrmm faster but still weirdly slow. Proceeds to spend a ton of time optimizing IPC and figuring how to get code in multiple processes to communicate.
I’ve never heard this story.
GIL removal solves the relatively small problem of, “I have a big workload but not so big that I need multiple nodes.”
Small workloads are fine and don’t need free threading. Large workloads are going to use IPC anyway to coordinate across hundreds of nodes.
Today you must use the IPC overhead approach for medium workloads and it is some extra work. But then if your application grows you’ve already done much of the scaling part.
You just summarized a week of wasted efforts at my job.