why i cant enter to the page
Earth-shattering news, but looks like it will take a s**t-ton of work before it becomes mainstream.
Yes, but it’s been under development for a while, they’re well aware of the complexity, and IIRC the work is being funded by Meta. So I think it will come sooner than one might expect.
Can’t wait. I’ve been squeezing every last inch of computing power out of Python for the last two years, leveraging asyncio and distributed architectures, and I think that it’s far from the slow tool that many people imagine. Now this.
I’ve been wishing to point out for a while (but I’m more of a lurker than a talker) that the real power of asyncio lies in well thought-out architectures based on cooperative multitasking: hard to fine tune, but impressively effective.
If this can be achieved, Python’s world domination will be well underway.
Python is already No. 1 in the TIOBE Index, and mutithreading is currently one of Python’s weakest points. I know I’ve decided not to use Python for a personal project a few times because multithreading was important, and I can’t be the only one.
What’s wrong with Python’s multithreading? I’ve seen some other accounts that it’s not its strong suit. Is it because it leverages operating system level abstractions to make it happen or something else?
Because of the GIL. Multithreading in python does not work as people would expect from other languages, it can’t do computation in parallel
Actually, it’s even worse than that. The GIL protects prevents you from trashing your interpreter, but you still have to synchronize your Python code or else you get race conditions.
False. As for now, you can just use multiprocessing instead of multi threading to achieve parallel computation (with a little of overhead though)
A little overhead? Each interpreter spawned adds 50mb.of RAM used. Doesn’t sound like much, but on an 8 core, 16 thread CPU, spawning 15 additional interpreters, eats up nearly a gig of ram on its own. On Windows (unsure about Linux/Mac), it also adds time to startup, and you get way less computational power out of it than using something else. Idk if anyone else does this, but I start the processes on program startup so they’re always available.
It’s likely the end consumer doesn’t know/doesn’t care about the slight performance gains, especially when competitors in my niche get away with crap like “your search is in queue, we’ll email you when you’re done”, but I find that abhorrent and lazy and all around stupid, so I take all performance advantages I can get.
They said multithreading can’t do parallel computing, what part of that is false?
Besides, going to multiprocessing isn’t just “a little overhead” you need to switch from a shared data model to inter process communication, which isn’t always trivial
There is a common dev story in python: Hrmm this is running slow, maybie I can use threads to make it go faster. Weird, not faster, discovers GIL. Maybe I can use multiprocessing. Hrmm this sucks I have to use IPC and serialize things to pass them. Hrmm faster but still weirdly slow. Proceeds to spend a ton of time optimizing IPC and figuring how to get code in multiple processes to communicate.
You just summarized a week of wasted efforts at my job.
There is a common dev story in python
I’ve never heard this story.
GIL removal solves the relatively small problem of, “I have a big workload but not so big that I need multiple nodes.”
Small workloads are fine and don’t need free threading. Large workloads are going to use IPC anyway to coordinate across hundreds of nodes.
Today you must use the IPC overhead approach for medium workloads and it is some extra work. But then if your application grows you’ve already done much of the scaling part.
You know I’m using tensorflow and it seems to do a great job utilizing all my available cores. I know there’s underlying C code that makes that possible, but I don’t see why that’s a problem. This isn’t a weak point in python. World class C integration is one of the best things about python.
BECAUSE, as you’ve discovered, the part that does the actual calculations in parallel is indeed written in C/C++/CUDA. You are just using python as basically a glorified bash at that point (which is great, because that’s EXACTLY what it was designed for). Python is the PERFECT tool for that job.
There are people out there who want are not satisfied with that. They want to write THOSE performant bits (or ones like it) in pure python. Not with some compilable subset of the language or jit compiler, but with ACTUAL, interpreted, python code. There are people who want to write non-io-limited multi-threaded algorithms IN python. They all believe they are hamstrung by the GIL, and instead of just picking a (compiled w/ proper threading) language much better-suited for that particular job, they are perpetually trying to fit a round peg into a square hole.