Place like /r/singularity and hackernews are eating up the hype and buying into it. Even with mathematicians explaining carefully the errors in the proofs and why they are badly written and useless to actually progress in mathematics the boosters are treating that like sour grapes or irrational fear. If mathematicians tried outright ignoring it, OpenAI’s narrative would go even more unchallenged.

You know how AI proofs are supposed to be credible because of formalization with Lean?

There is another problem… apparently Lean4 hasn’t actually been fully formally verified. And in fact could have more bugs lurking in it, maybe even some foundational problems… Lesswrong post explaining the problems.

OpenAI can more easily market that their AI is intended to fully replace mathematicians, and that this is only the herald of their general superintelligence (and not that math is the only thing working out for them).

Yeah, this has been driving me crazy… reading places like hackernews or r/singularity, I keep seeing people assuming mathematicians and just bitter or engaging in sour grapes and that is why they are complaining about the “proofs” OpenAI dumped out. Like it is just some small niggling detail that the proofs can’t be read by humans and that Lean may have bugs and that some of OpenAI’s proofs have already been shown to have critical mistakes.

Edit: I had the thought to collect some links in one place Here are some more discussions on lean and autoformalization for anyone looking for good sources:

• https://terrytao.wordpress.com/2026/10/09/what-mathematicians-should-know-about-the-lean-theorem-proverquestions-of-reliability-and-ai/ a general blog post breaking down some of the bugs found and potential for more
• https://arxiv.org/pdf/2610.08144 preprint discussing the problems with natural language to lean in detail and pointing out some blatant inconsistencies in the recent Navier-stokes “proof”
• /r/betteroffline discussion of above preprint: https://www.reddit.com/r/BetterOffline/comments/1x0b49i/navierstokes_lost_in_translation_why_lean/

Edit 2: Made a top level post: https://awful.systems/post/9997783

Or cite it, but in the most passive aggressive way you can get away with.

…You should check the literature really carefully to make sure that whatever proof the AI “generated” doesn’t actually already exist somewhere and it regurgitated it without proper accreditation.

Yeah lesswrong is weird about China. Its an ongoing thing. It alternates between fear (that China won’t appreciate the risk of making AI Satan or will beat the US to making AI God) and occasional hope (that they might manage to “safety-pill” China and China will find the magic formula to AI alignment). I pulled together some posts previously here: https://awful.systems/post/4103825

If it had been only a year they’ve been saying that, I might almost buy the logic. But it has been 4 years since Chat-GPT came out and they’ve been making the same promises the entire time.

Something of a tangent… thinking back to 2022-2023 my expectations for what the LLM companies would actually achieve by now were actually too high? Like I would have thought they would have figured out a better architecture for gluing functionality and symbolic logic to LLMs than just calling tools with the output text stream and feeding the tool outputs into the context window.

I also thought of Open Philanthropy, but couldn’t remember the Eliezer post that best illustrated his resentment towards them. (In addition to the top-level post you’ve linked, I’ve seen various bitter lesswrong comments along the same lines).

• “a giant rich organization”: they claim to have directed over 7 billion in grants since 2014, so this actually seems like a fair comment?

• “stole some of my ideas”: they’ve shown interest in AI stuff, including Eliezer’s ideas, they just have longer (but still pretty fantastical, like AGI by 2050) timelines for stuff like AGI and that influences their funding priorities in a way Eliezer disagrees with

• “ignored everything I said about what not to do”: eh, Eliezer has said a lot of things, they probably didn’t follow his advice exactly

• “and are going around telling people I made them do it”: I’ve seen a few people here and there in various rationalist-adjacent circles blame Eliezer for helping OpenAI network and get started and thereby kindle the very doom he warned about, so. It seems plausible some of them were connected to coefficient giving?

Its not stealing your ideas if they cite you Yud, they just draw different conclusions

That sounds like the normal academic process, which Eliezer neither understands nor appreciates.

Yud has said he got bored with LessWrong

Yeah, him gradually losing interest in LessWrong in the mid 2010s created room for SSC to grow in popularity. Also, that would have been around the time it became apparent Eliezer had been very wrong about neural networks, so we missed the chance to see Eliezer actually have to overcome biases and admit he was wrong in a non-self-aggrandizing way. (Eliezer has admitted he was wrong before, but only in ways that made his next idea even more important, see his progression from nanotech to coding a seed AI in Lisp to solving AI safety).

They actually both converge to a retelling of SCP-8008.

Not only that, they view commonsense moral and ethical reasoning as a PR constraint, i.e. look at the discourse around the hiring of Caroline Ellison to an EA org that handles significant sums of money. Lot of commenters were merely framing it as a PR problem and not a basic commonsense ethics problem.

Considering how much compensation he gets a MIRI, he could afford to hire an artist out of his own pocket and he would end up with something better and not be funding the technology that he claims is going to kill us all…

I think past AI winters is a somewhat useful analogy, but yeah, I agree, drawing too strong a comparison is misleading.

Likewise for people comparing the AI boom to the dotcom boom and bust. Ed Zitron has ripped into this comparison repeatedly. Basically, the fiberoptics laid by the dotcom boom where useful for decades, whereas heavily used GPUs fail in ~6 years.

The thing about the “yet you participate in society, curious!” meme is that you don’t really get the option to not participate in society. you do however get the option to not participate in AI, and there are plenty of people happily taking that option

Yeah, rationalist utterly fail at understanding collective action, or really anything besides lib-brained centrist methods of action. Well, maybe even lib-brained centrist is giving them too much credit, because even milquetoast liberals will occasionally avoid using products for ethical reasons. See Only Law Can Prevent Extinction, where Eliezer basically explains how all other action besides an international total ban on AI and GPUs is pointless. (I think he is trying to avoid people bombing data centers, but Elizer’s “logic” is also implicitly against stuff like going to town halls to complain about data centers.)

Yeah, the review nitpicks the quality of the AI generated slop, but overall is in favor of doing a better job slopping, not avoiding slop altogether.

Well, we didn’t believe them that their AI can make deadly bioweapons when they already showed clear indisputable evidence (the AI passing a biology multiple choice test), so clearly they need to have it make actual bioweapons so we’ll take the threat seriously!

Edit: had a thought

T-Virus

I mean rationalists, particularly the e/acc breed, would probably count the T-virus as a success. After all, wasn’t Umbrella corporation able to study it and come up with some really powerful transhuman augmentation for their various leaders? Just overlook the part where it turns you into an insane monster … I could easily envision someRationalist fanfic of Resident Evil portraying Umbrella corporation as misunderstood anti-heroes…

it just has that feel.

yeah. Looking past the initial quiz, a lot of the complaints about Eliezer’s predictions feel like rationalist inside-baseball about takeoff speeds and alignment strategies, as opposed to calling the whole thing bunk.

Calculators also have a negligible chance of screwing up something basic, and you can in fact learn the right way to input problems into them such that they are guaranteed not to misinterpret you.

Wow… I actually failed to get some of the times Eliezer has been wrong! His predictions are even worse than I remembered!

Thanks! I had failed to check back two stubstacks ago.

In short, here, I described this not as collusion

other similar systems

Something of a side note to your excellent write-up… I think we should push back on the “sub-agent” “agent swarm” language used to describe LLM workflows.

The workflows often described as swarms or whatever are the same LLM (or at least related LLMs), just prompted a bunch of different ways in parallel. Individual agents don’t really act like coherent agents in the lesswrong rationalist sense, or in any sort of philosophical sense of selfhood or agency or unified purposes, they just get described that way for convenience, so treating “swarms” of agents and subagents as some special category is lending too much credence to the anthropomorphisizing boosters like to do. “Agent swarms” really just means that someone set up a slop machine to prompt itself a bunch of times in parallel without enough human supervision.

A lesswronger correctly notices that LLM companies’ “research” papers are often lacking in details necessary to replicate them! They fail to notice the reasons why that is, or do any broader questioning or soul searching about the state of LLM and “AI Safety” “research”.