The major selling point for these LLM services is that people don’t need to learn a strict formal query language to interact; the LLM consumes instructions and data all as an undistinguished stream, and statistically infers a plausible response with absolutely no regard to the input’s meaning.

That is: despite the media reporting these as “injection attacks”, that term means nothing when all its input, every time, is treated as data and instruction simultaneously. This isn’t some special class of attack; it’s a fundamental designed-in flaw of the system.

The correct way to ensure protection from these vulnerabilities is long established, from decades of experience. You establish a firm boundary: never treat the input data as instructions, but instead have a separate channel for extremely well formalised query instructions, and reject bad input on that channel.

But of course that would kill the major appeal for most people who love these things, the fact they don’t need to learn any strict formal language and can just say anything at all and get some useful-looking response. Take that away, and you lose any hope that the masses will want to use these systems.

And so the makers and promoters of these systems will never make the one change that could even feasibly allow safety from these attacks; they will never make any improvement to security that might reduce the apparent ease of use of these things.

For as long as that remains, these systems will continue to inevitably have these exploits because the corporations won’t close the exploit surface on these systems. These are staggeringly insecure by design, and can’t be fixed without being completely replaced.

A decade ago, when I was Talking About Cryptocurrency (approving), in presentations etc., this was indeed a very frequent assumption of my audience. They would be confused and/or shocked when I referred to earlier in the presentation to show that no, every transaction on the blockchain is visible to everyone, forever, otherwise it can’t even work.

I suspect a lot of the shocked response was due to the (unspoken) follow up question “then what the heck is the point of this, why would I use this?”

But I can see why the assumption comes along. Cryptography is all about secret, obscured communication; why wouldn’t cryptocurrency be about secret money transactions?

At least now we can Talk About Cryptocurrency (derogatory) and dispel that assumption, where we can.

David G: “What happens after the bubble pops? What do we do amongst the wreckage of what was once the AI bubble?”

Tante: “A question that really concerns me is [the data centres that have been built]; who will they [the future owners of data centres, after the hyperscalers stop paying] turn to, to get their money back? The only entities that I can think of that have the money to make this worth it, is the military, and the security apparatus. […]

“That is useful for a lot of shit that we don’t want. That is a problem that really concerns me. Who has problems that fit to pattern-matching machines; and who has the money [to pay data centres to address those problems]? And that’s gonna be a lot of military applications, that’s gonna be a lot of security apparatus applications, that we all probably should not want.”

David G: “That’s wonderfully probable and depressing.”

Tante: “I’m fun at parties.”

The end result will probably be some sort of copy-protection system which will benefit those who want to censor the web and create paywalls and other gatekeeping methods to block access to content. We’re better off with the web as it is now, written entirely in plaintext. Besides, obfuscation goes against the spirit the World Wide Web was founded on: free and open access to information for the benefit of humanity.

Emphatically agreed. The open web was already being closed down before the LLM hyperscalers; it is closing down even more rapidly, partly to defend against the LLM hyperscalers dunning every website with pointless scraping all the time.

If we respond by obfuscating the text on the web, we hurt ourselves enormously and still won’t stop the LLM hyperscalers from ruining the web. It’s a misguided, technical non-solution, for something that needs a political response.

AI systems are never going to start getting dumber.

…hasn’t this already happened many times???

I mean, I guess it can be seen as technically true, because LLMs (the subset of AI they’re clearly talking about here) already know exactly zero about reality or what’s true. Can’t know less than that.

It’s a minor tell, but I’m always alert to people claiming to address global problems who cite calendar time with “summer 2026” and the like, as though the northern hemisphere is the only one that matters. Revealing they actually don’t give any thought to the world outside their local province.

finding the [human created] data to support the project is uncharted territory, he added.

Fortunately for the executives at hyperscaler corporations, they know this isn’t their problem.

They plan to cash out well before the model collapse is revealed, and leave less-wealthy suckers holding the bag.

It’s like killing the goose that lays the golden eggs.

Goose was not valued

I have no idea why they would do this

I do. For the same reason they are cramming LLMs into every other product:

They need to pretend to their investors that there is an enormous market hungry for more of this, so the investors will not demand their money back.

By turning more users of existing services, into “users of Gemini”; by turning more translate requests, more email messages, more search requests, into a “Gemini usage”; they can with a straight face claim more and ever more usage and demand, propped up by users who never agreed to any of this.

From TFA:

So what do we search with instead of Google? There isn’t a lot of choice. There’s various flavours of Google or Bing.

[…] Kagi resells Google’s actual link results for a monthly subscription. Kagi is really an AI company that sells search as a sideline, and they’ve got a terrible AI slop news service too. But the search has a good reputation for now.

I went to the landing page, expecting garbage genAI images; it did not disappoint.

Good news every time this happens.

Even better, this blog post is a great reference of the many reasons GitHub is shitty, becoming shittier, and everyone should abandon that platform.

Thank you for your continuing work to maintain this instance. May the hostile bots turn on their greedy masters so we can all laugh.

GEMA versus OpenAI. Whoever wins, we all lose.

But I’m still happy to see them fight.

So many of the criticisms here survive essentially unchanged, fifty years later:

If he calls the main loop of his program “UNDERSTAND”, he is (until proven innocent) merely begging the question. He may mislead a lot of people, most prominently himself, and enrage a lot of others.[…] If Quillian <1969> had called his program the “Teachable Language Node Net Intersection Finder”, he would have saved us some reading. (Except for those of us fanatic about finding the part on teachability.)