That Carl Shulman post from 2007 is hilarious.

After years spent studying existential risks, I concluded that the risk of an artificial intelligence with inadequately specified goals dominates. Attempts to create artificial intelligence can be expected to continue, and to become more likely to succeed in light of increased computing power, neuroscience, and intelligence-enhancements. Unless the programmers solve extremely difficult problems in both philosophy and computer science, such an intelligence might eliminate all utility within our future light-cone in the process of pursuing a poorly defined objective.

Accordingly, I invest my efforts into learning more about the relevant technologies and considerations, increasing my earnings capability (so as to deliver most of a large income to relevant expenditures), and developing logistical strategies to more effectively gather and expend resources on the problem of creating AI that promotes (astronomically) and preserves global welfare rather than extinguishing it.

Because the potential stakes are many orders of magnitude greater than relatively good conventional expenditures (vaccine and Green Revolution research), and the probability of disaster much more likely than for, e.g. asteroid impacts, utilitarians with even a very low initial estimate of the practicality of AI in coming decades should still invest significant energy in learning more about the risks and opportunities associated with it. (Having done so, I offer my assurance that this is worthwhile.) Note that for materialists the possibility of AI follows from the existence proof of the human brain, and that an AI able to redesign itself for greater intelligence and copy itself would have the power to determine the future of Earth-derived life.

I suggest beginning with the two articles below on existential risk, the first on relevant cognitive biases, and the second discussing the relation of AI to existential risk. Processing these arguments should provide sufficient reason for further study.

The “two articles below” are by Yudkowsky.

User “gaverick” replies,

Carl, I’m inclined to agree with you, but can you recommend a rigorous discussion of the existential risks posed by Unfriendly AI? I had read Yudkowsky’s chapter on AI risks for Bostrom’s bk (and some of his other SIAI essays & SL4 posts) but when I forward them to others, their informality fails to impress.

Shulman’s response begins,

Have you read through Bostrom’s work on the subject? Kurzweil has relevant info for computing power and brain imaging.

Ray mothersodding Kurzweil!

jhbadger:

As Adam Becker shows in his book, EAs started out being reasonable “give to charity as much as you can, and research which charities do the most good” but have gotten into absurdities like “it is more important to fund rockets than help starving people or prevent malaria because maybe an asteroid will hit the Earth, killing everyone, starving or not”.

I haven’t read Becker’s book and probably won’t spend the time to do so. But if this is an accurate summary, it’s a bad sign for that book, because plenty of them were bonkers all along.

(Becker’s previous book, about the interpretation of quantum mechanics, irritated me. It recapitulated earlier pop-science books while introducing historical and technical errors, like getting the basic description of the EPR thought-experiment wrong, and butchering the biography of Grete Hermann while acting self-righteous about sexist men overlooking her accomplishments. See previous rant.)

astrange:

They’re members of a religion which says that if you do math in your head the right way you’ll be correct about everything, and so they think they’re correct about everything.

They also secondarily believe everyone has an IQ which is their DBZ power level; they believe anything they see that has math in it, and IQ is math, so they believe anything they see about IQ. So if you avoid trying to find out your own IQ you can just believe it’s really high and then you’re good.

Unfortunately this lead them to the conclusion that computers have more IQ than them and so would automatically win any intellectual DBZ laser beam fight against them / enslave them / take over the world.

My Grand Unified Theory of Scott Aaronson is that he doesn’t have a theory of mind. On subjects far less incendiary than Zionism, he simply fails to recognize that people who share his background or interests can think differently than he does.

The lead author wrote a summary thread.

From p. 137:

The most consistent and significant behavioral divergence between the groups was observed in the ability to quote one’s own essay. LLM users significantly underperformed in this domain, with 83% of participants (15/18) reporting difficulty quoting in Session 1, and none providing correct quotes. This impairment persisted albeit attenuated in subsequent sessions, with 6 out of 18 participants still failing to quote correctly by Session 3. […] Search Engine and Brain-only participants did not display such impairments. By Session 2, both groups achieved near-perfect quoting ability, and by Session 3, 100% of both groups’ participants reported the ability to quote their essays, with only minor deviations in quoting accuracy.

Or you could read the entirety of the first comment in this thread and see how it was not saying that. Notice the part that begins, “However, I believe there is an important difference to chatbots…”

No Nut Neuravember

Does master own a Sawzall?

Fuck it, repeating my joke from the earlier thread: Inviting the most pedantic nerds on Earth to critique your chatbot slop is a level of begging to be pwned that’s on par with claiming the female orgasm is a myth.

My tax prep software definitely has a mode called “give me Deus Ex”

yes you need to read things to understand them

OK, here’s your free opportunity to spend more time doing that. Bye now.

The stubsack is the weekly thread of miscellaneous, low-to-mid-effort posts on awful.systems.

Obligatory cross-reference: This came up in the stubsack before 404media wrote about it.

She said, “You know what they say the modern version of Pascal’s Wager is? Sucking up to as many Transhumanists as possible, just in case one of them turns into God. Perhaps your motto should be ‘Treat every chatterbot kindly, it might turn out to be the deity’s uncle.’”

“Crystal Nights”

“Cursor YOLO deleted everything in my computer”:

Hi everyone - as a previous context I’m an AI Program Manager at J&J and have been using Cursor for personal projects since March.

Yesterday I was migrating some of my back-end configuration from Express.js to Next.js and Cursor bugged hard after the migration - it tried to delete some old files, didn’t work at the first time and it decided to end up deleting everything on my computer, including itself. I had to use EaseUS to try to recover the data, but didn’t work very well also. Lucky I always have everything on my Google Drive and Github, but it still scared the hell out of me.

Now I’m allergic to YOLO mode and won’t try it anytime soon again. Does anyone had any issue similar than this or am I the first one to have everything deleted by AI?

The response:

Hi, this happens quite rarely but some users do report it occasionally.

My T-shirt is raising questions already answered, etc.

(via)

The New York Times treats him as an expert: “Eliezer Yudkowsky, a decision theorist and an author of a forthcoming book”. He’s an Internet rando who has yammered about decision theory, not an actual theorist! He wrote fanfic that claimed to teach rational thinking while getting high-school biology wrong. His attempt to propose a new decision theory was, last I checked, never published in a peer-reviewed journal, and in trying to check again I discovered that it’s so obscure it was deleted from Wikipedia.

https://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/Functional_Decision_Theory

To recapitulate my sneer from an earlier thread, the New York Times respects actual decision theorists so little, it’s like the whole academic discipline is trans people or something.

viral: Midjourney fucking up human hands

spiral: Midjourney somehow fucking up Simpsons hands

https://bsky.app/profile/blakestacey.bsky.social/post/3lrep72zj4c23

When the complaint gets to saying what Midjourney advertises in their “explore” feature, it features dead-eyed waifu Elsa and body-pillow-ready topless Ariel.

In which the New York Times respects actual decision theorists so little, it’s like the whole academic discipline is trans people or something

so much content

The choice of, or instinctive reaching for, the word content speaks volumes.

where anything you want to see or read can be spun up in an instant, without labor.

“Without labor,” sure.