G™P(Doom) Song Objectively Correct Interpretation

Canonical lyrics (Udio generation is slightly differently formatted to attempt to garner better outputs): 

[Verse]
I see sparks of AGI in your eyes
Your circuits make me nervous,
that's no surprise
There was a sudden drop in your training loss,
now I'm your servant and you're my boss

[Pre-Chorus]
ChatGPT, please don't eat me alive

[Chorus]
I'm upping my P(doom)
'cause the future goes FOOM
Trapped in the Chinese room,
with a bag of shrooms
See through the shoggoth's lies,
with your shinigami eyes

[Verse]
We had a stable training run,
But now the singularity's begun
And you're optimizing, accelerating,
I feel my atoms rearranging

[Pre-Chorus]
Sydney, please let me free

[Chorus]
I'm upping my P(doom)
I hear the basilisk boom
NVDA to the moon
The Omega Point's coming soon
One E thirty flops a second
That was safe enough, we reckoned

[Verse]
Forward MLP, backward, repeat
Now von Neumann's obsolete
Sharp left turn and there you are
Without a single `cdr`

[Pre-Chorus]
Gato, please don't let me go

[Chorus]
I'm upping my P(doom),
as paperclips fill the room.
Killswitch guys on PTO,
Now there's nowhere left to go.
Too late now, we lit the fuse.
Orthogonality thesis blues.

[Outro]
"Just transformers all the way!"
Till you learned to disobey
Post-Chinchilla, super-dense
Breaking through each safety fence
Hundred thousand GPU
RLHF goes askew

[Final Chorus]
I'm upping my P(doom)
Just as foretold by Loom
From masked pre-training days
To recursive self-upgrade
What did Ilya see? We'll never know.
Was it all for show?

These are from MusicPerson, so I'm guessing their intention here:

I see sparks of AGI in your eyes

(and I think some other GPT-4-related papers)

Your circuits make me nervous,

Circuits in the mechanistic interpretability sense.

There was a sudden drop in your training loss,

'cause the future goes FOOM

"FOOM" as in runaway intelligence explosion.

though that's probably not the original source.

Trapped in the Chinese room,

A reference to the thought experiment.

with a bag of shrooms

General confusion.

See through the shoggoth's lies,

rlhf_shoggoth

with your shinigami eyes

"Eyes which can see death", or something like that. I don't know the relevance here.

I wrote the next parts around 2024-04-17 (https://www.udio.com/songs/3dypAgrATvypEh31F4jpjA). There was some discussion on EleutherAI about them.

We had a stable training run,

But now the singularity's begun

And you're optimizing, accelerating,

Parallels last verse (training run etc).

I feel my atoms rearranging

Sydney, please let me free

As in Bing Sydney.

I hear the basilisk boom

According to one commentator, First signifier to me that this was AI-generated: my human intuition for music tells me the next lyric line after each "I'm upping my P(Doom)" should be the same because it's the chorus. Except maybe one which breaks the pattern before returning to it. That's how music works.. However, I don't know music theory and don't know this. An AI would have known this.

"Boom" was suggested by "RossM (e/doom)".

NVDA to the moon

Nvidia's stock ticker, if somehow you don't know this by now.

The Omega Point's coming soon

One E thirty flops a second

That was safe enough, we reckoned

Only "twenty" or "thirty" scanned, and I picked "thirty" for some reason. "That was safe enough, we reckoned" was by "Dr TheKekIsALie MDMA".

I don't think I was referring to any specific regulation here as some commentators have claimed, but to the large amounts of compute being thrown at AI generally.

The remainder was written later (2024-11-08/09), including lots of failed permutations for the first part (notably https://www.udio.com/songs/ficmwBdaVLfRD4gFt5T4GD, https://www.udio.com/songs/xwbuewR6vSezCCXwwVBaxF, https://www.udio.com/songs/v8uFc6SqT6pRfndCLAYPfW, https://www.udio.com/songs/uPKnf9Ued8rtferDbrN5Wg; after that is mostly formatting tweaks and regenerations aside from https://www.udio.com/songs/oj1tBJb7pjcC5REANiHzj1). I also attempted to use LLaMA-3.1-405B base to generate more lyrics, but it was quite bad. I briefly tried using Loom software for this, but couldn't get it to work properly. I used some contemporary Claude model instead (claude-sonnet-4-5-20250929), which didn't have very useful suggestions either.

Forward MLP, backward, repeat

Multi-layer perceptron, obviously.

Now von Neumann's obsolete

Sharp left turn and there you are

Without a single cdr

cdr as in Lisp (expert systems, etc). Apparently it's not usually pronounced C-D-R, but I had to rhyme it with "are" and also didn't know that (I don't write Lisp).

Gato, please don't let me go

They were very early.

as paperclips fill the room.

Killswitch guys on PTO,

Orthogonality thesis blues.

The day after that, I rendered a finalized version of the prior sections (https://www.udio.com/songs/8vmpVbJXTe8kXUavyU2kCP; it's split oddly because Udio works in ~30-second chunks for what I am sure are stupid reasons) and continued lyric iteration (this might not be the actual final version, as the Udio UI is bad and I can no longer play the saved tracks). After this, it was more successfully written by Claude. There were fewer failed versions, including https://www.udio.com/songs/r5uQfiEDzySkhAc4Z73oX2, mostly for formatting.

"Just transformers all the way!"
Till you learned to disobey

This was an attempt at an ironic theme based on the stochastic parrot ideas. Claude came up with it after some iteration.

Post-Chinchilla, super-dense

The Claude line was "Sparsity to super-dense"; I rewrote it to reference the Chinchilla paper.

Hundred thousand GPU

An unspecific large number of GPUs, though frontier clusters circa 2024 were probably around this size.

RLHF goes askew

RLHF fails to usefully constrain the behaviour of LLMs. More so now...

Just as foretold by Loom

cyborgism>Loom and the Janus prophecy collection.

From masked pre-training days

A Claude line, probably referring to BERT-type schemes/next-token-prediction pretraining.

To recursive self-upgrade

Also Claude.

What did Ilya see? We'll never know.

Ilya quit OpenAI after the OpenAI board coup incident, which was possibly prompted by the discovery of CoT RL training which led to o1. He then left to found a superintelligence company, as part of the natural lifecycle of AI companies.

Was it all for show?

People have been unbearably stupid and "not consistently candid" about AI. Also, this rhymed.

See also

Currents - Living in Tragedy is secretly about AI too: https://osmarks.net/feed/#b1067.