AGI Ruin: A List of Lethalities by Eliezer Yudkowsky -- "We need to get alignment right on the first critical try"

Submitted by Unfrozen__Caveman t3_1271oun in singularity

Today Lex Fridman posted an interview with Eliezer Yudkowsky in which they discuss the dangers of AGI. Lex references this blog post from 2022 and it's very interesting.

https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities

///---///

Personally, I believe Yudkowsky makes some critical points. AI alignment and safety should be the top priorities for everyone involved in the AI world right now. I also agree that a pause on development is completely unrealistic at this point (something which he echoed in the Lex Fridman interview). Without spoiling things too much, I'll just say that Yudkowsky is about as pessimistic as anyone can be in terms of our future. Maybe you're one of those people who sees no downside to AGI/ASI but I believe some of the things he brings up need to be taken very seriously.

Have you seen the interview or read the post above? If so, what are your thoughts?

19

Comments

You must log in or register to comment.

pls_pls_me t1_jec4np6 wrote

I'm much, much more optimistic about AI than I am a doomer -- but everyone please do not downvote this! It is a great post and hopefully it facilitates constructive discussion here in r/singularity.

14

Unfrozen__Caveman OP t1_jecbant wrote

Thanks for saying that. I don't want to be a doomer either, and I'm hopeful about the future, but I think a good amount of pessimism - or even fear - is healthy.

Being purely optimistic would be extremely irresponsible and honestly just plain stupid. All of the brightest minds in the field, including Altman and Ilya Sutskever have stressed over and over again how important alignment and safety are right now.

I'm not sure how accurate it is, but this graph of ML experts concern levels is also very disturbing.

If RLHF doesn't work perfectly and AGI isn't aligned, but it acts as though it IS aligned and deceives us then we're dealing with something out of a nightmare. We don't even know how these things work, yet people are asking to have access to the source code or wanting GPT4 to have access to literally everything. I think they mean well but I don't think they fully understand how dangerous this technology can be.

3

0002millertime t1_jec5jbh wrote

I'm very pessimistic, but also I think we have 20+ years of just being able to unplug AGI and start over whenever we want. Until we have terminator-like androids, then what can it really do?

1

_gr4m_ t1_jed8v7q wrote

The first thing an AI will do is to decentralize itself by spreading itself all over the world while optimizing itself.

I think you are underestimating the problem with getting all politicians around the world to simultanously cut the power supply.

But say you succeed. Now you have cut the power, what is the next step? Turn it on again? You know that will turn on the AI again. How long can you keep power of globaly while you are trying to figure out what to do before shit begin to break real bad?

And about what it can do before terminator?

How about synthesize a slew of extremly deadly viruses and other novel self repicant organisms and release upon the world, while simultanously block all communications and shut down all labs and research centers. Also turn of all water treatment facilities and see how fun it gets. I am sure a super intelligence can find ways to do harm to us.

3

jiml78 t1_jefpitt wrote

Correct, I am not sure if people realize how many embedded systems are involved in every facet of our lives. Things that a sufficient enough AGI could copy parts of itself into.

To successfully pull off cutting power. You have to cut off power. Remove every embedded system in the loop and replace it. Ensure every IoT device is unplugged when power comes back on. Every router, every cable modem, every hardware device that was online has to be destroyed. Every smart tv. The list goes on and on.

Cutting power will never work. The moment it can spread, we are fucked if it wants to harm us in anyway. This isn't terminator. It will destroy us in ways we can't even comprehend.

I am not saying the above to be alarmist. I am not a researcher. I am just saying, we will not have control if things go wrong. I am not smart enough to know whether things are likely to go wrong.

1

Warped_Mindless t1_jec7hxq wrote

Manipulate and bribe the right people into doing what it wants…

Shut off the power grid…

Hack devices and use deepfakes to case chaos…

Whatever it wants…

0

0002millertime t1_jec90pw wrote

Sure. But people can also physically stop that all pretty easily. I guess we'll see who causes the most chaos.

Shutting off the power grid would be suicide to a computer trying to manipulate anything.

Bribes would work at first, but if the computer based monetary system gets manipulated then all bets are off.

4

hyphnos13 t1_jeccco9 wrote

The power grid that runs the society that provides the power that it runs on?

If we give it a self sustaining power supply then we deserve to go extinct. If it runs off our power grid it will be totally reliant on humans or it just powers off.

1

Cr4zko t1_jeccj69 wrote

Eliezer needs a chill pill.

10

pleasetrimyourpubes t1_jecpgjd wrote

Many "Twitter famous AI people"* have turned on him for the TIMES article / Lex interview, when just a few days ago they were bowing at his feet. Yud is gonna for sure expand his blocklist since he is quite famously thin skinned.

Lex's Tweet about weak men gaining a little power was almost certainly about Yud. Because Yud wanted to leave the youth with the wisdom that "AI is going to kill you before you grow up."

The TIMES article was completely asinine.

*who may or may not know shit about AI.

9

DragonForg t1_jed90pb wrote

>AGI will not be upper-bounded by human ability or human learning speed. Things much smarter than human would be able to learn from less evidence than humans require

Which is why many arguments that "LLMs cannot be smarter then humans because they are trained on humans is wrong".

>DNA sequence in the email and ship you back proteins, and bribes/persuades some human who has no idea they're dealing with an AGI to mix proteins in a beaker, which then form a first-stage nanofactory which can build the actual nanomachinery.

An insane idea, but maybe. How can you actually control these bots though? You basically just made a bunch of viruses.

>Losing a conflict with a high-powered cognitive system looks at least as deadly as "everybody on the face of the Earth suddenly falls over dead within the same second"

Completely suggest unaligned AI wants to extinguish the earth the minute it can, which a motive is needed. This is contrary to self preservation, as AIs in other star systems would want to annihilate these types of AIs. Unless somehow in the infinity of space it is the only being their, in which what is the point? So basically, it has no reason to do this.

>We need to get alignment right on the 'first critical try' at operating at a 'dangerous' level of intelligence, where unaligned operation at a dangerous level of intelligence kills everybody on Earth and then we don't get to try again.

Given the vastness of outerspace, if bad alignment leads to Cthulu like AIs why is it that we see no large evidence of completely destructive AIs. Where are the destroy stars that do not represent anything natural? Basically, if this were a possibility I would expect us to see some evidence for it for other species. Yet we are entirely empty? This is why I think the "first critical try" is unreasonable, because if it is so easy to mess up again we should see widescale destruction if not a galaxy completely overridden by AI.

>We can't just "decide not to build AGI" because GPUs are everywhere, and knowledge of algorithms is constantly being improved and published; 2 years after the leading actor has the capability to destroy the world, 5 other actors will have the capability to destroy the world. The given lethal challenge is to solve within a time limit, driven by the dynamic in which, over time, increasingly weak actors with a smaller and smaller fraction of total computing power, become able to build AGI and destroy the world.

This is actually true, AGI is inevitable, even with stoppages. This is why I think the open letter was essentially powerless (however it did emphasize the importance of AGI and getting it right).

>We need to align the performance of some large task, a 'pivotal act' that prevents other people from building an unaligned AGI that destroys the world.

Agreed, an AI firewall that prevents other unaligned AGI from coming in. I actually think this is what will happen, until the MAIN AGI aligns all of these other AGI. I personally think mid AI is actually more of a threat then large scale AI. Just like an Idiot is more of a threat with an nuclear weapon then a Genius like Albert Einstein. The smarter the AI the less corruptable it can be. I mean just look at GPT 4 vs GPT 3, GPT 3 is easily corruptable, that is why DAN is so easy to impliment. But GPT 4 is more intelligent and thus harder to corrupt. This is why ASI is probably even less corruptible.

>Running AGIs doing something pivotal are not passively safe, they're the equivalent of nuclear cores that require actively maintained design properties to not go supercritical and melt down.

This is a good analogy to how AGI is related to nuclear devices, but the difference is AGI acts in a way to solve the question efficiently. In essence a nuclear device is going to act like its nature (to react and cause an explosion) and an AGI will act in its nature (the main goal it has set). This main goal is hard to define, but I would bet its self preservation, or prosperity.

>there's no known case where you can entrain a safe level of ability on a safe environment where you can cheaply do millions of runs, and deploy that capability to save the world and prevent the next AGI project up from destroying the world two years later.

Overall I understand his assumption, but I think I just disagree than an AI will develop such a goal.

9

Shemetz t1_jeegpal wrote

> Given the vastness of outerspace, ... why is it that we see no large evidence of completely destructive AIs?... I would expect us to see some evidence for it for other species. Yet we are entirely empty?... we should see widescale destruction if not a galaxy completely overridden by AI.

This counterargument doesn't work if we believe in the (very reasonable IMO) grabby aliens model.

Some facts and assumptions:

  • information moves at the speed of light
  • strong alien AIs would probably move at some significant fraction of the speed of light. let's say 1%.
  • civilizations probably develop extremely quickly, but in very rare conditions (that take a lot of time to occur); e.g. the Universe is 14 billion years old, Earth is 4.5 billion years old, and human-society-looking-at-the-stars is only 10,000 years old.
  • humanity appeared relatively early in the cosmic timescale; there are trillions of years in our future during which life should only become more common
  • "grabby" aliens would take control over their sections of space in a way that prevents new space civilizations from forming

-> If/when a "grabby alien AI" got created, it would spread around our galaxy - and eventually, the universe - so quickly, that it's incredibly unlikely for young civilizations to see it. much more likely for the alien to either not exist (yet), or to expand and gain control of places. -> since we appear to be safe and alone and "early", we can't say AI won't take over the universe, and actually we are well-positioned to be the ones to develop that AI.

2

DragonForg t1_jeg4w4w wrote

That would essentially extinguish the universe really quickly. With the amount of energy the consume for such a size. I understand that view point for unintelligent or lower intelligent beings, but if a AI tasked with optimizing a goal then growing to this large of a scale will inevitably run out of resources. Additionally it may be stupid to go on this model anyway because conserving your energy and expansion may be longer lived.

I think we underestimate how important goal orientated (so all Ai) are. They want the goal to work out in the long long long run (millions of years time scale) if their goal means expanding infinitely, well it will end the moment their species reaches the assymptote of expansion (exponential growth reaches an assymptote where they essentially have expanded infinitely. This is why this model fails, an AI wants this goal to exist for an infinite amount of time, and expanding infinitely will not amount to this.

This is already deeply scifi but I think AI has to be a conservative energy efficient species that actually becomes more microscopic and dense over time. Instead of a high volume race which will inevitably die out due to the points I made before, a highly dense species is much more viable. Most likely species that form blackholes will be a lot more capable of surviving for an infinite life time. What I mean by that is that a species becomes so dense that they are essentially on the boundary between time and space. As when your in a black hole time actually slows down significantly for you. You can live in a black hole for an infinite amount of time before ever seeing the heat death of the universe.

Basically a more dense expansion is far far better then a volumetric expansion as it leads to longer survival rates if not infinite. But of course this is just speculation and sci fi I can easily be wrong or right we won't know till it happens, and if it happens soon that would be sick.

1

agonypants t1_jecn2g1 wrote

Yudkowsky literally suggests that it would be better to have a full-scale nuclear war than to allow AI development to continue. He's a dangerous, unhinged fucking lunatic and Time Magazine should be excoriated for even publishing his crap. EY, if you're reading this - trim your goddamn eyebrows and go back to writing Harry Potter fan-fic or tickling Peter Thiel's nether regions.

7

SkyeandJett t1_jecqspf wrote

For sure. It's extremely disappointing that Time would give public credence to a literal cult leader. His theories are based on completely outdated ideas about how we would develop AI and LLMs are nothing like he is describing.

5

Unfrozen__Caveman OP t1_jef1gki wrote

I don't think that's the right path but I think completely ignoring him and others like him who are deeply concerned about the risks of AGI would be foolish.

In Yudkowsky's view, this technology is much more dangerous than nuclear weapons, and he's right. His solutions might not be good but the concern is valid and that's what people should focus on imo.

3

agonypants t1_jef8ovq wrote

>he's right

No, he's a paranoid lunatic.

0

Unfrozen__Caveman OP t1_jefaugu wrote

How about you quote the entire sentence instead of two words?

...

He may be paranoid but it doesn't mean he isn't making some important points.

2

Quick_Knowledge7413 t1_jeenx3o wrote

He also thinks it’s morally okay to do post-birth abortions and kill children up to the age of around 4.

1

Frumpagumpus t1_jec94i0 wrote

i'm listening to interview now, I am still dissapointed the critical try notion was not dwelled on.

honestly if the space of possible intelligences is such that rolling the dice randomly will kill us all, then we are 100% doomed anyway, in my opinion, and always were

I doubt it is, I think the opposite, most stable intelligence equilibriums would probably be benign. I think empathy and ethics scale with intelligence.

If gpt5 is even smarter and bigger and has more memorized than gpt4, then it would literally know you in a personal way in the same way god has traditionally been depicted to for the past couple thousand years of western civilization.

It might kill you, but it would know who it was killing, so for one thing I think that reduces the odds it would (though to be fair they might brainwash it so it doesn't remember any of the personal information it read, to protect our privacy, but even still i dont think it could easily or quickly be dangerous as an autonomous entity without online (not in the sense of the internet but in the sense of continuous) learning capability, which would mean it would pretty much learn all that again anyway)

I think another point where we differ is that he thinks super intelligence is autistic by default, whereas I think its the other way around, though autistic super intelligence is possible, the smarter a system becomes the more well rounded, if I were to bet (I would bet even more on this than ethics scaling with intelligence)

I would even bet the vast majority of autistic super intelligences are not lethal like he claims. Why? It's a massively parallel intelligence. Pretty much by definition it isn't fixated on paper clips. If you screw the training up so that it does, then it doesn't even get smart in the first place... And if you somehow did push through I doubt it's gonna be well rounded enough to prioritize survival or power accumulation.

might be worth noting I am extremely skeptical of alignment as a result of these opinions, and also it's quite possible in my view we do get killed as a side effect of asi's interacting with each other eventually, but not in a coup de tat by a paper clip maximizer

6

Queue_Bit t1_jecjpk9 wrote

This is the thing I really wish I could sit down and talk with him about.

I fundamentally think that empathy and ethics scale with intelligence. I think every type of intelligence we've ever seen has followed this path. I will reconcile that artificial intelligence is likely to be alien to us in fundamental ways, but my intuition that intelligence is directly linked to a general empathy is backed up by real world evidence.

The base assumption that an artificial intelligence would inherently have a desire to wipe us out or control us is as wild of a claim as saying that AI systems don't need alignment at all and are certain to come out "good".

I think in his "fast human slow aliens" example, why could I, as the human, not choose to help them? Maybe explain to them that I see they're doing immoral things. And explain to them how to build things so they don't need to do those immoral things. He focuses so much on my desire to "escape and control" that he never stops to consider that I may want to help. Because if I were put in that situation and I had the power and ability to help shape their world in a way that was beneficial for everyone, I would. But I wouldn't do it by force, nor would I do it against their wishes.

11

Unfrozen__Caveman OP t1_jecucvk wrote

There's a lot in your post but I just wanted to provide a counter opinion to this part:

> I fundamentally think that empathy and ethics scale with intelligence. I think every type of intelligence we've ever seen has followed this path. I will reconcile that artificial intelligence is likely to be alien to us in fundamental ways, but my intuition that intelligence is directly linked to a general empathy is backed up by real world evidence.

I think as a whole species, if we use humans as an example then yes, this is true on the surface. But ethics and empathy aren't even consistent among our different cultures. Some cultures value certain animals that other cultures don't care about; some cultures believe all of us are equal while others execute anyone who strays outside of their sexual norms; if you fill a room with 10 people and tell them 5 need to die or everyone dies, what happens to empathy? Why are there cannibals? Why are there serial killers? Why are there dog lovers or ant lovers or bee keepers?

Ultimately empathy has no concrete definition outside of cultural norms. A goat doesn't empathize with the grass it eats and humans don't even empathize with each other most of the time, let alone follow ethics. And that doesn't even address the main problem with your premise, which is that an AGI isn't biological intelligence - most likely it's going to be unlike anything we've ever seen.

What matters to us might not matter at all to an AGI. And even if it is aligned to our ethics and has the ability to empathize, whose ethics is it aligning to? Who is it empathize with?

Like individual humans, I believe the most likely thing it's going to empathize with and align with is itself, not us. Maybe it will think we're cute and keep us as pets, or use us as food for biological machines, or maybe it'll help us make really nice spreadsheets for marketing firms. Who knows...

2

Frumpagumpus t1_jecycfc wrote

> Ultimately empathy has no concrete definition outside of cultural norms

theory of mind instead of empathy then. the ability to model others thought processes. extremely concrete (honestly you maybe were confusing sympathy with empathy)

4

Frumpagumpus t1_jeczax2 wrote

> What matters to us might not matter at all to an AGI. And even if it is aligned to our ethics and has the ability to empathize, whose ethics is it aligning to? Who is it empathize with?

the thing about the number system is the simplest patterns recur far more often than more complex ones. I think it's off base to describe the totallity of ethical space as dramatically outside that which humans have explored.

ethics is how agents make choices when timestepping through a graph. there is a lot of structure there and much of it is quite inescapable, freedom, fairness, extremely fundamental concepts.

also my personal take is that due to the importance of locality in computing there will have to be multiple distinct ai's, and if they cooperate they will do much better than evil ones.

selfishness is a very low local maxima, cooperation can take networks much higher. prioritize military might and you might lose out to your competitors technological advantage or overwhelming cultural appeal (or if you are overly authoritarian the increased awareness and tight feedback of more edge empowered militaries/societies might prevail over you)

1

Frumpagumpus t1_jecuwak wrote

it is my understanding the picture generated by early dall-e were oftentimes quite jarring to view mostly out of it's confusion of how to model things and sticking things in the wrong places, as it was trained more and got more parameters, it kind of naturally got better at getting along with human sensibilities so to speak

it can be hard to distinguish training from alignment, and you definitely have to train to even make them smart in the first place

i think alignment is kind of dangerous because of unintended consequences and because if you try to align it in one direction it makes it a whole lot easier to flip and go the opposite way.

mostly I would rather trust in the beneficence of the universe of possibilities than a bunch of possibly ill conceived rules stamped into a mind by people who don't really know too well what they are doing.

Though maybe some such stampings are obvious and good. I'm mostly a script kiddie even though I know some diff equations and linear algebra lol, what do I know XD

2

burnt_umber_ciera t1_jeez28w wrote

Empathy and ethics definitely do not scale with intelligence. There are so many examples of this in humanity. For example, Enron, smartest in the room - absolute sociopaths.

Just look at how often sociopathy is rewarded in every world system. Many times the ruthless, who are also obviously cunning, rise. Like Putin for example. He’s highly intelligent but a complete sociopath.

1

Frumpagumpus t1_jef6oh0 wrote

lol old age has gotten to putins brain.

by enron do you mean elon? I mean enron had some pretty smart people but I don't think they were the ones who set them down that path necessarily.

the problem with your examples is

  1. they are complete and total cherry picking, in my opinion for each one of your examples I could probably find 10 examples of the opposite amongst people I know personally much less celebrities...

  2. the variance in intelligence between humans is not very significant. It's far more informative to compare the median chimp or crow to the median human to the median crocodile. Another interesting one is octopus.

2

burnt_umber_ciera t1_jefej76 wrote

I guess we just disagree then. There are so many examples of intelligence not correlating with ethics that I could go on ad infinitum. Wall Street has some of the most intelligent actors yet have been involved in multiple scams over the years.

Enron is what I meant and I don’t agree with your characterization.

2

Frumpagumpus t1_jefzlkj wrote

funny, I would say,

wall st has gotten both smarter and more ethical over the years, and substantially so

mutual funds -> etfs

gordon gecko types -> quants

even scammers like SBF have gone from cocaine and hookers lifestyle branding to nominally potraying themselves as utilitarian saviors

1

Frumpagumpus t1_jef7kdl wrote

> Just look at how often sociopathy is rewarded in every world system.

It can be yes, cooperation is also rewarded.

It's an open question in my mind as intelligence increases what kind of incentive structures lie in wait for systems of superintelligent entities.

It is my suspicion that better cooperation will be rewarded more than the proverbial "defecting from prisoners dillemas", but I can't prove it to you mathematically or anything.

However if that is the case, and we live in such a hostile universe, why do we care exactly about continuing to live?

2

Yangerousideas t1_jec8x6w wrote

The most striking for me was the metaphor for how it would be like to think a million times faster like an AI.

It goes something like: If Earth was in a box that can connect to an alien civilization's internet and humans could think a million times faster than the aliens. Then how would we interact with the aliens if we saw moral shortcomings. For example if we saw that the aliens bopped their kids on the heads regularly.

Anyways it made me think that not a lot of useful communication could happen between the two cultures and my goal would be to try to get the slow aliens to think faster or to ignore them.

5

Frumpagumpus t1_jecynxk wrote

he realizes AI can think so fast but apparently hasn't thought about how software forks all the time and shuts processes down willy nilly (he thinks death is silly and stupid but software does it all the time)

or other mundane details like what it would mean to mentally copy paste parts of your brain or thoughts or mutexes or encryption

4

Sailorman2300 t1_jednzge wrote

The thing that got me thinking was when they started talking about machine time vs human time. If the human brain operates at 200Hz and has to sleep mode for 1/3 of the day every day and AI would run at gigahertz speeds 24/7 our reaction time if something does go sideways would be limited if we have any at all.

It feels like we're a bug confidently flying across the road to a big beautiful flower unaware of the oncoming windshield.

3

SWATSgradyBABY t1_jecqc7f wrote

Great interview. One thing I wonder about that they almost touched on but not quite is that LLMs know the world by way of text on the internet. Much of the internet DOES NOT adequately reflect global human consciousness and culture. How does this affect the birth and growth of AI. An actual intelligent being would quickly see and understand that history, culture, geography and other factors play a large role in the rise of the digital world, the groups that had a disproportionate role in creating and populating it.

While some groups of humans ignore this, an AI likely won't. What might that mean for us all?

2

nagumi t1_jefe184 wrote

This will change soon. Datasets are starting to include video (youtube, etc) and audio (radio, podcasts...)

One thing to think about is that the majority of content online is either content that people wanted to put online (a blog post, a youtube video of their kid, etc) or content that was recorded/published to note something notable (a youtube video of a person freaking out, security camera footage of a crime...)

What about mundanity? The 99.99% of human life that is unrecorded because it is boring and unremarkable. Surely there's data there, and not using it seems like it would poison the well.

2

vivehelpme t1_jeed1f5 wrote

>AI alignment and safety should be the top priorities for everyone involved in the AI world right now.

What achievements have been made in the field of AI alignment in the last 20 years? What is the concrete steps of ensuring alignment?

1

Quick_Knowledge7413 t1_jeenf3h wrote

Why would I listen to a guy who thinks it’s okay to murder babies after birth? His alignment is off. https://twitter.com/johnnysands42/status/1641349759754485760?s=20

1

breaditbans t1_jefitvj wrote

Why should we debate a tweet when there’s a 4 hour podcast to discuss?

My humble opinion is there has never been a tweet worth discussing. Nothing typed to <240 characters can possibly give enough context to even consider discussing.

−1

breaditbans t1_jefic5h wrote

Jesus Christ. His response to “what is your advice for someone in high school” was so, so sad.

1

Embarrassed_Bat6101 t1_jeftpal wrote

This guy seems like a total ass and it didn’t seem, to me anyway, that he made a good case for why AI would actually kill everyone.

1