ihateshadylandlords t1_iv3z7of wrote on November 5, 2022 at 2:43 AM

Reply to comment by ReadSeparate in How do you think an ASI might manifest? by SirDidymus

> Even if an ASI is an oracle alignment is still just as much of an issue. It can tell them to do something that sounds completely harmless to even the smartest of humans and even non-ASI AGIs, but in reality lets it out of the box.

You’re assuming the ASI will be sentient. Teams are doing everything to ensure it’s not sentient.

> What do you mean? That's exactly what ASI is. We're talking about something orders of magnitudes more intelligent than Albert Einstein here. A machine like that will be capable of recursively improving its own intelligence at an insane rate and will eventually know how to achieve any goal compatible with the laws of physics in the most efficient way possible for any possible set of constraints. That is basically by definition a magical genie that can do anything in a split second.

Okay. Then the owners will probably use this non-sentient tech to take care of themselves and the rest of us next.

ReadSeparate t1_iv3zj0j wrote on November 5, 2022 at 2:45 AM

I’m not assuming it’ll be sentient, I’m just saying an Oracle ASI equally as dangerous as one with agency. It MIGHT be sentient. Or it might NOT be sentient, but still dangerous, I.e. the paper clip maximizer scenario.

> Okay then the owners will probably use this Non-sentient tech to take care of themselves

Like just AGI you mean? Yeah I agree with that of course. But ASI, again, seems short sighted. If Google makes human level AGI, but it’s just as smart as say Einstein, yeah of course they’ll use it to get richer. But if they create something that makes Einstein look like an ant, they’d be foolish to use it in such a way.

ihateshadylandlords t1_iv40poj wrote on November 5, 2022 at 2:56 AM

> I’m not assuming it’ll be sentient, I’m just saying an Oracle ASI equally as dangerous as one with agency. It MIGHT be sentient. Or it might NOT be sentient, but still dangerous, I.e. the paper clip maximizer scenario.

Meh, the dangers of an ASI can be discussed in another thread. We were initially talking about how an ASI might manifest, so it’s getting off course.

>Like just AGI you mean? Yeah I agree with that of course. But ASI, again, seems short sighted. If Google makes human level AGI, but it’s just as smart as say Einstein, yeah of course they’ll use it to get richer. But if they create something that makes Einstein look like an ant, they’d be foolish to use it in such a way.

Okay. Just don’t be surprised if companies keep doing what they’ve been doing for literally thousands of years and use their products to make a profit.