You say:

​

>CAIS also assumes people won’t build generalist agents to start with.

​

No, it doesn't. See, for instance, section 7: "Training agents in human-like environments can provide useful, bounded services":

​

>Training agents on ill-defined human tasks may seem to be in conflict with developing distinct services provided by agents with bounded goals. Perceptions of conflict, however, seem rooted in anthropomorphic intuitions regarding connections between human-like skills and human-like goal structures, and more fundamentally, between learning and competence. These considerations are important to untangle because human-like training is arguably necessary to the achievement of important goals in AI research and applications, including adaptive physical competencies and perhaps general intelligence itself. Although performing safely-bounded tasks by applying skills learned through loosely-supervised exploration appears tractable, human-like world-oriented learning nonetheless brings unique risks.

​

You say:

​

>if you don’t think a LLM can become dangerous you aren’t thinking hard enough.

​

Any AI can be dangerous depending on factors like its training data, architecture and usage context. That said, LLM as currently understood have a well defined way to produce and compare next token candidates, and no intrinsic tendency to improve on this routine by gathering computing resources or any similar instrumental goals, and simply adding more computing power and training data doesn't change that.

Gato and similar systems are interesting but at the end of the day, the architecture behind useful real-world AIs like Tesla's Autopilot is more suggestive of CAIS than of Gato, and flexibility, adaptability and repurposing are achieved through good old abstraction and decoupling of subsystems.

The advantages of generalist agents are derived from transfer learning. But this is no panacea, for instance, in the Gato paper they admit it didn't offer much advantage when it comes to playing Atari games, and it has obvious costs and drawbacks. For one, the training process will tend to be longer, and when something goes wrong you may need to start over from scratch.

And I must say, if I'm trusting an AI to drive my car, I'd actually prefer it if this AI's training data did NOT include videogames like GTA or movies like, say, Death Proof or Christine. In general, for many potential applications it's reassuring to know that the AI simply doesn't know how do certain things, and that's a competitive advantage in terms of popularity and adoption, regardless of performance.

​

You say:

>Narrow agents can also become dangerous on their own because of instrumental convergence

​

Yes, under some circumstances, and conversely, generalist agents can be safe as long as this pesky instrumental convergence and other dangerous traits are avoided.

There's a lot more to CAIS than "narrow good, generalist bad". In fact, many of Drexler's most compelling arguments have nothing to do with specialist Vs generalist AI. For instance, see section 6: "A system of AI services is not equivalent to a utility maximizing agent", or section 25: "Optimized advice need not be optimized to induce its acceptance".

Yes, which implies he doesn't believe his approach would work, like I said.

You don't have to take *me* seriously, but you should certainly read an FHI technical report before you take the liberty to yawn at it.

I don't keep up with every blogger who writes about AI alignment (which you stubbornly keep assuming to be the crux of all AI security) but I've been reading Eliezer and Nick Bostrom for long enough to know that their approach can't work, and now Eliezer seems to agree with that conclusion.

From skimming through your blog post it's quite clear you really need to read and try to understand Drexler's FHI report. For instance, your claims about tool AIs Vs agent AIs are irrelevant because the idea is not to avoid agent AIs, only the "friendly AI" paradigm. Also, you'd know that Drexler's paradigm is a natural extension to current best practices in AI design, not just for some abstract big-picture AI security but also for expediency in development of AI capabilities and more mundane safety concerns. So it's the exact opposite of what you claim: the "friendly AI" paradigm is the alien, unwelcome newcomer that wants to turn the AI research community on its head for dubious reasons, while Drexler tells them to keep doing what they are doing.

There's no need to "align an AGI". That's in fact the whole point, and you missed it.

I respect the fact that he started thinking seriously about AI security early on, but he got stuck in a dead end paradigm of AI security ("friendly AI") where we assume the AI will be all-powerful and our only hope is for its goals to be perfectly aligned with ours. As the flaws of this approach became increasingly apparent, instead of considering other AI security paradigms he just became increasingly pessimistic. He also seems to have a pretty big ego, which prevents him from reconsidering.

You'll be fine but it's a long, dry book. I'd start with "Engines of Creation", which is way more fun to read and provides all the basic notions. The Wikipedia entry mentions an "updated version" (from 2007), feely available online. The link is to a web archive of a pdf, but it works. I only recall reading the 1986 version, which I think is still very much relevant despite its age. A more recent introductory text by Drexler is Radical Abundance (2013), but I haven't read it. I say, read Engines, then skim through Nanosystems and keep it for reference, and get deeper into sections you find particularly interesting or where some frequent doubt or objection is addressed.

That's a reasonable question. There are a few, very limited experimental examples of positionally controlled chemical reactions. Regarding more general capabilities that may be available in the future, here's, for instance, an interesting and relevant peer-reviewed theoretical analysis:

http://www.molecularassembler.com/Papers/TarasovFeb2010.pdf

That link is from Freitas's website. You can also see the abstract in the publisher's site, but the full text seems to be paywalled:

http://www.aspbs.com/ctn/contents-ctn2010.htm#v7n1

https://www.ingentaconnect.com/content/asp/jctn/2010/00000007/00000002/art00002;jsessionid=132a35vdij2o1.x-ic-live-02

Yes, I see no other way to put it. To be fair, this is more the rule than the exception. I've seen just as bad or worse from people who should have (and probably did) know better. The fact that big bucks from the National Nanotechnology Initiative were at stake sure didn't help. See, for instance:

https://en.wikipedia.org/wiki/Drexler%E2%80%93Smalley_debate_on_molecular_nanotechnology

That's a really bad short article. Actual nanobot designs don't have "small metallic arms and claws", they are not made of metal but dense covalent macromolecules (usually "diamondoid", ie substituted diamond-like lattices), or some variation on graphene.

Proposals by Drexler, Freitas, Merkle and others in the field (as opposed to Sci-Fi BS) generally have been tested with the same ab initio quantum chemistry and molecular mechanics tools used by computational physical chemists to study and design real chemical reactions, later corroborated with experimental data.

Unfortunately, much of the introductory material is very dated, especially in style and presentation. Probably a good place to start for the technically inclined in Drexler's MIT dissertation, which is the basis for the book Nanosystems (one of the best sources), and can be freely downloaded here:

https://dspace.mit.edu/handle/1721.1/27999

​

Some chapters of Nanosystems are also available online:

https://www.zyvex.com/nanotech/nanosystems.html

​

Then you can check out some videos on current experimental research at the Foresight website.

https://foresight.org/technologies/nanotech-molecular-machines/

Freitas's and Merkle's websites look very dated but they contain some interesting links.

http://www.rfreitas.com/

http://www.ralphmerkle.com/

I hope that helps.

That's a bit like someone in the early 20th century wondering whether cars, plastics and chemical fertilizers will be reserved for the elites, while the poor keep riding horses, using wooden and metal objects and eating organic food. See how quaint that concern sounds? In a similar way, most post-singularity wonders we anticipate, like enduring perfect health and youth and a seemingly luxurious lifestyle , will be cheap parlor tricks nobody will care about, let alone trying to keep them away from the masses.

And to what end would they even try, when AI agents will be better servants than any human slave ever could? In fact, the main risk involving evil elites is that they may simply decide to get rid of everyone else, but they just won't have the political power to do that.

I do expect that there will be conflict about how resources are used, what risks are deemed acceptable and what governance structures and bodies should call the shots, but the relevant issues will be on a scale spanning the whole galaxy and beyond and millions of years into the future.

In theory there must be a limit defined by the laws of physics, but we haven't even scratched the surface. If there's a limit to what's physically possible, it doesn't look like a cooler phone or a better therapy for some types of cancer, it looks like artificial superintelligence, full-fledged molecular nanotechnology and atomically precise manufacturing as described by Drexler and others, mind uploading and radical enhancement, matrioshka brains, cosmological lifespans, Von Neumann probes all over the galaxy and beyond, that kind of thing, and probably lots of stuff we haven't even thought of yet.

You are not missing the point. There's a myth going around in health-related media that too much testing can be a bad thing. Assuming the tests themselves are harmless (and they do, that's not the issue), that's total nonsense, of course, because you can always get the "less testing" alleged advantage by simply not reading the results of some tests.

IIRC, the source of this myth is a study in the UK on the effects of a program for mass periodic testing for uterine cancer (or something like that). The tests were done, I think, every 3 years or so, which meant that most detected tumors were benign, because most aggressive tumors gave symptoms before they were detected in routine tests. So many women were having surgeries they didn't need, and few benefited from early detection of aggressive tumors. I'm quoting from memory, so take it with a grain of salt, but I think that's the gist of it.

You see how dumb it is to say this was a problem of "too much testing", right?

Given current trends, I'm pleasantly surprised it's not about turning it into feed for cattle or even food for humans.

The problem I see with this is that you are using a powerful, sophisticated technology to trick your brain into believing a lie, the lie that you are talking to your dead loved ones rather than to a bunch of chatbots made to superficially sound a bit like them based on a few questions, recordings and the like. This can only end in two ways: either you permanently erase the distinction between truth and lies and lose your mind, or you come back to reality with fresh grief and disappointment after each conversation.

Some point out that people often cling to old pictures, letters, personal belongings and video recordings, and these chatbots can be seen as an extension of that practice. Well, for starters, that should be done in moderation too, but more to the point, those things don't mess with your mind nearly as much because they don't try to simulate a fresh interaction.

The only thing stopping some countries from having nuclear weapons is other countries with nuclear weapons. Maybe not all countries will have killer drones, but some probably will. Yes, in theory you could use nuclear blackmail to enforce a global ban on killer drones, for a while, but at some point someone will call the bluff. Nuclear weapons are just too blunt as a pressure tool, and military drones make too much sense to stay banned for long, being so versatile and precise in avoiding civilian casualties, especially compared to nuclear weapons.