SuperintelligenceI’ve just finished reading Nick Bostrom’s
Superintelligence: Paths, Dangers, Strategies. I have a strong interest in the development of artificially intelligent machines in general and follow the weekly advances and announcements in the field of artificial intelligence as our species inches towards the ability to birth artificial general intelligence. I’ve read quite a bit of the philosophy concerning the arrival of machines smarter than ourselves and am generally familiar with the profound opportunities and risks associated with the arrival of such entities for humanity. But I consider myself loosely informed, not well informed, on this topic and so it was only a matter of time before I sat down with Nick’s book.
If you’re in a similar position to myself, have an interest in these topics, are curious about the rise of artificial intelligence, possess opinions on the likelihood of such intelligences arising or the chances of existential risks, then
Superintelligence should be relevant and of interest to you. If you’re a futurist or a transhumanist then
Superintelligence should be required reading.
Nick’s style of writing is, in a single word,
thorough. Nick writes with a level of rigour, clarity, and thoroughness that I have rarely encountered. This can be tedious at times and a bit of a slog but fortunately he punctuates the flow with moments of dry, cutting wit. Nick is verbose and possesses a vocabulary that, I have no doubt, exceeds your own. I don’t care who you are or your level of education you will learn at least a dozen (give or take) new English words that you have never encountered before. I opened / read all of the footnotes because 95% of the time Nick’s footnotes are incredibly informative and worth digging into; you should do the same. Chapters are well organised and the arguments and innumerable examples within are well crafted; just when you think a line of thought and argument sounds familiar and other critical points come to your mind independently from your own readings and you think of something Nick hasn’t thought of, he yet again proves to be one step ahead of you by articulating the argument in more detail than you’d seen before. Then you come across gems like this memorable sentence:
Anthropics, the study of how to make inferences from indexical information in the presence of observation selection effects, is another area where the choice of epistemic axioms could prove pivotal. and if you’re like me you refuse to give in. Instead you parse it, chunk it up like you did to simple sentences as a young child learning to read, hit Google and Wikipedia repeatedly, and re-read the thing a half-dozen times before the meaning finally becomes clear. Another thing that is clear throughout is Nick’s strong stance on ethics and morality.
Nick includes some two dozen notable luminaries in the acknowledgements that he consulted with in preparing this book, including superstars such as Demis Hassabis and Jurgen Schmidhuber to name just two. Also of note is Elon Musk, and I can’t help but wonder - given the timing - if this also influenced Elon’s recent comments on the perils and dangers of superintelligent machines.
Superintelligence weaves through everything from the history of artificial intelligence research to developing strategic options for moving safely forward. We get exposed to the main paths leading to superintelligence that include classic artificial intelligence, whole brain emulation, biomolecular cognition enhancers, brain-computer-interfaces, and the collective intelligence of human networks and organisations. We learn about the main forms of different superintelligence including speed superintelligence, collective superintelligence, and most importantly quality superintelligence and the key sources of advantage that superintelligences can and will possess. The timing, optimisation, and explosivity - and factors affecting each - of the kinetics of an intelligence explosion are discussed, as well as the nature of paths leading towards a decisive strategic advantage and subsequent omnipotent singleton. We get a crash-course in superintelligent cognitive superpowers, superintelligent will, and the differences between intelligence and motivation.
Nick devotes a lot of discussion to existential risk, catastrophes, malignant failure modes and key concepts such as perverse instantiation, infrastructure profusion, and mind crime. A key part of the book focuses on the control problem and the main considerations around capability control and motivation selection. We get an overview of the abilities, benefits, and risks of the four different superintelligences comprising oracles, genies, tools, and sovereigns and then delve into the effects on humans and human societies. One of the most important parts of the book concerns the value-loading problem, different methods to pre-load a superintelligence with appropriate human values and the hope for being able to offload this task to the superintelligence itself via indirect normativity and coherent extrapolated volition.
Obviously I thought this was a very worthwhile book and one that needed to be written. To finish I’ll end with a selection of passages from the text if you needed any further prompting.
ExcerptsThe availability of the brain as template provides strong support for the claim that machine intelligence is ultimately feasible. The further into the future we look, the greater the likelihood that the secrets of the brain’s functionality will have been decoded sufficiently to enable the creation of machine intelligence in this manner.With gene synthesis we could take the genome of an embryo and construct a version of that genome free from the genetic noise of accumulated mutations. If one wished to speak provocatively, one could say that individuals created from such proofread genomes might be “more human” than anybody currently alive, in that they would be less distorted expressions of human form.Far from being the smartest possible biological species, we are probably better thought of as the stupidest possible biological species capable of starting a technological civilization—a niche we filled because we got there first, not because we are in any sense optimally adapted to it.It seems fairly likely, however, that even if progress along the whole brain emulation path is swift, artificial intelligence will nevertheless be first to cross the finishing line: this is because of the possibility of neuromorphic AIs based on partial emulations.The simplest example of speed superintelligence would be a whole brain emulation running on fast hardware. An emulation operating at a speed of ten thousand times that of a biological brain would be able to read a book in a few seconds and write a PhD thesis in an afternoon. With a speedup factor of a million, an emulation could accomplish an entire millennium of intellectual work in one working day.Normal human adults have a range of remarkable cognitive talents that are not simply a function of possessing a sufficient amount of general neural processing power or even a sufficient amount of general intelligence: specialized neural circuitry is also needed. This observation suggests the idea of possible but non-realized cognitive talents, talents that no actual human possesses even though other intelligent systems—ones with no more computing power than the human brain—that did have those talents would gain enormously in their ability to accomplish a wide range of strategically relevant tasks. were we to gain some new set of modules giving an advantage comparable to that of being able to form complex linguistic representations, we would become superintelligent.On this view, our most celebrated philosophers are like dogs walking on their hind legs—just barely attaining the threshold level of performance required for engaging in the activity at all.The gap between a dumb and a clever person may appear large from an anthropocentric perspective, yet in a less parochial view the two have nearly indistinguishable minds. It will almost certainly prove harder and take longer to build a machine intelligence that has a general level of smartness comparable to that of a village idiot than to improve such a system so that it becomes much smarter than any human.Superintelligent superpowers: Intelligence amplification, Strategizing, Social manipulation, Hacking, Technology research, Economic productivity. Superpowers are possessed by an agent as superpowers only if the agent’s capabilities in these areas substantially exceed the combined capabilities of the rest of the global civilization.The AI develops a robust plan for achieving its long-term goals. (In particular, the AI does not adopt a plan so stupid that even we present-day humans can foresee how it would inevitably fail. This criterion rules out many science fiction scenarios that end in human triumph.)In other words, assuming that the observable universe is void of extraterrestrial civilizations, then what hangs in the balance is at least 10,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000 human lives (though the true number is probably larger). If we represent all the happiness experienced during one entire such life with a single teardrop of joy, then the happiness of these souls could fill and refill the Earth’s oceans every second, and keep doing so for a hundred billion billion millennia. It is really important that we make sure these truly are tears of joy.The instrumental convergence thesis holds that superintelligent agents having any of a wide range of final goals will nevertheless pursue similar intermediary goals because they have common instrumental reasons to do so.One feature of a malignant failure is that it eliminates the opportunity to try again. The number of malignant failures that will occur is therefore either zero or one.Examples of perverse instantiation show that many final goals that might at first glance seem safe and sensible turn out, on closer inspection, to have radically unintended consequences. If a superintelligence with one of these final goals obtains a decisive strategic advantage, it is game over for humanity.A small error in either the philosophical account or its translation into code could have catastrophic consequences. Consider an AI that has hedonism as its final goal, and which would therefore like to tile the universe with “hedonium” (matter organized in a configuration that is optimal for the generation of pleasurable experience). To this end, the AI might produce computronium (matter organized in a configuration that is optimal for computation) and use it to implement digital minds in states of euphoria.Indirect normativity is a very important approach to motivation selection. Its promise lies in the fact that it could let us offload to the superintelligence much of the difficult cognitive work required to carry out a direct specification of an appropriate final goal.It might be hard to ensure that a complex, evolved, kludgy, and poorly understood motivation system, like that of a human being, will not get corrupted when its cognitive engine blasts into the stratosphere.While one might consider creating a physically confined genie, for instance one that can only construct objects inside a designated volume—a volume that might be sealed off by a hardened wall or a barrier loaded with explosive charges rigged to detonate if the containment is breached—it would be difficult to have much confidence in the security of any such physical containment method against a superintelligence equipped with versatile manipulators and construction materials.Some emulations may prefer to retain most of their functionality and handle tasks themselves that could be done more efficiently by others. Those emulations would be like hobbyists who enjoy growing their own vegetables or knitting their own cardigans. Such hobbyist emulations would be less efficient; and if there is a net flow of resources from less to more efficient participants of the economy, the hobbyists would eventually lose out. The bouillon cubes of discrete human-like intellects thus melt into an algorithmic soup.We could thus imagine, as an extreme case, a technologically highly advanced society, containing many complex structures, some of them far more intricate and intelligent than anything that exists on the planet today—a society which nevertheless lacks any type of being that is conscious or whose welfare has moral significance. In a sense, this would be an uninhabited society. It would be a society of economic miracles and technological awesomeness, with nobody there to benefit. A Disneyland without children.Human nature, after all, is flawed and all too often reveals a proclivity to evil which would be intolerable in any system poised to attain a decisive strategic advantage.No ethical theory commands majority support among philosophers, so most philosophers must be wrong. It is also reflected in the marked changes that the distribution of moral belief has undergone over time, many of which we like to think of as progress . . . Very likely, we are still laboring under one or more grave moral misconceptions.Indirect normativity is a way to answer the challenge presented by the fact that we may not know what we truly want, what is in our interest, or what is morally right or ideal. Instead of making a guess based on our own current understanding (which is probably deeply flawed), we would delegate some of the cognitive work required for value selection to the superintelligence.Our coherent extrapolated volition is our wish if we knew more, thought faster, were more the people we wished we were, had grown up farther together; where the extrapolation converges rather than diverges, where our wishes cohere rather than interfere; extrapolated as we wish that extrapolated, interpreted as we wish that interpreted.The point of superintelligence is not to pander to human preconceptions but to make mincemeat out of our ignorance and folly.Before the prospect of an intelligence explosion, we humans are like small children playing with a bomb. Such is the mismatch between the power of our plaything and the immaturity of our conduct. #superintelligence #artificialintelligence #existentialrisk
