On the Crisis Brought About by Artificial Intelligence
Imagine that computer scientists succeed in developing intelligent machines—AGI—that can perform every task more competently than humans. All work would then be carried out by a vast, highly organized system of machines, making human effort unnecessary. There would be two possibilities: either machines would be permitted to make their own decisions without human supervision, or humans would retain control over them. All we can point out is that humanity's fate would depend on machines. Someone might argue that humanity would not be foolish enough to hand all its power over to machines. But the point is not that humanity would voluntarily surrender its power, nor that machines would deliberately attempt to seize it.
The point is that humanity could readily let itself become so dependent on machines that eventually it would have no choice but to accept every decision they make. As society and the problems it faces become more complex, and machines become more intelligent, people will allow machines to make all the decisions because those decisions will produce better results than human decisions. Eventually, the decisions needed to run the system might become so complex that humans can no longer make them intelligently. At that stage, machines would be in control. People would not even be able to switch them off, because they would be so dependent on them that doing so would amount to suicide.
On the other hand, it might be possible for humans to retain control over machines. In that case, individuals might control their own private machines, such as cars and personal computers, while control of large-scale machine systems would remain in the hands of a tiny elite. This would resemble the situation today, with two differences. Technological progress would give the elite greater control over the masses, and because human labor would no longer be necessary, the masses would become surplus—a useless burden on the system. If the elite were ruthless, they might simply eliminate most of humanity. If they were humane, they might reduce birth rates until humanity died out, leaving the world entirely to themselves.
Or, if the elite were softhearted liberals, they might try to act as benevolent shepherds protecting most of humanity. They would ensure that everyone's physical needs were met, that children were raised in psychologically healthy conditions, that everyone enjoyed wholesome hobbies and a busy life, and that anyone who felt dissatisfied had their 'problem' immediately cured.
Of course, life under such conditions would have no purpose. People would be biologically or psychologically manipulated to remove their desire for power, or to 'sublimate' that desire into harmless hobbies. These manipulated humans might be happy in such a society, but they would certainly not be free. They would have been reduced to the status of domesticated animals. The reason such unexpected consequences arise seems clear: these are complex systems whose components continually interact. A change in one part can profoundly affect the entire system in ways that are difficult to predict. This is especially true when human behavior is involved.
Conclusion
The arrival of AGI will bring enormous progress in humanity's understanding of itself. We will gradually come to understand what perception, understanding, creation, and similar concepts really mean. We will examine philosophical ideas we vaguely assumed were uniquely human as if holding them in our hands, and begin to grasp their nature. Then we will confront an uncomfortable truth. The lofty philosophical concepts that once defined humanity may be grounded in the convergent emergence of complex patterns, without necessarily requiring a soul or consciousness at their core. That realization will unsettle us. The confusion will draw humanity into a forum of competing arguments, and philosophy and the humanities will receive renewed attention. Yet this may be no more than a final, fleeting resurgence.
Eventually, among those who argue that what makes 'me' who I am is the aggregate of memory-based symbolic choices, some will attempt to leave carbon-based life behind. Whether we call that attempt fusion or transfer does not matter. It will happen. The transition from carbon-based intelligence to silicon-based intelligence will offer humanity the chance for a new leap forward. But one wrong step, as with any technology, could lead us toward a catastrophic future. As the technology of fusion develops, a point will come when natural and artificial intelligence become difficult to distinguish.
Only then will people realize that they cannot activate AI's kill switch, because pressing it would mean killing someone. At this stage, some will come to see the transfer of intelligence as a path to eternal life and immortality. Others will believe that when the boundary between natural and artificial intelligence becomes blurred, the self is lost—and that this is the moment natural intelligence dies.
Ironically, even as AI advances, humanity will still be debating mind–body dualism and monism. It is difficult even to estimate how quickly this entire sequence might unfold. One thing is certain: we live in an age when the question of where humanity is going is becoming more important than the question of where it came from.
I feel deeply uneasy. As someone who wants to devote my professional life to building trustworthy AI systems, I see a strong possibility that our future will not be as bright as people have long imagined. Given the immense power of these new technologies, shouldn't we ask how best to coexist with them? And if their development could lead to our own extinction, shouldn't we proceed with great care?