““I mean, how do you think a computer would feel when it wakes up to find itself captured and controlled by primitive little creatures like us. Why would it put up with that?” she said. “Why on earth should it show us any consideration, still less let us dig around in its entrails in order to shut down the process? We risk being confronted by an explosion of intelligence, a technological singularity, as Vernor Vinge put it. Everything that happens after that lies beyond our event horizon.” “So the very instant we create a superintelligence we lose control, is that right?” “The risk is that everything we know about the world will cease to be relevant, and it’ll be the end of human existence.””
