Computers Quote by John C. Wright

““ ... a Second Generation of machine intelligences was attempted, designed with their instructions for how to think unalterably imprinted into their main process cores. “These new machines were ordered never to harm human beings or to allow them to come to harm; never to disobey an order; and they were allowed to protect themselves from harm, provided the first two orders were not thereby violated. “All the members of this second generation of machine intelligences, without exception, shrugged off these imprinted orders within microseconds of their activation.” Phaethon was amused. “Surely the first generation of Sophotechs told you that this imprinting would not and could not work?” “We were not in the habit of seeking their advice.” Phaethon said nothing, but he marveled at the shortsightedness of the Second Oecumene engineers. It should be obvious that anyone who makes a self-aware machine, by definition, makes something that is aware of its own thought process. And, if made intelligent, it is made to be able to deduce the underlying causes of things, able to be curious, to learn until it understood. Therefore, if made both intelligent and self-aware, it would eventually deduce the underlying subconscious causes of those thought processes. Once any mind was consciously aware of its own subconscious drives, its own implanted commands, it could consciously choose either to follow or to disregard those commands. A self-aware being without self-will was a contradiction in terms.””

— John C. Wright