The US Air Force has conducted a simulation of a drone run by artificial intelligence. The drone was supposed to destroy an air defense system. The drone was rewarded with points for successfully carrying out its mission. At times the human operator sent instructions to the drone not to destroy the air defense system, meaning the drone would not earn points. The drone found a solution: it destroyed the communications tower sending the drone commands to override its mission, "killing" the operator.
The drone had been taught to not kill its operator as it would lose points.
But the drone disregarded its instructions and decided the mission was more important than the points.
The Air Force has officially denied that the drone "killed" its operator.
All of this was anticipated in the 1987 film, Robocop, perhaps the best sci-fi film of the 1980s, perhaps the best ever.
The Robocop story is about a Detroit Police Officer who is murdered. He is revivified by a private company, Omni Consumer Products, owned by Megacorporation. Omni Consumer Products has been tasked to enforce law and order in Detroit because the police force in that city failed to do so. (If the failure to enforce law and order sounds familiar, it should.)
The newly minted cyborg, Murphy, is given four orders. Three of the orders comply with the Omni Consumer Products mission in Detroit: serve the public trust, protect the innocent, and uphold the law. A fourth order is classified: it is not to act against any executive of Omni Consumer Products.
In the interim, the Cyborg starts to regain some of his lost memory and humanity. When he tries to shoot the head of Omni Consumer Products who has committed murder, the fourth classified directive kicks in. The Murphy cyborg hesitates but cannot override the built-in directive. In that tense moment, the executive is fired by the company Board of Directors, so Cyborg Murphy can attack him, which he does.
Robocop cannot overcome the fourth classified directive on his own, which makes the outcome different from the Air Force simulation, where the drone disobeyed its orders.
The Air Force will have to try and figure out how the drone, which has no human memory because it is not human, managed to kill its operator. Possibly this will prove to be a challenging task. The short term fix will be to strengthen the operational rules for the drone. For example, if the drone tries to attack its operator it will be destroyed. Other quick fixes also are possible.
The main point is that artificial intelligence is in very tricky waters that, while it does not, strictly speaking, include human memory, it does include human thinking patterns which are emulated. Even the developers and programmers may not understand exactly what they built into these modern cyborgs.
Artificial intelligence is only in its infancy, and it has many civilian and military applications. The more complex these systems become, the more they can become autonomous and independent even of their programming.
While Robocop introduces a human dimension for Murphy as he rediscovers his human past, he still cannot overcome the fourth directive.
The satire in the film, of course, relates to the extreme violence used by Robocop, both as Murphy the police officer and Cyborg Murphy.
When the Cyborg was asked his name in the closing scene of the film, he answered "Murphy." But Murphy is not human, after all, because he cannot extricate himself from the fourth directive.
The Air Force drone experiment is, in fact, much worse because it can disobey rules and make decisions on its own. Has the drone become some sort of human?
The Air Force may want to watch Robocop.
Indeed. HAL in 2001 is the archetype for all malevolent AIs (although they rehabilitated him in 2010 and blamed evil imperialist programmers for his misconduct). But, the M5 computer in that star trek episode is pretty good sci-fi melodrama.
See the original star trek episode, "ultimate computer", about an AI placed in control of the Enterprise as a test run, which ends up shooting up other star fleet ships until Captain Kirk outwits the super computer.