Roboticizing aspects of warfare gained steam in the U.S. during the administrations of George W. Bush and Barack Obama. As country after country follows the U.S. military’s lead and builds their own force of UAVs, it is clear that robot fighters are here to stay. This represents a shift in the way future wars will be fought, comparable to the introduction of the cross-bow, the Gatling gun, aircraft, and nuclear weapons.

What remains undecided is whether robotic war machines will become fully autonomous. Will they pick their own targets and pull the trigger without the approval of a human? Will there be an arms race in robotic weaponry or will limits be set on the kinds of weapons that can be deployed? If limits are not set, eventually the natural escalation of robotic killing machines could easily progress to a point where robust human control over the conduct of warfare is lost.

The executive order I envisaged and proposed could be a first step in establishing an international humanitarian principle that “machines should not be making decisions about killing humans.” A vision of the president signing the executive order accompanied the idea, as if from its inception this initiative was a done deal. Like many dreamers I was wooed to action by the fantasy that this project would be realized quickly and effortlessly. Life is seldom that easy. To this day, I have no idea whether my proposal or any other campaign would lead to a ban of lethal autonomous weaponry. Nevertheless, from that first moment it was clear to me that an opportunity to pursue a ban presently exists but will disappear within a few years. The debate focuses upon whether the development of autonomous killing machines for warfare should be considered acceptable. But in a larger sense, we have begun a process of deciding whether people will retain responsibility for the actions taken by machines.

The first formal call for a ban of lethal autonomous robots had been formulated fifteen months earlier (October 2010) at a workshop in Berlin, Germany organized by the International Committee for Robot Arms Control (ICRAC). At that time, ICRAC was nothing more than a committee of four scholars (Jürgen Altmann, Peter Asaro, Noel Sharkey, and Rob Sparrow). The gathering they organized included arms control specialists, experts in  international humanitarian law (IHL),  roboticists, leaders of successful campaigns  to stop the development of nasty weapons such as cluster bombs, and most of the small community of scholars that had written about lethal autonomous robots. Colin Allen and I were invited to participate for the work we had done in co-authoring the book Moral Machines: Teaching Robots Right From Wrong.

The word autonomous in reference to robots denotes systems capable of initiating actions with little or no ongoing human involvement. The drones that were used successfully by the U.S. in Afghanistan and Pakistan are unmanned but remotely controlled by personnel often thousands of miles away. They are not what others and I are worried about. Officers at the remote location made decisions as to when a legitimate target is in sight, and then authorize missiles to be dispatched. A human is “in the loop” making decisions about who to kill or what to destroy. Increasingly, however, more and more functions are being turned over to computerized systems. For example, in 2013, the Northrop Grumman X-47B, a prototype sub-sonic aircraft with two bomb compartments and a 62-foot wingspan, autonomously took off from and landed on an aircraft carrier. The proposed ban on autonomous lethal robots is focused upon ensuring that in the future, selecting a target and pulling the “trigger” is always a decision made by a human and never delegated to a machine. There must always be a human in the loop.

Today’s computers do not have the smarts to make discriminating decisions such as who to kill or when to fire a shot or a missile. Thus, a ban is directed at future systems that have not yet been deployed, and in nearly all cases, have not yet been built. There is still time to make a course correction. Nevertheless, there already exist dumb autonomous or semi-autonomous weapons that can kill. For example, a landmine is autonomous and will go off without a human making a decision to harm the specific individual who trips the device. Unfortunately, all too often it is children who trip landmines. In addition, defensive weaponry including antiballistic missile systems such as the U.S. Navy’s Phalanx or Israel’s Iron Dome can autonomously intercept incoming missiles well before military personnel would have time to make a decision. A ban would probably not include defensive weaponry, although often the difference between declaring a weapon system defensive or offensive is merely a matter of what direction the weapon points. Nor would a ban affect autonomous  weapons viewed as the last line of defense in an area of known hostility. The Samsung Techwin, a stationary robot capable of gunning down anything that moves in the Demilitarized Zone (DMZ) separating North and South Korea, has been deployed since 2010. These stationary robots are unlikely to have even modest success should North Korea decide to send a million troops across the DMZ to invade the South.

The proposal for a presidential order that I wrote up and circulated received only a modest degree of attention and support. However, I’m certainly not alone in calling for a ban. Soon after the November 2012 presidential election, Human Rights Watch (HRW) and the Harvard Law School International Rights Clinic entered the campaign with a high-profile report that called for a ban of lethal autonomous robots (LARs). Three months later, HRW and a coalition of other nongovernmental organizations (NGOs) launched the international campaign to ban killer robots. That campaign is directed at activating worldwide support for an arms control agreement covering robotic weaponry. In addition, a growing community of international experts advocates the need for robot arms control. In a 2013 report, Christof Heyns, U.N. Special Rapporteur on extrajudicial, summary or arbitrary executions, called for a moratorium on the development of LARs as a first step toward considering an international ban.

These efforts have had significant  success in catalyzing governments around the world to give a ban serious attention. In May 2014, the U.N. Convention on Certain Conventional Weapons (CCW) convened a meeting in Geneva to discuss the dangers autonomous weapons pose. One hundred seventeen nations are party to the CCW, which restricts the use of specific weapons deemed to cause unjustifiable harm to combatants or civilians. On November 14, 2014, the CCW voted to continue deliberations over LARs, an important first step in acknowledging the importance of the issue.

Opponents of robotic weaponry contend that their use could lower the barriers to starting new wars. The potential loss of one’s own troops has been one of the few major deterrents to starting wars. Autonomous weaponry contribute to the illusion that wars can be started and quickly won with minimal costs. Once a war begins, however, not only the lives of soldiers, but also those of civilians will be lost. Furthermore, a decision about who or when to kill by an autonomous weapon could accidentally initiate hostilities. They could also be dangerous from an operational perspective if robotic weapons, for example, escalated an ongoing conflict or used force indiscriminately or disproportionally. For a military commander, the possibility of autonomous systems acting in a manner that escalates hostilities represents a loss of robust command and control.

In addition to saving the lives of one’s  soldiers, two other ostensibly strong arguments get put forward as objections to banning LARs. The first considers LARs to be a less lethal option than alternative weapons systems. Presuming  LARs are more accurate than other available weapons systems, they will cause less loss of civilian lives (less collateral damage). Shortsighted contentions such as this do not fully factor in the future dangers once many countries have robotic armies. The long-term consequences of roboticizing aspects of warfare could far outweigh the short-term benefits.

The second argument proposes that future machines will have the capacity for discrimination and will be more moral in their choices and actions than human soldiers. Ronald Arkin, a roboticist at Georgia Tech, takes this position. Arkin has been working toward the development of means for programming a robot soldier to obey the internationally established Laws of Armed Conflict (LOAC). He contends that robot soldiers will be better at following the LOAC because “the bar is so low.” In this Arkin refers to research that shows, for example, that human soldiers will not squeal on their buddies even when atrocities have been committed. Nevertheless, the prospect is low for developing robot soldiers any time soon with the ability of making an appropriate judgment in a complex situation. For example, a robot would not be good at distinguishing a combatant from a non-combatant, a task that humans also find difficult. Humans, however, bring powers of discrimination to bear for meeting the challenge; capabilities that will be difficult, if not impossible, for robots to emulate. If and when robots become ethical actors that can be held responsible for their actions, we can then begin debating whether they are no longer machines and are deserving of some form of personhood. But warfare is not the place to test speculative possibilities.

Widespread opposition to killer robots already exists according to a 2013 poll of a random sampling of a thousand Americans by Charli Carpenter, a Professor of Political Science at the University of Massachusetts. Overall, 55 percent of respondents oppose the development of autonomous weapons, with 39 percent strongly opposed. In other words, there exists substantial national and international public support for a ban. But without a continual campaign actively championed by a large segment of the public, policy makers will dismiss support for a ban as the product of fears generated by science fiction. Interestingly, 70 percent of respondents in the active military rejected fully autonomous weapons. Nevertheless, military planners have retained the option to build autonomous killing machines.

Traditional forms of arms control that depend upon inspection protocols provide a poor model for limiting the use of robotic weapons. The difference between an autonomous system and one requiring the actions of a human to proceed may be little more than a switch or a few lines of software code. Such minor modifications could be easily hidden from an inspector, or added in the days following the inspector’s departure.

Furthermore, arms control agreements can take forever to negotiate, and there’s no reason to think that negotiations to formulate procedures for the verification and inspection of robotic weaponry will be any different. Continual innovations in lethal autonomous weaponry would also require a need to periodically revise the arms agreements that had been negotiated.

Ban proponents are also up against powerful forces in the military-industrial complex who want sophisticated robotic technology funded by defense budgets. Weapons production is a lucrative business, and research funded by defense budgets can often be sold to our allies or spun off to develop non-military technologies. During the years before a ban is enacted, there will be an alignment of countries and corporations that have a vested interest in continuing the development of robotic weaponry and in defeating any attempts to limit their use. That is why an inflection point exists now, before autonomous weapons become a core weapons system around which major powers formulate their defense strategy. In the U.S. plans to reduce active personnel and increase the deployment of military technology were announced in 2014. Half of the bombers on aircraft carriers are slated to become a version of the unmanned X-47. Future models will fly in swarms and even make autonomous decisions about bombing targets. The inflection point for limiting lethal autonomous weapons that presently exists could easily disappear within a matter of a few years. The length of time the window stays open is dependent upon whether or not the campaign to ban killer robots gains enough clout to impact the willingness of governments to invest in developing the required technologies.

An international ban on LARs is needed. But given the difficulty in putting such an agreement in place, the emphasis should initially be upon placing a moral onus on the use of machines that make life-and-death decisions. A long-standing concept in just war theory and international humanitarian law designates certain activities as evil in themselves —what Roman philosophers called mala en se. Rape  and the use of biological weapons are examples of activities considered mala en se.  Machines making life-and-death decisions should also be classified as mala en se.

Machines lack discrimination, empathy, and the capacity to make the proportional judgments necessary for weighing civilian casualties against achieving military objectives. Killer robots are mala en se, not merely because they are machines, but also because their actions are unpredictable, cannot be fully controlled, and attribution of responsibility for the actions of autonomous machines is difficult if not impossible to make. Delegating life-and-death decisions to machines is immoral because machines cannot be held responsible for their actions.

A RED LINE

The Terminator is clearly science fiction, but it speaks to a powerful intuition that the robotization of warfare is a slippery slope—the end point of which can neither be predicted nor fully controlled. Machines capable of initiating lethal force and the advent of a technological singularity are future-oriented themes, but function as placeholders  for more immediate concerns. The computer industry has begun to develop systems with a degree of autonomy and capable of machine learning. Engineers do not know whether they can fully control the choices and actions of autonomous robots.

The long-term consequences of roboticizing  warfare must neither be underestimated nor treated cavalierly. But given that arms manufacturers and a few countries are focused on the valuable short-term advantages of developing killer robots, the momentum behind the robot arms race is likely to quickly spin beyond control. Setting parameters now on the activities acceptable for a lethal robot to perform offers one of the few tools available to restrain a robot arms race that mirrors that of the Cold War. One difference, however, is that, unlike nuclear weapons, robotic delivery systems will be relatively easy to build.

In the same November 2012 week in which Human Rights Watch called for a ban on killer robots, the U.S. Department of Defense (DoD) published a directive titled, “Autonomy in Weapons Systems.” The timing of the two documents’ release could have been coincidental. Nevertheless, the directive read as an effort to quell any public concern about the dangers posed by semiautonomous and autonomous weapons systems. Unlike most directives from the DoD, this one was not followed up by detailed protocols and guidelines as to how the testing and safety regimes noted in the document should be carried out. The DoD wants to expand the use of self-directed weapons, and in the directive it explicitly asks the public not to worry about autonomous robotic weaponry, stating that the DoD will put in place adequate oversight on its own. Military planners do not want their options limited by civilians who are fearful of speculative possibilities.

Neither military leaders nor anyone else wants warfare to expand beyond the bounds of human control. The directive repeats eight times that the DoD is concerned with minimizing “failures that could lead to unintended engagements or loss of control of the system.” Unfortunately, this promise overlooks two core problems. First, the DoD has a specific mission and cannot be expected to handle oversight of robot autonomy on its own. During the war against al-Qaeda in Afghanistan and Pakistan, military and political necessities caused Secretary of Defense William Gates to authorize the deployment of second-generation unmanned aircraft before they had been fully tested. In fulfilling its primary mission, the DoD is prepared to compromise other considerations. Second, even if the American public and U.S. allies trust that the DoD will establish robust command and control in deploying autonomous weaponry, there is absolutely no reason to assume that other countries and non-state actors will do the same. Other countries including North Korea, Iraq, or whoever happens to be the rogue government du jour, will almost certainly adopt robotic weapons and use them in ways that are entirely beyond our control.