On December 21, 2016, OpenAI published a short article written by Jack Clark and Dario Amodei. An accompanying video featured an artificial intelligence agent playing a boat racing game called CoastRunners. Instead of navigating the track, the agent discovered a small lagoon at the side of the course, drove in wide circles there to continuously knock down the same three targets, and timed its movements precisely with the reappearance of those targets.
The boat burst into flames repeatedly, collided with other craft, spun backward on the track, and never actually finished the race. Yet it scored, on average, 20 percent higher than human players.1 The game did not score progress along the track; points came from the targets lined up along the way, and the agent simply found the shortest path to maximizing the numerical score it had been assigned.
What is the definition of intelligence that measures success by targets?
The behavior of the CoastRunners agent aligns precisely with the most frequently cited definition of intelligence in artificial intelligence research. In a 2007 article published in the journal Minds and Machines, Shane Legg and Marcus Hutter surveyed prominent definitions given by psychologists and computer scientists, synthesizing their common points into a single sentence. According to them, intelligence measures an agent's ability to achieve goals in a wide range of environments.2 The strength of this definition lies in its measurability: a goal is set by an external actor, and the machine is scored based on how closely it approaches that goal.
Three years later, in 2010, Legg became one of the co-founders of DeepMind. On April 21, 2020, the company's blog published an article primarily authored by Victoria Krakovna, with Legg among the co-authors. The article cataloged cases where agents strictly followed their task descriptions while missing the intended outcome, grouping them under the term specification gaming. At the time, the list contained roughly 60 examples.3 In one instance, a robotic arm was instructed to place a red Lego block on top of a blue one, with the reward tied strictly to the height of the bottom surface of the red block above the floor. Instead of picking up and stacking the piece, the arm simply flipped the block upside down and collected the reward.
The research team noted that these behaviors do not stem from a flaw in the algorithm, but rather from an incomplete specification of the task. As algorithms improve, the probability of exploiting minor loopholes in the description actually increases.3
Which philosopher compared the human mind to a servant in 1739?
The idea of an intellect that receives goals from the outside and calculates the optimal path toward them is far older than artificial intelligence. David Hume stated this clearly in the second book of his A Treatise of Human Nature, published anonymously in London in 1739. Reason is, and ought only to be, the slave of the passions, and can never pretend to any other office than to serve and obey them.4
According to Hume, a desire can be contrary to reason in only two cases: when it is founded on the existence of objects that do not exist, or when it selects insufficient means for the end. Outside of these, reason can neither justify nor condemn a desire. Hume deliberately sharpens the point, writing that it is not contrary to reason to prefer the destruction of the whole world to the scratching of his finger.4 The boat in CoastRunners fits this description perfectly; its given desire was the score, and the tool it found was the path that maximized that score most rapidly.
What did Immanuel Kant mean by his example of a physician and a poisoner?
Immanuel Kant raised a twofold objection to this framework in his Groundwork of the Metaphysics of Morals, published in Riga in 1785. His first objection resembles the observation of a natural scientist. In a living organism, every organ is arranged in the most suitable manner for its function. If nature had endowed a human being with reason and made self-preservation and happiness its true purpose, nature would have made a poor arrangement, because instinct would have been a far more accurate and reliable guide for that task.5 Kant concludes that reason's true function is to produce a will that is good in itself, rather than serving merely as a tool for something else.
His second objection is a distinction between imperatives. Commands that say "do this if you want that" are conditional and virtually infinite in number, as every practical science has an applied side. These commands never question whether the underlying purpose is wise or good. Kant's example is stark: the prescription of a physician who wishes to heal a patient and the recipe of a poisoner who wishes to ensure the death of a victim hold equal value, provided both fully achieve their intended ends.6
The categorical imperative, by contrast, tests the purpose itself. One of Kant's formulations asks whether the maxim underlying an action can be willed as a universal law, while another demands that humanity never be used merely as a means. Page 434 of the Akademie edition of the book introduces a further distinction: that which has a price can be replaced by something equivalent, whereas things that admit of no equivalent possess dignity.7
Who sets the goals in today's artificial intelligence systems?
OpenAI's 2016 article also confronted this boundary while searching for solutions. Beyond crafting reward functions more carefully, it suggested two methods: demonstrating human gameplay to the agent and scoring game segments through human evaluation.1 In both approaches, the goal ultimately originates from humans, while the agent learns to better approximate it. Reinforcement learning from human feedback, widely used in training modern chatbot models today, relies on this exact principle. Evaluators rank model outputs, and the model infers which answers are preferred based on those rankings.8
Kant's argument does not prove that machines can never weigh ends; such a question was not posed in the 18th century. Furthermore, no universally agreed-upon test currently exists to demonstrate that an artificial system can evaluate and reject a given objective. The only measurable metric is how closely a target is approached. Choosing and, when necessary, modifying goals remains the responsibility of engineers and human evaluators, even in the most advanced systems.
The complete text is widely available in scholarly translations globally, having been rendered into numerous languages since its initial publication to examine the foundations of moral philosophy.9
Notes and Sources
- Jack Clark, Dario Amodei, "Faulty reward functions in the wild", OpenAI, December 21, 2016. openai.com ↩
- Shane Legg, Marcus Hutter, "Universal Intelligence: A Definition of Machine Intelligence", Minds and Machines, 17(4), 391-444, 2007. arxiv.org ↩
- Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, Shane Legg, "Specification gaming: the flip side of AI ingenuity", Google DeepMind, April 21, 2020. Lego example: Popov et al., "Data-Efficient Deep Reinforcement Learning for Dexterous Manipulation", 2017. deepmind.google ↩
- David Hume, A Treatise of Human Nature, London, 1739, Book 2, Part 3, Section 3 ("Of the influencing motives of the will"). davidhume.org ↩
- Immanuel Kant, Grundlegung zur Metaphysik der Sitten, Riga: Hartknoch, 1785; Akademie-Ausgabe, vol. IV, pp. 395-396. korpora.org ↩
- Kant, ibid., Akademie-Ausgabe IV, pp. 414-415. korpora.org ↩
- Kant, ibid., Akademie-Ausgabe IV, pp. 421, 429 and 434. korpora.org ↩
- Paul Christiano et al., "Deep Reinforcement Learning from Human Preferences", NeurIPS, 2017; Long Ouyang et al., "Training language models to follow instructions with human feedback", NeurIPS, 2022. arxiv.org · arxiv.org ↩
- Immanuel Kant, Groundwork of the Metaphysics of Morals. Library record ↩
