On a testing range operated by the U.S. military’s research agency DARPA, a robot equipped with high-end sensors sat on a traffic island with a singular, mission-critical objective: detect any human presence. For six days, a squad of Marines walked, ran, and roamed around the robot, allowing the machine to learn the precise visual signature of a human being. On the seventh day, the mission changed. The challenge for the soldiers was simple: walk up to the sensor without being detected. All eight Marines succeeded.1

How did the Marines bypass the AI detection system?

The story, recounted in the 2023 book by former Pentagon analyst Paul Scharre, stems from DARPA’s Squad X program, which aimed to augment infantry units with advanced sensors. When faced with the task of evading the system, the Marines resorted to tactics that seemed more suited to a cartoon than a military simulation. Two soldiers spent 300 meters somersaulting toward the device. Others hid inside a cardboard box, waddling forward—much to the amusement of the program manager, who could hear the soldiers giggling from inside the box. Another draped himself in pine branches, shuffling forward like a walking shrub. The system, which had spent nearly a week analyzing the soldiers, failed to identify them.1

The failure occurred because the system was programmed to recognize a specific model of a walking human. A human somersaulting, a human inside a box, or a human concealed by branches did not exist within the system’s training data. In the world of artificial intelligence, what is not in the data does not exist.

How can a computer be tricked into misidentifying objects?

This weakness is not limited to military field tests; researchers have been demonstrating it in controlled laboratories for over a decade. In a seminal 2015 study, a computer vision network was shown a picture of a panda and correctly identified it with 58 percent confidence. After researchers added a calculated layer of digital noise—too subtle for the human eye to detect—the computer identified the same image as a 'gibbon' with 99 percent confidence.2 These traps, known as adversarial examples, extend into the physical world. In 2018, a team at MIT 3D-printed a turtle designed with specific patterns that forced Google’s image recognition software to label it as a rifle from almost every angle.3 That same year, researchers used simple stickers to alter stop signs, tricking autonomous vehicle systems into reading them as 45 mph speed limit signs.4 The common thread is that the world perceived by the AI is fundamentally different from the world perceived by humans.

AI did not identify the humans; it identified the humans within its training data. The difference between the two is the size of a cardboard box. Summary of the Squad X experiment

Why is artificial intelligence so easily deceived?

The vulnerability is not a software bug but an inherent quality of current machine learning methods. Modern image recognition systems do not understand the world conceptually; they derive statistical patterns from millions of data points. To the machine, a human is merely a statistical cluster of pixels representing legs, a torso, and a specific silhouette. While humans can reason that a box does not walk on its own because we possess a mental model of physics and intent, the AI lacks such a model. Anything falling outside the distribution of its training data is treated as statistical noise. Researchers call this the 'distribution shift' or 'edge case' problem. As Scharre warns, these edge cases will persist because an adversary will always have the capacity to adapt.1

What does the cardboard box experiment teach us about AI?

Dismissing this as proof that artificial intelligence is 'stupid' would be a mistake. These same systems operate at speeds and scales that humans cannot match. The lesson is that AI is not an infallible sage, but a remarkably fast—yet narrow-minded—statistician. From facial recognition gates to self-driving cars, these systems are powerful within the boundaries of their training data but fragile beyond them. Consequently, major institutions now employ 'red team' testing, paying professionals to do exactly what the Marines did with a cardboard box: attempt to break the system from the outside. The laboratory equivalents, such as the 3D-printed turtle or the doctored stop signs, serve as a permanent reminder that the systems we trust are only as robust as their training.

Notes and Sources

  1. Scharre, P. (2023). Four Battlegrounds: Power in the Age of Artificial Intelligence, W. W. Norton. DARPA Squad X test; accounts by program manager Phil Root. ↩
  2. Goodfellow, I., Shlens, J. & Szegedy, C. (2015). "Explaining and Harnessing Adversarial Examples", ICLR 2015. ↩
  3. Athalye, A., Engstrom, L., Ilyas, A. & Kwok, K. (2018). "Synthesizing Robust Adversarial Examples", ICML 2018. ↩
  4. Eykholt, K. et al. (2018). "Robust Physical-World Attacks on Deep Learning Visual Classification", CVPR 2018. ↩