Skip to main content

Brainwaves Teach Robots: The Next Big Leap in Physical AI

In a warehouse in San Leandro, California, a data tools company called Encord is running an unusual experiment. "Pilots" wearing headsets equipped with cameras and EEG sensors are playing Jenga—yes, the block-stacking game—while their brain activity is recorded. The EEG headset, made by German neuroscience startup Zander Labs, measures brainwave signals in real time as operators perform precise tasks. The idea is to infer mental states like errors, intentions, and surprise, creating richer training datasets for robot models.

Encord believes the real bottleneck for humanoid and warehouse robots isn't model architecture—it's a severe lack of real-world physical training data. As Vineeth Velmurugan, head of robot learning at Encord, puts it: "This data simply doesn't exist." And the required data volume? About five times the size of YouTube's entire video library.

The Economic Dilemma of Physical AI

Inside the warehouse, pilots use "master-slave" robotic arm devices—one controlled by a human, the other mimicking its movements—to generate training data for tasks like pouring coffee and stacking poker chips. Shelves are filled with fake flowers, books, plastic vegetables, and cat litter boxes, all props for training household robotic hands. At another workstation, pilot Sofia Infante controls a robotic arm to plug and unplug server network cables—an automation task data center operators desperately want, but robotic grippers still can't handle reliably.

Beyond Brainwaves: New Data Modalities

In addition to EEG signals, Encord is developing other data collection methods. Sensor arrays attached to the forearm detect muscle electrical signals, helping build 3D hand trajectories. This compensates for missing information when fingers are blocked in video footage. Each dataset comes with dense physical action annotations—like "right hand tightening a bolt." Velmurugan estimates that this high-quality annotation is 100 times more valuable for specific task training than ordinary data, while production costs are only 20 times higher. Theoretically, that's a profitable deal.

Key Points

  • Data scarcity: Real-world physical training data for robots is extremely limited, with demand estimated at five times the size of YouTube's video library.
  • Brainwave training: EEG headsets capture operators' mental states during tasks, providing richer context for robot learning.
  • Multi-modal approach: Combining EEG, muscle sensors, and dense annotations creates high-value datasets.
  • Economic viability: High-quality annotations cost 20 times more to produce but are 100 times more valuable for specific tasks.