Skip to main content

Brainwaves Teach Robots: The Next Frontier in Physical AI

In a warehouse in San Leandro, California, a data tools company called Encord is running an unusual experiment. Workers wearing headsets equipped with cameras and EEG sensors play Jenga, stacking blocks with careful precision. The headsets, made by German neuroscience startup Zander Labs, measure brainwave signals in real time, capturing mental states like errors, intentions, and surprise. The goal? To create richer training datasets for robot models.

Encord believes the real bottleneck for humanoid and warehouse robots isn't model architecture—it's the extreme lack of real-world physical training data. As Vineeth Velmurugan, head of robot learning at Encord, puts it: "This data simply doesn't exist." And the required data volume is about five times the size of YouTube's entire video library.

The Economic Dilemma of Physical AI

Inside the warehouse, pilots use "master-slave" robotic arm devices—one directly controlled by humans, another mimicking its movements—to generate training data for tasks like pouring coffee and stacking poker chips. Shelves are filled with fake flowers, books, plastic vegetables, and cat litter boxes, all used as props for training household robotic hands. At another workstation, pilot Sofia Infante controls a robotic arm to plug and unplug server network cables—an automation task highly desired by data center operators, but currently out of reach due to insufficient flexibility of robotic grippers.

Beyond EEG signals, Encord is developing new data modalities. They attach sensor arrays to the forearm to detect muscle electrical signals, building 3D hand trajectories to compensate for missing information when fingers are blocked in videos. Each dataset comes with dense physical action annotations, such as "right hand tightening a bolt." Velmurugan estimates that this high-quality annotation is 100 times more valuable for specific task training than ordinary data, while production costs are only 20 times higher—theoretically a profitable deal.

Key Points

  • Data scarcity: Real-world physical training data is extremely limited, with required volumes dwarfing existing video libraries.
  • Brainwave capture: EEG headsets record mental states during precise tasks, adding a new dimension to training data.
  • Multi-modal approach: Combining brain signals, muscle signals, and video creates richer datasets.
  • Cost efficiency: High-quality annotations are 100x more valuable but only 20x more expensive to produce.