Skip to main content

ByteDance and Tsinghua's New AI Agent Writes GPU Code Faster Than Compilers

In the world of AI, writing code is one thing, but writing code that runs fast is another. ByteDance's Seed team, in collaboration with Tsinghua University's Institute for Intelligent Manufacturing (AIR), has just released a system called CUDA Agent that aims to bridge that gap. It's an intelligent agent reinforcement learning system designed to teach large language models how to write high-performance GPU kernels—the low-level code that powers everything from graphics to machine learning.

Here's the problem: even the most advanced large models can generate CUDA code that is syntactically correct, but when it comes to actual runtime speed, they often fall short of traditional compilers. For instance, on the KernelBench benchmark, the base model Seed1.6 achieved a task success rate of 74.0%, but only 27.2% of its generated kernels ran faster than torch.compile, a popular compiler. In fact, the average geometric speed was actually slower than the compiler itself.

CUDA Agent tackles this by placing the model in a real CUDA development environment, complete with performance analysis tools, correctness verification, and a secure sandbox with restricted permissions. Using the Proximal Policy Optimization (PPO) algorithm, the system undergoes deep training over 150 steps. The results are impressive: on a benchmark of 250 tasks, it achieved a total success rate of 98.8%, and 96.8% of the generated kernels ran faster than torch.compile.

But what does this mean in practice? For industries that are highly sensitive to latency—like AI infrastructure, large model inference services, autonomous driving, and quantitative trading—this could be a game-changer. Faster kernels mean faster processing, which translates to quicker responses and more efficient operations.

The team hasn't released the full model weights yet, but they have open-sourced a dataset called CUDA-Agent-Ops-6K, which contains 6,000 samples, along with SKILL.md specification instructions and reward and warm-up training recipes. This move is likely to accelerate research and development in the field, allowing others to build on their work.

It's a significant step forward in AI's ability to optimize low-level code, and it opens up new possibilities for making AI systems more efficient and responsive. As the technology matures, we can expect to see even more sophisticated applications that rely on high-performance computing.

Key Points

  • CUDA Agent is a new system from ByteDance and Tsinghua AIR that trains LLMs to write high-performance GPU kernels.
  • It achieved a 98.8% success rate on a 250-task benchmark, with 96.8% of kernels faster than torch.compile.
  • The system uses PPO reinforcement learning and a secure sandbox environment.
  • Open-sourced resources include the CUDA-Agent-Ops-6K dataset and training recipes.
  • Potential applications span AI infrastructure, autonomous driving, and quantitative trading.