Skip to main content

Tencent's WorkBuddy Bench: A New AI Testing Ground That Can't Be Gamed

Tencent's WorkBuddy Bench: A New AI Testing Ground That Can't Be Gamed

For a while now, evaluating coding AI agents has felt a bit like grading a student on a test they've already seen the answers to. Benchmarks have been siloed—fix bugs on SWE-bench, do front-end work on Design2Code—and many production environment benchmarks are kept behind closed doors. Tencent is trying to change that with WorkBuddy Bench, a multi-domain evaluation suite that brings together code, web, office, and security tasks. The paper is already up on arXiv.

What's really interesting isn't just the number of tasks (260 in total), but how they're built. Tencent's team went out of their way to make sure these tasks can't be gamed by memorization. Instead of pulling questions from public datasets, they reverse-engineered them from real code commits, pull requests, and business scenarios. Then they rewrote them into short, colloquial, role-playing-style requests. The idea is that you can't just search online for the corresponding PR or commit—there's no shortcut. The dataset is public (including task directories, environment images, evaluation tools, test cases, and reference solutions), so the resistance to contamination comes from the construction method and version control, not secrecy.

Image

The four subsets—code (80 tasks), web (70), office (50), and security (60)—share the same task directory format, but each has its own validation method. That means scores between subsets aren't directly comparable, so Tencent doesn't report an overall score. Code tasks are scored based on hidden test pass rates; web tasks use rule checks combined with LLM/VLM evaluation and agent evaluation, requiring delivery of artifacts along a declared path without connecting to the live internet; office tasks use deterministic rule checks combined with evidence-based LLM evaluation, with weights ranging from 0.70 to 0.95 favoring rules; security tasks use a deterministic scoring.py run three times and averaged, without using large models as judges. The security subset also includes five layers of anti-cheating measures—disabling literal scanning, renaming inputs, covering tampered tests, encoding dependencies, and low-weight bait—with 38 red team tasks and 22 blue team tasks. White-box audits are anchored on real-world CVEs like binutils, curl, and nginx.

The task construction follows a unified pipeline: source collection, rewriting into real requests, assembling the workspace, isolating assessment assets after rounds, and packaging into independent directories. User data is never touched. Code tasks assign five roles (developer, algorithm engineer, product manager, QA, operations), while security tasks assign professional roles, and deliberately provide incomplete information—they don't tell you the target file, pattern, or boundary, so the agent must retrieve the context itself. Scoring assets are introduced only after the agent completes its run, and it never sees them during the process.

Evaluation runs in isolated containers, with the model and sandbox separated. The framework uses CodeBuddy Code (default) and Claude Code, with increased reasoning efforts and a context window of 200k, and WebSearch and AskUserQuestion are disabled. Turning off online search is a deliberate choice—it's a way to verify whether the resistance to contamination is truly effective.

The leaderboard features multiple model families (scores 0–100, thinking mode, three averages). Claude Opus4.8 dominates the majority of the top positions in code, web, office, and security, taking five first places; GLM-5.2 tops two security rankings, and GPT-5.5 takes the office cc first. The framework's sensitivity is also significant—the security subset shows the most dramatic re-ranking, with an average absolute change of 8.6 points; Claude Opus4.8 rejected 13 answers under the cc framework, while GPT-5.5 rejected only 2 under cbc. In terms of efficiency, GPT-5.5 outputs the fewest tokens but maintains a decent score, while DeepSeek-V4-Pro ran 44 rounds on code, with high token input and output, appearing more "demanding."

Key Points:

  • WorkBuddy Bench spans four domains: code, web, office, and security, with 260 tasks total.
  • Tasks are reverse-engineered from real-world scenarios to prevent memorization.
  • Each subset has its own validation method, so scores aren't directly comparable.
  • Claude Opus4.8 leads in most categories, but the security subset shows significant re-ranking.
  • The benchmark is designed to be resistant to contamination through construction methods, not secrecy.