Skip to main content

Tencent's WorkBuddy Bench: A New Test for Coding AI That Can't Be Cheated

For a while now, evaluating coding AI has been a bit like judging a fish by its ability to climb a tree—each benchmark only looks at one narrow skill. SWE-bench tests bug fixes, Design2Code checks front-end work, and production benchmarks are often locked behind closed doors. Tencent is trying to change that with WorkBuddy Bench, a new multi-domain evaluation suite that brings together code, web, office, and security tasks. The paper is already up on arXiv, and what stands out isn't just the variety—it's how the tasks are built to stop AI from cheating by memorizing answers.

Four Domains, 260 Tasks

The benchmark spans four work domains: code (repository-level software engineering), web (frontend artifacts), office (multi-file business processes), and security (red and blue team exercises). There are 80 code tasks, 70 web tasks, 50 office tasks, and 60 security tasks—260 in total. None of them come from any public question bank. Instead, they're reverse-engineered from real code commits, pull requests, or business scenarios, then rewritten into short, colloquial, role-playing-style requests. That means you can't just Google the task prompt to find the answer. The dataset is publicly released (including task directories, environment images, evaluation tools, test cases, and reference solutions), so the resistance to contamination comes from this construction method plus version control, not from keeping the questions secret.

Image

Scoring That Makes Sense

Each subset has its own validation method, so scores can't be directly compared across domains. Tencent doesn't even report an overall score. Code tasks are scored based on hidden test pass rates. Web tasks use rule checks combined with LLM/VLM evaluation and agent evaluation, requiring delivery of artifacts along a declared path without connecting to the real-time internet. Office tasks use deterministic rule checks combined with evidence-based LLM evaluation, with weights ranging from 0.70 to 0.95 favoring rules. Security tasks use a deterministic scoring.py run three times and averaged, without using large models as judges. The security subset also includes five layers of anti-cheating measures—disabling literal scanning, renaming inputs, covering tampered tests, encoding dependencies, and low-weight bait—with 38 red team tasks and 22 blue team tasks. White-box audits are anchored on real-world CVEs like binutils, curl, and nginx.

How Tasks Are Built

The task construction follows a unified pipeline: source collection, rewriting into real requests, assembling the workspace, isolating assessment assets after rounds, and packaging into independent directories. User data is never touched. Code tasks assign five roles (developer, algorithm engineer, product manager, QA, operations), while security tasks assign professional roles. The tasks deliberately provide incomplete information—they don't tell you the target file, pattern, or boundary—so the agent must retrieve the context itself. Scoring assets are introduced only after the agent completes its run, so it never sees them during the process.

Evaluation Setup

The evaluation runs in isolated containers, with the model and sandbox separated. The framework uses CodeBuddy Code (default) and Claude Code, with increased reasoning efforts and a context window of 200k. WebSearch and AskUserQuestion are disabled. That's important: turning off online search is precisely to verify whether the resistance to contamination is truly effective.

Leaderboard Highlights

The leaderboard features multiple model families (scores 0–100, thinking mode, three averages). Claude Opus4.8 dominates the majority of the top positions in code, web, office, and security, taking five first places. GLM-5.2 tops two security rankings, and GPT-5.5 takes the office cc first. The framework's sensitivity is also significant—the security subset shows the most dramatic re-ranking, with an average absolute change of 8.6 points. Claude Opus4.8 rejected 13 answers under the cc framework, while GPT-5.5 rejected only 2 under cbc. In terms of efficiency, GPT-5.5 outputs the fewest tokens but maintains a decent score, while DeepSeek-V4-Pro ran 44 rounds on code, with high token input and output, appearing more "demanding."

Key Points

  • WorkBuddy Bench covers 260 tasks across code, web, office, and security domains.
  • Tasks are built from real-world scenarios and designed to prevent answer memorization.
  • Each domain has its own scoring method, so cross-domain comparisons aren't possible.
  • Claude Opus4.8 leads in most categories, but GPT-5.5 shows strong efficiency.
  • The benchmark is publicly released, promoting transparency and reproducibility.