Skip to main content

OpenBMB's MathForm 8B: A New Hope for Automated Math Proofs

In the quest to build artificial general intelligence, one of the toughest nuts to crack is getting machines to truly understand mathematics. It's not just about crunching numbers—it's about reading a theorem, grasping its meaning, and then verifying it in a formal language like Lean4. That's a tall order, but the OpenBMB team just took a big leap forward with their new open-source project, MathForm.

For years, the stumbling block has been that formalizing math isn't a simple translation job. You can't just take a natural-language proof and turn it into code. A rigorous model has to map every single concept to the right type and definition in Mathlib, a massive library of formalized math. And here's the kicker: even if a statement passes the compiler, it might still be semantically off—it might not actually describe the original problem. That's a silent failure that's been plaguing the field.

So, what did OpenBMB do differently? They built a framework that's all about retrieval and verification. The system starts with a "retrieval planner" that digs up the necessary Mathlib definitions and existing formalized models. Then, a generator takes over, but it doesn't just spit out code and call it a day. It revises its output up to three times, using feedback from the Lean compiler and a semantic consistency check. This back-and-forth ensures the code is not only syntactically correct but also mathematically faithful.

On the data side, they introduced FormalVerse, a dataset packed with over 367,000 verified Lean4 examples, spanning a wide range of mathematical fields. When they trained models on this dataset, the results were striking. Under the same training budget and starting conditions, the consistency check rate hit 60.32%—a big jump over the 46.53% from FineLeanCorpus and 41.49% from NuminaMath-LEAN. That's a clear win for their data quality.

The star of the show is MathForm-8B, a model that's surprisingly compact. In six benchmark tests, it aced the syntax check with an 88.06% pass rate and a 72.37% consistency check pass rate. But here's the real jaw-dropper: it outperformed much larger models like ReForm-32B and Goedel-Formalizer-V2-32B, despite having only a quarter of their parameters. On the toughest subsets, FATE-H and FATE-X, it achieved consistency check success rates of 63% and 37%, respectively—beating the strongest specialized benchmarks by 10 and 12 percentage points. That's not just a marginal improvement; it's a statement.

What does this mean for the future? For one, it shows that you don't need a massive model to do heavy lifting in formal math. Efficiency and smart data can trump sheer size. And with MathForm being open-sourced, researchers and developers can start building on this foundation, potentially accelerating progress in automated theorem proving and bringing us closer to AI that can genuinely reason.

If you're curious, you can dive into the code and datasets on GitHub. The project is live, and the community is already buzzing. Whether you're a mathematician, a developer, or just someone fascinated by AI, this is a development worth watching.

Key Points

  • MathForm framework: Uses retrieval-augmented generation and verification-guided revision to ensure both syntactic and semantic correctness.
  • FormalVerse dataset: Over 367,000 verified Lean4 examples, leading to a 60.32% consistency check rate—far ahead of existing datasets.
  • MathForm-8B model: Outperforms larger models like ReForm-32B and Goedel-Formalizer-V2-32B on benchmarks, with 88.06% syntax and 72.37% consistency pass rates.
  • Open source: The project is available on GitHub, inviting collaboration and further innovation.