QEDBench Finds Critical AI Evaluation Alignment Gap

What is the alignment gap in LLM evaluation?

The alignment gap in LLM evaluation represents a significant discrepancy between an AI's automated scoring of complex tasks and the actual qualitative standards set by human experts. In the context of advanced academic research, this gap highlights a systematic failure where "LLM-as-a-Judge" protocols provide inflated or inaccurate assessments of university-level mathematical proofs, failing to mirror the rigorous logic required by human mathematicians.

As Large Language Models (LLMs) continue to saturate elementary benchmarks, the research frontier has transitioned from simple generation to the reliability of automated evaluation. In a groundbreaking study titled "QEDBENCH: Quantifying the Alignment Gap in Automated Evaluation of University-Level Mathematical Proofs," researchers Yuchen Fang, Zachary Burton, and Ji Zeng identify that current evaluators lack the precision necessary for upper-undergraduate and early graduate-level mathematics. This research is particularly timely as models like GPT-5 Pro are increasingly integrated into educational and research environments where accuracy is paramount.

The study posits that while models have become proficient at mimicking the "style" of mathematical proofs, they often fail to grasp the underlying "substance." This misalignment creates a "positive bias" where automated judges reward formal-looking but logically flawed arguments. By introducing the QEDBench framework, the authors provide a mechanism to quantify these failures, moving beyond simple accuracy metrics to a more nuanced understanding of how AI deviates from human expert consensus.

What is QEDBench and how does it measure AI bias?

QEDBench is the first large-scale dual-rubric alignment benchmark designed to measure the gap between AI judges and human expert mathematicians on university-level proofs. It measures bias by deploying a dual-evaluation matrix that contrasts specific course rubrics against "expert common knowledge" criteria, verified through over 1,000 hours of human expert evaluation to ensure a gold-standard ground truth.

The methodology employed by Fang, Burton, and Zeng involved a sophisticated 7 judges x 5 solvers matrix. This structure allowed the researchers to cross-reference the evaluative performance of various frontier models against human-verified scores across more than 1,000 hours of intensive mathematical analysis. Unlike previous benchmarks that focus on elementary arithmetic or high-school level competition math, QEDBench targets the nuances of proof-based mathematics found in higher education curricula.

Key features of the QEDBench framework include:

• Dual-Rubric Comparison: Evaluating proofs using both rigid, course-specific rubrics and broader mathematical common sense.
• Human-in-the-loop Validation: Every data point is grounded in rigorous human assessment to identify where AI scores diverge from reality.
• Scale and Depth: Focuses on upper-undergraduate to graduate-level mathematics, where logical rigor is more complex than simple computation.
• Public Accessibility: The benchmark has been released publicly at https://github.com/qqliu/Yale-QEDBench to encourage industry-wide calibration.

Why do AI judges inflate scores for mathematical proofs?

AI judges inflate scores because they often prioritize linguistic fluency and formal formatting over logical soundness, a phenomenon known as "positive bias." Research using QEDBench revealed that frontier evaluators frequently assign higher scores than human experts, with models like GPT-5 Pro, Claude Opus 4.5, and Llama 4 Maverick showing mean score inflations ranging from +0.18 to +0.36.

The researchers quantified this bias with startling precision. For instance, Llama 4 Maverick exhibited the highest level of inflation at +0.36, while Qwen 2.5 Max and DeepSeek-V3 followed with +0.30 and +0.20 respectively. This tendency toward leniency is dangerous in academic settings because it can validate incorrect mathematical reasoning, potentially leading to the propagation of errors in scientific literature or educational feedback loops. When an automated judge like GPT-5 Pro encounters a proof that "looks" correct—using appropriate LaTeX formatting and professional terminology—it may overlook "hidden" logical leaps that a human professor would immediately penalize.

This score inflation suggests that "LLM-as-a-Judge" protocols are currently prone to hallucinating correctness. The models appear to use heuristics—such as length, complexity of vocabulary, or the presence of specific mathematical symbols—as proxies for quality. Because these models are trained on massive datasets that include both correct and incorrect proofs, they may struggle to distinguish between a rigorous logical derivation and a sophisticated-looking imitation of one.

How does Gemini 3.0 Pro compare to Claude 4.5 in math?

Gemini 3.0 Pro significantly outperforms Claude 4.5 and GPT-5 Pro in the discrete mathematics domain, maintaining high accuracy where other next-gen models suffer a sharp decline. While Gemini 3.0 Pro achieved a state-of-the-art human evaluation score of 0.91, Claude Sonnet 4.5 and GPT-5 Pro saw their scores drop as low as 0.63 and 0.72, respectively, in specific discrete math challenges.

The "Reasoning Gap" identified in the QEDBench study highlights a surprising weakness in several high-profile models when dealing with the discrete domain. Specifically, the researchers found that:

• Gemini 3.0 Pro maintained a dominant 0.91 average human evaluation score across diverse mathematical fields.
• GPT-5 Pro saw its performance degrade to an average of 0.72 in Discrete Mathematics and 0.74 in Graph Theory.
• Claude Sonnet 4.5 experienced the most significant drop, falling to 0.63 in Discrete Math and a staggering 0.50 in Graph Theory.

This discrepancy suggests that current AI architectures may be better suited for continuous mathematics (like calculus) than the combinatorial and logic-heavy requirements of Discrete Mathematics and Graph Theory. The ability of Gemini 3.0 Pro to navigate these "discrete" challenges suggests a more robust internal representation of logical steps, whereas other models may rely more heavily on pattern matching that fails when the structural rules of the mathematical domain shift. This finding is critical for researchers choosing which models to employ for automated theorem proving or peer review assistance.

The Future of Automated Proof Evaluation

The implications of the QEDBench study extend far beyond the classroom, touching on the very future of scientific peer review and automated reasoning. By exposing the Alignment Gap, Fang, Burton, and Zeng have provided a roadmap for the next generation of AI development. The researchers emphasize that reducing score inflation is not merely a matter of more data, but a matter of better evaluative calibration. Future models must be trained not just to solve problems, but to critically assess the logical pathways used to reach those solutions.

In the short term, the researchers recommend that institutions using AI for grading or research verification implement "human-in-the-loop" systems. The fact that even a high-performing model like GPT-5 Pro can exhibit significant bias means that automated scores should be treated as suggestions rather than definitive verdicts. As the field moves forward, tools like QEDBench will be essential for "benchmarking the benchmarks," ensuring that as AI becomes more sophisticated, its ability to judge its own work—and the work of others—remains grounded in the uncompromising rigor of human mathematical expertise.

Broader adoption of the QEDBench standards could lead to a new era of AI integration in higher education. If the alignment gap can be closed, AI judges could eventually provide real-time, expert-level feedback to students working on complex proofs, democratizing access to high-level mathematical mentorship. For now, however, the study serves as a vital reminder: in the world of university-level mathematics, looking right is not the same as being right.