Multimodal and Document Analysis Benchmarks: Claude, GPT, and the Work Hidden Inside PDFs

Multimodal is not just image Q and A

The practical multimodal workload for enterprises is not a single image question. It is a packet: PDFs, screenshots, spreadsheets, browser pages, diagrams, and source references. A model must extract evidence and preserve where it came from.

Anthropic's model docs state that current Claude models support vision across the Fable 5.1, Opus 5, Sonnet 5, and Haiku 4.5 lineup, and its Fable 5.1 announcement reports computer-use evaluations directly. OpenAI's GPT-5.5 launch reports its own computer-use and vision tables. Both families are credible; neither should be accepted on a single benchmark row.

Partial and strict grading are different benchmarks

Anthropic's OSWorld 2.0 rows are reported twice, under partial and strict grading: Fable 5.1 at 77.9% partial and 41.7% strict, Opus 5 at 75.4% and 39.6%, Fable 5 at 72.9% and 36.1%. The spread between the two grading modes is roughly 36 points. That is not a rounding detail; it is the difference between "made meaningful progress on the task" and "completed the task as specified."

For anyone planning to deploy a computer-use agent, the strict column is the one that predicts whether work actually gets finished without a human closing the loop. Coverage that quotes only the partial number is reporting the flattering half of a two-part result.

The cross-vendor trap is the same as elsewhere. OpenAI reports GPT-5.5 on OSWorld-Verified; Anthropic reports OSWorld 2.0. Same benchmark family, different variant, different grading conventions, different harness. The 78.7% and the 77.9% look like a near-tie and are not evidence of one.

Computer use is a different benchmark class

Computer-use benchmarks test whether the model can operate an interface, not merely describe it. This is closer to insurance intake, back-office updates, spreadsheet repair, or research collection than any text benchmark gets.

The key caveat is tooling. If a model's benchmark score depends on a specific browser agent, screen parser, or tool policy, the result belongs to the system, not the model alone. Claude and GPT comparisons should name that layer explicitly. The Terminal-Bench authors make the general version of this point: agent and model performance are hard to decouple.

Document analysis needs citation discipline

Claude often gets evaluated in document-heavy settings because long context and careful synthesis are visible strengths of the product line. But document workflows fail in quiet ways: wrong page references, invented citations, missed tables, or overconfident summaries of image-only text.

A good benchmark therefore asks for page references, exact quote extraction within copyright-safe limits, table reconciliation, and contradiction detection. It should score refusal or uncertainty behavior instead of treating every non-answer as a failure.

Build a packet with a 100-page PDF, a spreadsheet, a screenshot, and conflicting instructions. Ask each model to answer, cite, update a small table, and explain uncertainty. Then grade factual accuracy, provenance, action correctness, elapsed time, and cost. That is the kind of reproducible desk test that exposes document-grounding quality, and it is not replaceable by a leaderboard row.

FAQ

Cite this page