StateSight: A New Benchmark Reveals Frontier VLMs Still Fail at Latent Spatial Reasoning

A new benchmark listed today on arXiv shows that GPT-5.5 and Claude Sonnet 5 — two of the most capable commercial vision-language models available — score well below a 30-person human baseline on every spatial reasoning task the study tests, with Claude Sonnet 5 reaching just 7.3% on the hardest of the three. The study, titled StateSight: Benchmarking Latent Spatial-State Reconstruction in Vision-Language Models and authored by Michelle Lin, introduces a procedurally generated benchmark specifically designed to isolate a capability that broader AI evaluations routinely fail to measure: reconstructing a hidden spatial structure from a single image, without the help of text labels, domain knowledge, or recognizable objects.

What makes the result harder to dismiss than most VLM spatial-reasoning findings is the scoring method. StateSight uses deterministic oracle labels — numbers generated algorithmically, not annotated by humans or judged by another language model — and requires exact-match answers. All model runs produced zero format errors. The models understood the question format; they simply failed to recover the spatial state the question was about.

Three Tasks That Strip Away Every Shortcut

StateSight's three task families are chosen to eliminate the confounds that plague most multimodal benchmarks: perception of real objects, optical character recognition, domain knowledge, and linguistic priors. Each presents a single procedurally generated image and asks a question that can be answered only by mentally simulating a spatial transformation or traversing a structural relationship.

The first task, cube-net opposite-face reasoning, shows a two-dimensional unfolding of a cube's six faces — a net — and asks which face lies directly opposite a designated face. Answering correctly requires mentally folding the net into a three-dimensional cube and tracking each face through the fold. GPT-5.5 scored 59.3% on this task; Claude Sonnet 5 scored 53.3%; the human baseline reached 80.8%.

The second task, occluded cube-tower counting, presents a stack of cubes in which lower cubes may be hidden by those above. The model must infer how many cubes total are present, including those not directly visible. This is a form of occlusion reasoning: the correct answer cannot be read off the visible surface but must be derived from visible geometry. GPT-5.5 reached 33.3%; Claude Sonnet 5, 18.7%; humans, 68.8%.

The third task, 4-neighbor connected-component counting, presents a grid image and asks how many distinct connected regions exist using 4-adjacency — meaning regions are connected only through up, down, left, and right neighbors, not diagonals. Unlike the other two tasks, this one requires non-local reasoning: a model cannot identify connected components by attending to local patches in isolation; it must mentally traverse the entire grid to determine what joins to what. Claude Sonnet 5 scored 7.3% — barely distinguishable from chance on grids of typical complexity. GPT-5.5 reached 28.3%. The human baseline was 64.3%.

Benchmark Design and Why Oracle Labels Matter

Each of StateSight's three task families contains 300 single-image prompts, for 900 total evaluation items. Human participants (30 in total) each answered 60 items — 20 per task family — and their responses were scored against the same oracle labels used to evaluate the models. The oracle labels are generated deterministically by the code that creates the benchmark, which means there is no annotator disagreement, no ambiguity in ground truth, and no possibility that a judge-model's own limitations inflate a tested model's score.

That last point matters. A growing share of AI benchmarks now use strong language models to evaluate responses, which creates a circularity problem: a model from the same family as the judge may benefit from stylistic familiarity or shared error patterns. StateSight sidesteps this entirely. A response is either correct or it is not, and correctness is established by an algorithm, not an opinion.

The paper also introduces StateSight-Steps, a companion dataset of 900 interleaved image-text examples with 3,600 deterministic intermediate visual states. StateSight-Steps is designed to support training approaches that provide step-by-step visual supervision — explicitly labeling the intermediate spatial states a correct solver would construct on the way to the final answer. Whether fine-tuning on StateSight-Steps actually closes the human-AI gap on these tasks is left as an open research question.

The Format-Fluency Dissociation

The most operationally significant finding is that every final direct run produced zero format errors. Both GPT-5.5 and Claude Sonnet 5 generated confident, properly structured answers to all 900 items. The failure is not that the models produced garbled output or refused to answer — it is that they produced fluent, parseable, confident wrong answers.

This dissociation between format fluency and spatial accuracy has direct implications for practitioners who use output quality as a proxy for reasoning correctness. In production agentic systems, response parsing and format validation are standard safeguards. StateSight's finding suggests that these safeguards catch nothing when the error is in spatial inference rather than output formatting. A robot controller, a logistics inspection system, or a 3D layout planner receiving a format-valid response from a VLM cannot infer from that validity whether the spatial inference behind it is correct.

The paper's abstract frames this as the core diagnostic: format-valid responses can mask failures to recover the spatial structure required for verifiable visual inference.

Read more: When AI outputs mislead: research on how verbal reports can mask what models actually compute

How StateSight Fits the Broader VLM Spatial Research Landscape

StateSight is not the first paper to document that frontier VLMs underperform humans on spatial reasoning, but it is meaningfully differentiated from prior work in its scope and methodology. OmniSpatial (published at ICLR 2026) tested state-of-the-art models across 50 fine-grained subtasks and found best-in-class models peaking at roughly 57% accuracy — more than 30 percentage points below human performance — but it used a broad, multi-domain design in which spatial reasoning was combined with other capabilities. CityCube (ACL 2026) found essentially no correlation between which tasks were hard for humans and which were hard for models, suggesting VLMs are sensitive to low-level visual features rather than performing human-like spatial modeling.

StateSight's contribution is isolation: by stripping out real-world objects, text, domain knowledge, and language priors, and by requiring latent spatial-state reconstruction specifically, it produces a signal that is harder to attribute to memorization, pattern matching, or shortcut exploitation. A VLM cannot recognize the correct folded-cube face because it has seen photographs of that cube in training data; the cube in each StateSight item was generated procedurally for that prompt.

GPT-5.5's lead over Claude Sonnet 5 is consistent across all three tasks, most dramatically on connected-component counting (28.3% vs. 7.3%). This specific gap is notable because Roboflow's Vision Evals study, published in July 2026, found Claude Sonnet 5 and Sonnet 4.6 essentially tied at 70% on 67 real computer-vision prompts, trailing Claude Fable 5 at 75% and Gemini 3.5 Flash at 79%. StateSight's connected-component task appears to exploit a specific weakness in Sonnet 5's visual processing pipeline that does not show up in perception-oriented evaluations — likely because counting connected regions requires non-local graph traversal, which patch-based attention mechanisms are not architecturally optimized to perform.

What the Gap Means for Robotics, Agentics, and Deployment

The three StateSight tasks are not arbitrary puzzles. Cube-net reasoning tests the kind of mental simulation required for object-assembly verification — determining whether a robot has placed parts correctly based on a single overhead or side view. Occluded-tower counting tests inventory reasoning under obstruction — a standard problem in warehouse robotics, surgical instrument tracking, and logistics inspection. Connected-component counting tests structural parsing of grid layouts — relevant to PCB inspection, terrain analysis, and any task where an agent must identify separate discrete objects in a structured visual field.

The practical consequence is that engineers deploying GPT-5.5 or Claude Sonnet 5 in any of these domains now have quantified calibration data rather than general marketing claims about multimodal capability. GPT-5.5 was positioned at launch as a powerful agentic and computer-use model with strong visual capabilities; StateSight's scores show that visual task fluency does not automatically translate into reliable latent spatial-state reconstruction, leaving a substantial gap relative to human performance on these specific tasks. Neither OpenAI nor Anthropic has responded to the paper as of today's listing.

Limitations and What Comes Next

Several important limitations constrain what StateSight's results can and cannot establish. The human baseline involves 30 participants answering 60 items — a modest sample, and the paper does not report variance across individual human participants. The model evaluations used default API configurations; it remains unknown whether extended-thinking or chain-of-thought configurations for GPT-5.5 would improve scores on cube-net or connected-component tasks, and whether reasoning-augmented variants of Sonnet 5 would narrow the gap. The study also evaluates only two models, leaving open how Gemini 3.5 Pro, DeepSeek V4 Flash with vision, and open-weight models such as Qwen3-VL would perform on the same tasks.

The StateSight-Steps dataset — providing 3,600 labeled intermediate visual states — is potentially the more consequential long-term artifact. If step-supervised training on explicit spatial-state sequences can teach VLMs to mentally simulate folding or graph traversal rather than relying on pattern recognition, it could produce measurable gains on the benchmark's own tasks. Whether those gains generalize to unseen spatial tasks is the question that will determine whether StateSight-Steps is a training resource or only a diagnostic scaffold.

For researchers building the next generation of multimodal models, the paper's core finding establishes a concrete target: the failure is not in output formatting, not in domain knowledge, and not in perception of visible surfaces. It is specifically in the reconstruction of latent spatial states — the hidden structure that a full understanding of the image requires but no single attention operation can extract directly.