Our best vision models are strangely blind in 3D
We have vision-language models (VLMs) that can read a menu, describe a meme, and explain a chart. Give one a photo and it will happily tell you there's "a green bowl on a wooden table."
Now ask that same model where the bowl is — its position in space, how far it is from the cup next to it, how much volume it occupies, whether a robot arm could fit a gripper around it. This is where most 2D VLMs get shaky. They were trained on flat pictures scraped from the web, so they inherited human-like 2D intuition but very little grounded, metric, three-dimensional understanding.
That gap is not academic. It's exactly the gap standing between "AI that talks about the world" and "AI that acts in the world" — the whole premise of Physical AI, the theme dominating robotics and AI in 2026.
VoxRep, a short 2025 paper from Menlo Research by Alan Dao and Norapat Buppodom (APSIPA ASC 2025), proposes a refreshingly low-tech way to close some of that gap.
Treat a 3D scene like a CT scan
Start with a voxel grid — 3D space chopped into little cubes (voxels), like Minecraft or a medical scan. Each voxel can store whether it's occupied and, say, its color. It's a clean, structured way to represent a 3D scene, and it's a format robots and simulators already produce constantly.
The problem: how do you get semantics out of raw voxels — "there's a green bowl, centered here, about this big"? The usual answer is to build a specialized 3D neural network. Those work, but they're heavy, they need 3D-specific training data, and they don't get to ride the enormous wave of progress happening in 2D foundation models.
Don't build a 3D model. Slice the 3D volume into a stack of 2D images — exactly like a CT scanner slices the human body — and hand that stack to a normal 2D VLM.
The pipeline: slice → pad → tile
- Slice. A 100 × 100 × 16 voxel grid is cut along the vertical (Z) axis into 16 flat slices, each a 100 × 100 cross-section at a given height.
- Pad & resize. Each slice is padded to 112 × 112 and scaled up to 224 × 224 (so every voxel becomes a tidy 2×2 pixel block).
- Tile. The 16 slices are laid out in a 4 × 4 grid to form one big 896 × 896 image — a "contact sheet" of the whole 3D scene.
That single tiled image is fed to the vision encoder of Gemma 3 (4B), an off-the-shelf open VLM. The model's language decoder then outputs structured "voxel semantics" — a JSON list, one entry per detected object:
{
"id": "0",
"color": "dark_green",
"description": "bowl",
"number_of_occupied_voxel": 1539,
"voxel_coords_center": { "x": 71, "y": 64, "z": 5 }
}Identity, color, size, and a 3D center — all read straight out of a voxel grid, by a model that was never designed to see in 3D.
Why this is genuinely novel
A lot of "3D + LLM" work either (a) trains bespoke 3D architectures or (b) renders a scene from a few camera angles and feeds those photos in. VoxRep does neither. Its bets are what make it interesting:
No specialized 3D architecture
VoxRep adapts an existing 2D VLM with a preprocessing pipeline (slice → pad → tile) rather than introducing 3D convolutions or a custom point-cloud encoder. That makes it practical to evaluate newer VLM backbones with the same basic interface.
The encoder does the 3D reasoning
The striking finding is that the vision encoder itself — pre-trained only on natural 2D photos — can learn to aggregate information across slices and reconstruct a coherent sense of 3D structure. 3D understanding emerges at the vision-encoder level, without 3D convolutions.
Cheap and scalable
No expensive 3D-specific pretraining. You fine-tune an existing model on synthetic data and you're off. Training used just 20,000 synthetic scenes from ModelNet40.
Results after training
| Capability | Metric | Before → after |
|---|---|---|
| Localization | Avg. center distance | 26.05 → 9.17 voxels |
| Color recognition | Accuracy | 0.22 → 0.78 |
| Volume estimation | Avg. voxel-count error | 438 → ~185 |
| Object classification | Accuracy | 0.18 → ~0.58 |
The honest part: object classification stayed hard (~0.58). Telling a stool from a chair from voxel slices is genuinely tough — the flattening can wash out fine 3D shape cues. Location, color, and size are where slicing shines. It's tested on clean synthetic data, not noisy LiDAR/depth from the real world, and on mostly isolated objects, not cluttered scenes.
Where Physical AI comes in (2026)
2026 is being called the year Physical AI "gets a body." The mainstream framing — from NVIDIA, IBM, and others — is simple: Physical AI is AI that can perceive, reason about, and act in the physical world, pairing foundation models with sensors and actuators.
The catch is that acting in the world demands the exact skill web-trained VLMs are weakest at: grounded 3D spatial understanding. A robot doesn't need a poetic caption of your kitchen; it needs to know what is where, how big it is, and how to reach it.
Voxels are already the robot's native language
Occupancy grids and voxel maps are bread-and-butter in robotics — how autonomous vehicles, drones, and mobile manipulators represent free vs. occupied space. VoxRep takes the representation robots already build and adds a language-grounded semantic layer on top — object identity, attributes, and metric location — in a format (JSON) that downstream planners can consume directly.
It rides the sim-to-real pipeline
Physical AI is trained overwhelmingly in simulation before hitting the real world. Voxels are trivial to generate and auto-label in simulation — which is exactly how VoxRep was trained. After real depth or LiDAR observations are voxelized, the same slice-and-tile representation can, in principle, be produced from either a simulator or a sensor pipeline.
A simpler systems interface
Replacing a specialized 3D perception architecture with a standard VLM plus deterministic preprocessing can simplify experimentation and deployment. The potential efficiency claim is about avoiding a separate 3D model family, not proof of real-time robot performance.
The honest connection: VoxRep is not a robot and doesn't claim to be. It tackles one early link in the Physical AI chain — turning a structured 3D representation into language-grounded semantics — and it does so on synthetic, tidy data. It's a proof of concept that the perception-to-semantics step can be borrowed almost for free from 2D foundation models.
A better way of looking
VoxRep's charm is that it reframes a hard problem into an easy one. Instead of asking "how do we build a 3D-native VLM?", it asks "what if we just show a 2D VLM the 3D scene the way a radiologist reads a CT scan?" — and shows that a general-purpose model can pick up spatial location, color, and volume from that view remarkably quickly.
In the context of 2026's Physical AI push, that's exactly the kind of pragmatic bridge the field needs: a way to reuse the giant, fast-moving world of 2D foundation models for the grounded, three-dimensional understanding that robots and embodied agents actually require.
The classification numbers say there's plenty left to do, and real-world sensor noise is the next mountain to climb — but the direction is a good one. Sometimes the most useful idea isn't a bigger model. It's a better way of looking.
