MessyNav: Zero-Shot Navigation via Manipulation in Messy Environments

Abstract

Real-world environments are messy. Paths are often blocked by everyday objects like chairs and toys, requiring interactive navigation (the ability to alter the environment to clear a path) rather than standard navigation alone. Existing methods rely on extensive simulation-based learning or per-scenario instructions specifying which objects to avoid or manipulate. They struggle to generalize across novel objects, diverse environments, and varied contexts such as a cart of fragile glassware that should be avoided while an empty one can be pushed. We formalize this problem as open-world interactive navigation and present MessyNav, a zero-shot framework based on vision-language models (VLMs). At its core, MessyNav performs explicit VLM-based affordance reasoning before task planning to decide whether to push, pick up, or avoid an object and produce a high-level plan. Deployed on a mobile manipulator across multiple real-world indoor and outdoor environments, MessyNav achieves a 27% higher success rate than existing baselines without any task-specific training.

The problem

What makes Interactive Navigation Hard?

Interactive navigation is hard because the robot must decide what it can safely move before it acts. That judgment has to hold zero-shot across novel objects, unfamiliar environments, and shifting context. A wrong call often cannot be undone, so the robot has to get it right the first time.

Framework

MessyNav Framework

We present MessyNav, an interactive navigation framework that leverages VLMs for zero-shot affordance reasoning and task planning, to go beyond closed-set prediction or environment-specific tuning.

Our central design decision is to separate what to interact with from how to interact: an explicit affordance reasoning stage first decides which objects can be moved, and only then does a task planner decide how to move them. We make affordance reasoning an explicit stage rather than letting a task planner implicitly handle affordance, allowing it to make more grounded decisions in context-dependent scenarios.

1. Affordance Reasoner Identifies which objects are interactive based on robot feasibility and object properties.
2. Task Planner Generates a sequence of navigation and manipulation actions based on the affordance labels to clear the blocking objects and reach the goal.
3. Open-Language Skill Library Grounds open-language object references in task plans to 3D world coordinates and executes motion primitives.

Interactive demo

Affordance Reasoning Interactive Demo

See MessyNav predict the interactivity labels of objects. Upload an image.

Results

Affordance reasoning

We first evaluate affordance reasoning in isolation from the full framework. Given an RGB image containing one or more obstacles, the task is to classify each obstacle as interactive (it can be swiped or picked up by our robot) or non-interactive (it must be avoided). We collect and manually annotate a benchmark of 100 real-world scenes totalling 217 objects across 14 locations. It spans 69% indoor and 31% outdoor scenes; bright artificial (66%), natural (28%), and dim (6%) lighting; 1–4 foreground objects per scene; and obstacle sizes from small electronics to large trees. Of the 217 objects, 94 (43.3%) are interactive and 123 (56.7%) are non-interactive. Labels were annotated by an expert familiar with the robot's physical capabilities, and the set is used only for evaluation, never for training or finetuning.

Figure 1: Affordance accuracy. MessyNav reaches 85.7% and outperforms all baselines. Even closed-set-oracle [4], an upper bound whose categories are drawn from the evaluation set itself, reaches only 66.4%, while the height heuristic [5] and closed-set [4] baselines reach 59.9% and 24.4%. Takeaway: predefined category-level labels are insufficient for context-dependent affordance reasoning.

Example scenes from the benchmark. Hover over an image to reveal the annotated evaluation view: each object is boxed and labeled with its affordance ([I] interactive, highlighted green; [N] non-interactive, highlighted blue).

Real-world interactive navigation

We then evaluate full end-to-end execution on a real robot across 15 real-world scenes in two environments, a messy bedroom and a conference room, with 5 object types and 3–7 context variations, for a total of 60 robot trials across the four methods. Success rate is the percentage of trials where the robot reaches the goal without executing unsafe actions. We deploy MessyNav on a TidyBot++ platform: a 7-DoF Kinova arm on a holonomic mobile base, with a Unitree L2 LiDAR and two ZED 2 cameras.

Table 1: Real-world interactive navigation. Execution time excludes VLM queries and is averaged over successful trials. The two ablations isolate our core design decision: removing the affordance reasoner (vlm-planner-only) drops success from 67% to 40%, and replacing it with fixed category-level labels (closed-set-planner) drops it further to 27%. Takeaway: inaccurate prior affordances are worse than none at all.

Scaled offline evaluation

To quantify VLM variability and isolate high-level decision quality from low-level execution failures, we additionally evaluate MessyNav and the strongest baseline offline over 15 repeated trials across 55 scenes in 5 environments (n = 825 per method), assuming perfect low-level execution.

Table 2: Scaled offline evaluation. MessyNav improves overall success by +7.0 pp and wins in 13 of 15 runs, a statistically significant gain under a paired t-test over per-run success rates (paired t(14) = 4.71, p < 0.001; 95% CI on the difference [+3.8, +10.2] pp). Gains are largest outdoors in daylight. Nighttime remains challenging for both methods and is the one setting where MessyNav trails the baseline, highlighting robustness under degraded visual conditions as important future work.

Example scenes from the offline evaluation. Egocentric robot views drawn from the 55 scenes across 5 environments, spanning indoor spaces, outdoor paths in daylight, and outdoor walkways at night.

Cite

BibTeX

@inproceedings{llontop2026messynav,
  title     = {MessyNav: Zero-Shot Navigation via Manipulation in Messy Environments},
  author    = {Llontop, Kenneth and Ho, Cherie and Zhou, Dylan and Banwasi, Anuva
               and Sundaresan, Priya and Parés-Morlans, Carlota and Bohg, Jeannette},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2026},
}

Acknowledgments

This work was supported in part by Toyota Research Institute. Kenneth Llontop is supported by the SEED Scholarship at the University of California, Berkeley, SURF at Stanford University, and NSF CISE REU. Cherie Ho is supported by the Croucher Fellowship for Postdoctoral Research. Priya Sundaresan is supported by a National Science Foundation Graduate Fellowship. Carlota Parés-Morlans is supported by a graduate fellowship from Knight-Hennessy Scholars at Stanford University. We thank Satvik Sharma, Francis Engelmann, and Jennifer Grannen for helpful discussions.