Affordance Field Intervention
Enabling VLAs to Escape Memory Traps in Robotic Manipulation
When the scene changes, VLAs replay the trajectory they memorized in training instead of going to where the object actually is. AFI is a training-free agent + policy framework. A high-level agent (GPT-4o) breaks the task into stages, tracks which stage the robot is in, and grounds the current target in 3D. When the frozen VLA is trapped, and only then, control skills take over: they move the gripper back onto a promising path and hand control back to the VLA.
The problem
VLAs fall into memory traps
Vision-Language-Action models map images and instructions straight to actions. Under distribution shift they often ignore the new scene and drive the end-effector toward where the object used to be in the training data. End-to-end policies have no explicit 3D reasoning to catch this mistake.
Agent + Policy
Put the gripper in the right place, then let the VLA take over
Pairing a reasoning agent with a learned policy has become one of the most active directions in robot learning. One idea keeps coming back: VLAs are strong at contact-rich, last-centimeter manipulation, but weak at deciding where to go once the scene changes. So an agent handles the where: it perceives the scene, tracks task progress, and calls motion skills. The how stays with the VLA. AFI, released on arXiv in December 2025, is built exactly this way.
Breaks the instruction into stages, tracks which stage the robot is in, and switches the active target (e.g. carrot → pot). Grounded-SAM and depth turn that target into a 3D spatial affordance field (SAF).
Control skills the agent layer runs on the SAF: a memory-trap detector that decides when to step in, affordance-guided rollback, and a waypoint planner that scores VLA candidates.
Acts by default. After an intervention it proposes action chunks from each waypoint and finishes the manipulation, grasp included. No fine-tuning.
AFI (this work)
A GPT-4o agent decomposes the task and tracks its stages. Control skills decide when to intervene (a proprioceptive memory-trap detector) and where the gripper should go (rollback and affordance-guided waypoints). The frozen VLA decides how to finish the task. Repositioning the gripper before handing back to the VLA raises OOD success by +23.5% on real robots and +20.2% on LIBERO-Pro.
ENPIRE: Agentic Robot Policy Self-Improvement in the Real World
Their coding agent combines VLAs with procedural tool calls for long-horizon manipulation, and reports that it
“boosted the success rate of the GR00T VLA by using motion planning and detection tools to hover above an object before grasping.”
GPT 6 Astra as an Embodied Policy
π0.5 proposes action chunks, and GPT 6 Astra either accepts them or applies a short end-effector correction. On ten RoboDojo tasks the hybrid reaches a 48% success rate, compared with 26% for GPT 6 Astra alone. The model corrects only 14.4% of executed steps.
GPT-4o for perception, skills written by hand
GPT-4o was not yet reliable enough to steer an arm in closed loop. So AFI used it for what it did well: understanding the scene, splitting the task into stages, and noticing when a stage was done. When to intervene and how to move were written as explicit skills: a proprioceptive trap detector, affordance-guided rollback, and a waypoint planner that scores VLA candidates.
Agents that write and call their own skills
Current agents can design control skills and decide when to use them. ENPIRE's coding agent, for example, combines detection and motion-planning tools with a VLA and learns to hover before grasping. AFI's skills are the same kind of tool, validated on real robots and four VLA backbones. They are natural building blocks for such agents.
Method
How AFI intervenes
AFI runs as an agent + policy loop. The frozen VLA acts by default. The agent layer, GPT-4o plus three control skills, watches execution and takes over only when the VLA is trapped. Step through the loop below, or let it play.
Memory trap detection
A trap fires only when two conditions hold over a window Δt: the gripper is nearly static, and it is still far from the target. Standing still near the target is normal grasping. Standing still far away means the VLA is lost, so the agent takes over.
Affordance-guided rollback
From a buffer of recent end-effector positions, AFI returns to the one with the lowest affordance cost. That point becomes the root of the search. The path back was just traversed, so it is safe in static scenes.
SAF-guided exploration
Sample the N lowest-cost waypoints near the rollback point and move the gripper through them. At each one the VLA proposes K action chunks, which forward kinematics turns into end-effector paths. The path with the lowest cumulative SAF cost runs, and control returns to the VLA.
Agent · perception and stage tracking
The agent decides what to target, and when to switch
GPT-4o is the high-level agent. It breaks the instruction into ordered stages (e.g. carrot → blue pot), names the current target, and watches execution to detect when a stage is done, then switches to the next target. Grounded-SAM segments the target, and depth back-projects the mask into a voxelized workspace to build the spatial affordance field (SAF).
Two sub-fields are fused: a target guidance field that pulls toward the target centroid, and an obstacle avoidance field that pushes away from the scene geometry. After distance transforms and Gaussian smoothing, the result is a cost field in [0, 1] where lower means better. It is published at 2 Hz, and all three skills read from it.
Results
Consistent gains wherever the VLA gets lost
We evaluate on an AgileX Piper arm with two RealSense D435 cameras (20 trials per condition) and on LIBERO-Pro with spatial perturbations. AFI improves every backbone in every setting we tested.
Average success rate, without vs. with AFI
Mean over test conditions for each task and backbone
Show as table
| Setting | Backbone | Base VLA | + AFI | Gain |
|---|---|---|---|---|
| Place Carrot (real) | π0 | 61.0% | 87.0% | +26.0 |
| Remove Lid (real) | π0 | 63.0% | 80.0% | +17.0 |
| Slot Pen (real) | π0 | 60.0% | 82.0% | +22.0 |
| Stack Tape (real) | π0 | 64.0% | 86.0% | +22.0 |
| Stack Tape (real) | π0.5 | 61.0% | 82.0% | +21.0 |
| Place Carrot (real) | SpatialVLA | 53.3% | 78.3% | +25.0 |
| Place Carrot (real) | OpenVLA-OFT | 48.3% | 68.3% | +20.0 |
| Remove Lid + Place Carrot (long-horizon) | π0 | 43.8% | 63.8% | +20.0 |
| Remove Lid + Place Carrot (long-horizon) | π0.5 | 43.8% | 60.0% | +16.2 |
| LIBERO-Spatial (OOD) | π0.5 | 54.0% | 75.7% | +21.7 |
| LIBERO-Object (OOD) | π0.5 | 56.4% | 73.2% | +16.8 |
Real-world success rate by test condition
20 trials per bar · AgileX Piper
For reference, ReKep, a zero-shot VLM planner, reaches only 36.0% on Place Carrot, where π0-AFI reaches 87.0%. Pooling candidates from π0 and π0.5 under one SAF scorer gives 89.0% on Stack Tape.
Show full real-world table
| Task | Method | In Dist. | Position | Color | Task | Background | Average |
|---|---|---|---|---|---|---|---|
| Place Carrot | ReKep | 8/20 | 7/20 | 9/20 | 5/20 | 7/20 | 36.0% |
| π0 | 17/20 | 6/20 | 13/20 | 15/20 | 10/20 | 61.0% | |
| π0-AFI | 20/20 | 13/20 | 17/20 | 18/20 | 19/20 | 87.0%+26.0 | |
| Remove Lid | π0 | 20/20 | 8/20 | 17/20 | 5/20 | 13/20 | 63.0% |
| π0-AFI | 20/20 | 12/20 | 19/20 | 11/20 | 18/20 | 80.0%+17.0 | |
| Slot Pen | π0 | 16/20 | 11/20 | 13/20 | 15/20 | 5/20 | 60.0% |
| π0-AFI | 19/20 | 16/20 | 16/20 | 19/20 | 12/20 | 82.0%+22.0 | |
| Stack Tape | π0 | 18/20 | 9/20 | 16/20 | 13/20 | 8/20 | 64.0% |
| π0-AFI | 20/20 | 15/20 | 20/20 | 17/20 | 14/20 | 86.0%+22.0 | |
| π0.5 | 20/20 | 7/20 | 17/20 | 10/20 | 7/20 | 61.0% | |
| π0.5-AFI | 20/20 | 14/20 | 19/20 | 15/20 | 14/20 | 82.0%+21.0 | |
| π0+π0.5-AFI | 20/20 | 16/20 | 20/20 | 16/20 | 17/20 | 89.0%+25.0 |
Beyond single tasks
Long-horizon, cluttered, and other backbones
On the two-stage Remove Lid and Place Carrot task, AFI builds a separate SAF for each stage and switches between them mid-execution. It improves π0 by +20.0% and π0.5 by +16.2%. With 3–5 distractors on the table (Clutter), both backbones gain +25%.
AFI treats the VLA as a black box. It also improves the autoregressive OpenVLA-OFT (+20%) and the depth-aware SpatialVLA (+25%). SpatialVLA still drops from 75% to 30% under position shift, so training with depth alone does not escape memory traps.
LIBERO-Pro (spatial perturbations), per subtask
π0.5 official LIBERO checkpoint, without vs. with AFI
Ablations
What matters
All ablations use π0 on Place Carrot, 20 trials each. Without rollback, SAF waypoints can't recover from a position that is already bad. Fixed-step intervention trails adaptive detection. Around 10 waypoints balances guidance against over-constraining the VLA.
Components
Position shift · success rate
Number of waypoints
Position shift · success rate
Shift direction (cm)
π0 vs. π0-AFI · success rate
Does the field make sense?
SAF cost tracks task progress
Along successful rollouts, the SAF cost falls as the gripper approaches the current target. When the task moves to its next stage (e.g. from grasping the lid to placing it on the platter), the field re-targets: cost jumps up, then falls again toward the new goal.
Across more than 200 real-world trials, only about 5% of failures trace back to SAF guidance. Most failures come from the VLA itself (60%) or from contact physics (25%). We never observed the SAF sending the robot to a semantically wrong object.
Scope
Limitations
- Positional guidance only. The SAF tells the robot where to go. Grasp orientation, contact force, and tool use stay with the VLA. Tasks that need precise approach angles would call for SE(3) pose guidance.
- Static or semi-static scenes. Rollback replays a recently traversed path. Dynamic scenes would need explicit collision checking along it.
- Update rate. The SAF refreshes at 2 Hz through asynchronous ROS topics. Fast, dynamic tasks may need faster local VLM inference.
Citation
BibTeX
If you find AFI useful, please consider citing:
@inproceedings{xu2026afi,
title = {Affordance Field Intervention: Enabling VLAs to Escape Memory Traps in Robotic Manipulation},
author = {Xu, Siyu and Wang, Zijian and Wang, Yunke and Xia, Chenghao and Huang, Tao and Xu, Chang},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2026}
}