Test-time Skill Composition (Unseen in Training)
Why Robotics Needs Search and Test-Time Computation
A robot operating in the physical world cannot be shown every situation it will encounter. Objects begin in new poses, obstacles create unfamiliar constraints, tools have different reach, and small execution errors produce states that were absent from training. Even when the robot already possesses every individual behavior needed to solve a task, the number of useful sequences, handoffs, intermediate goals, and recovery strategies is combinatorial.
This is the same reason search matters elsewhere in intelligence. AlphaGo did not rely on its policy alone; it used that policy to guide search through possible futures. AlphaGeometry combined learned proposals with explicit deduction. In language models, test-time scaling increasingly means generating, evaluating, refining, and selecting among candidate solutions rather than committing to the first trajectory produced by the model. Across these systems, learned inference supplies a strong prior, while search resolves what the prior has not already encoded.
Robotics has the same need. Imitation and reinforcement learning can compress frequently useful behavior into a policy, making familiar decisions fast. But a fixed policy has no explicit mechanism for comparing counterfactual futures when the right solution is a novel composition: push around the obstacle, move the object into another arm’s reach, attempt a grasp, and recover if the grasp changes the scene. Covering every such trajectory with demonstrations is infeasible. The alternative is to construct a solution at inference time.
The important question, however, is not simply whether a robot should search. It is where in the robot’s decision process that search should occur. Search at the wrong level can be either computationally intractable or too abstract to produce a useful physical plan. To answer where it belongs, we need a clearer picture of the decisions a robot is making at any instant.
The Abstraction Hierarchy Tells Us Where Search Is Useful—and Where It Is Not
To help us answer the question of where search should apply in robotics, we introduce the concept of an abstraction hierarchy.
Robotic behavior can be understood as an abstraction hierarchy: an implicit hierarchy of goals, subgoals, temporally extended behaviors, and low-level actions that describes the robot’s “state of mind” at a given moment. The levels of the abstraction hierarchy have different temporal and computational properties:
- At the top: Task goals. This specifies what should be achieved, and remains constant throughout, but omits the intermediate structure needed to achieve it.
- In the middle: Subgoals and strategies. Choices such as clearing an obstacle, moving an object into reach, or deciding which tool should act expose the high-level, abstract structure of a solution.
- At the bottom: Raw actions. These are fast, continuous, and high-dimensional decisions such as joint targets or end-effector poses, which may change dozens of times per second.
In its most general form, the abstraction hierarchy is better thought of as a spectrum instead of discrete levels. However, this mental model is useful enough to tell us which computational tools fit which parts of the problem, and identify limitations in prior work that we aim to address.
Prior Robotics Research Suffers from Abstraction Mismatch
Without a conscious understanding of this hierarchy, it is easy to apply a powerful computational tool at a level that it is unsuited to handle. This mismatch has manifested in different ways:
Mismatch 1: World-model search over low-level actions
Learned world models have enabled robots to predict candidate futures and plan toward new goals. Deep Visual Foresight and Visual Foresight optimize action sequences through predicted visual dynamics; DayDreamer learns latent dynamics for physical control; and DINO-WM demonstrates zero-shot planning with learned visual features.
These works establish the value of predictive models. But when world modeling and planning are conditioned on raw or near-raw actions, the planner operates near the bottom of the abstraction hierarchy.
At this level, two factors work against search: high-dimensional controls create a large branching space, and fine control frequency means that reaching the same physical time horizon requires many more model steps. A planner must therefore evaluate more candidate trajectories, roll each one farther using more world model forward passes, and often run more optimization rounds such as CEM iterations.
Mismatch 2: Standard policy inference over high-level strategy
The reverse mismatch occurs when a reactive policy is asked to absorb the higher levels of the hierarchy. Large robot policies trained on broad datasets—including Open X-Embodiment and RT-X, Octo, OpenVLA, and π0—show how much perception, language grounding, and control can be compressed into a single model.
This approach provides fast inference and strong behavior where the learned distribution offers a reliable answer. However, since skill composition and search remains implicit in the weights, this approach will likely fail for scenarios that require a rare handoff, an unseen ordering of known behaviors, or a subgoal absent from the demonstrations.
The lesson is a division of labor. Policies are well suited to amortizing repeated, feedback-driven control. World models and planners are well suited to evaluating consequential alternatives. A robotic architecture should preserve both, while assigning each to the region of the hierarchy where its strengths matter most.
A Design Simplification: An Abstraction Boundary Defines the System’s Skills
Although the abstraction hierarchy is closer to a continuous spectrum than a set of universal layers, for this work, we make a deliberate simplification: draw one abstraction boundary that cleanly separates deliberation from execution.
- Above the boundary, world models and a planner reason at test time over slow, low-dimensional, semantically meaningful choices.
- Below the boundary, reactive policies handle fast, stochastic, high-dimensional control.
We call the behavioral unit exposed by this boundary a skill. In our work, a skill is defined by a skill policy model and a skill world model pair. The closed-loop policy model handles the low-level mechanics needed to execute the skill, and the world model provides temporally extended, subgoal-level predictions that are useful for a planner.
In this design, the planner does not choose every joint target involved in pushing; it chooses push toward this subgoal, then lets the pushing policy absorb the details.
However, the boundary cannot be placed arbitrarily. A useful skill must be low-level enough to expose meaningful intermediate behavior, yet high-level enough that the planner never reasons over raw control. A good skill is therefore:
- temporally extended enough to avoid high-frequency raw-action search;
- closed-loop enough to absorb low-level stochasticity;
- semantically meaningful enough for a planner to compose;
- predictable enough for a world model to estimate its aggregate effect; and
- fine-grained enough to expose the intermediate choices needed to solve novel tasks.
What counts as a good skill depends on the task and embodiment. A useful unit for tabletop manipulation may be wrong for locomotion or deformable-object handling. In this proof of concept, we make the simplifying choice of a small, manually specified set of discrete skills. More general systems may learn their boundaries, use several nested boundaries, or trade some planning efficiency for a broader and less handcrafted behavioral interface.
Respecting the Abstraction Hierarchy: Skill-Level Search, Action-Level Policies
The resulting design is simple: perform prediction, planning, and test-time computation at the skill level, while leaving raw control to skill policies. The planner asks which temporally extended behavior to invoke and which subgoal to give it. The selected policy then converts that choice into continuous actions using live feedback from the environment.
This is the robotic analogue of guided search elsewhere in AI. The learned policy provides an efficient behavioral prior, but the system can spend additional computation when the current scene demands a novel composition. Crucially, that compute is spent on the choices most likely to benefit from deliberation—not on rediscovering every motor command inside a behavior the policy already knows how to execute.
In the next section, we choose a robotic task and embodiment designed to test the proposed system’s generalization and skill composition ability from unseen and rare skill sequences.
Setting: Collaborative Pushing and Picking on Tabletop Mazes
Our demonstrations are designed as a proof of concept for the abstraction-boundary principle, not as a broad benchmark study. We ask whether isolated skills, skill-level world models, and test-time search can produce compositional behaviors that were never present in training, all without relying on teleoperation data.
To this end, we evaluate in a dual-arm tabletop manipulation task where the goal is to lift a movable cube to a target height; however, direct grasping may be impossible because the cube is blocked by obstacles, outside the gripper’s reachable region, or both.
The embodiment therefore creates a natural compositional structure: a non-prehensile stick arm can reposition the cube along the table and maneuver the cube around obstacles, while a prehensile gripper arm can lift it once it is unobstructed.
In the next section, we describe our choice of skill policies and world models. We show how they are trained only in single-arm settings, while evaluation occurs afterward in the full dual-arm environment. Thus, any successful stick-to-gripper sequence, inferred subgoal, or recovery handoff must be constructed at test time, without the system ever seeing a full end-to-end problem-solving trajectory at any point during training.
Skill Policies and Skill World Models
We instantiate the boundary with one goal-conditioned policy and one world model per skill. For skill k, the policy πk receives the current state and a subgoal, then emits low-level actions in a closed loop. Its corresponding world model fk predicts the task-level state that will result when that policy attempts the subgoal.
In our setting, we instantiate two skills: the stick skill, and the gripper skill:
We use separate models because each skill induces a qualitatively different transition distribution. A pushing rollout contains planar contact and displacement; a grasping rollout contains approach, closure, lift, and possible slip. A skill-specific model sees the more homogeneous prediction problem generated by one policy operating in one behavioral mode. This lets each model specialize in the capabilities and failure modes of its skill, and makes the system modular: adding a skill adds one policy and one corresponding predictive model.
This is a design choice, not a claim that every robotic system should use one network per skill. A shared model conditioned on skill identity may transfer structure across behaviors; learned latent skills may reduce manual specification; and a larger general-purpose model may trade efficiency and interpretability for broader coverage. Our per-skill construction is intentionally handcrafted. It isolates the abstraction-level principle and gives us a clean test of whether separately learned behaviors can be composed without composite demonstrations. Generalizing the interface is future work.
Skill-Level Search Using CEM, Action-Level Control using Policies
With skill world models in place, the robot can search in imagination at the level of (skill, subgoal) pairs. A candidate plan might be: push toward g1, push toward g2, then grasp and lift toward g3. Rolling the candidate through the corresponding sequence of skill world models predicts its terminal state without executing it in the environment.
We optimize these sequences with the Cross-Entropy Method (CEM). The planner samples a population of candidate skill-subgoal sequences, imagines each sequence through the appropriate world models, scores the predicted terminal states against the task goal, retains an elite set, and resamples around the most promising candidates. After several CEM optimization rounds, the system executes only the first pair from the best sequence by calling the corresponding skill policy model. It then observes the real state and searches again, allowing model error, slips, failed grasps, and incidental object motion to trigger a new plan.
The gain is not that skill-level planning makes search free. Candidate subgoals must still be proposed, world-model uncertainty still matters, and larger skill libraries create their own scaling challenges. The gain is that the search budget is spent at a level where alternatives are compact, consequential, and composable. In our demonstrations, meaningful multi-skill behavior emerges with planning horizons as short as five skill invocations—while every invocation can contain many low-level policy decisions.
This gives us the architecture explored in the rest of the post: independently trained skill policies, independently trained skill world models, and a test-time planner that composes them into behaviors that never appeared end to end in training.
Demonstrations
What Emerges When a Robot Reasons at Test Time
The central result of this work is not a task solved but a class of behaviors that emerge. When a system is equipped with predictive models and the ability to reason through them at test time, it exhibits capabilities that were never demonstrated, never explicitly trained, and never anticipated by the training pipeline. Rather than retrieving a solution, the model constructs one, and that distinction is the hallmark of genuine test-time intelligence.
We consistently find that our system smoothly stitches skills together to achieve goals that are not directly achievable or that require intelligent, constraint-aware subgoal discovery, with dynamic error recovery achieved by handing responsibility between skills, sometimes multiple times within an episode. We also find it robust across many different environments, a wide range of obstacle complexities including moving obstacles, and across different robot embodiments.
(1) Unseen Composite Trajectories
To successfully perform this task, our system was able to construct composite skill trajectories dynamically at test time: first use the stick to move the cube away from obstacles, then lift the cube using the gripper. No version of this full sequence existed in training; it was reasoned into existence at inference time.
It is important to remember that the stick arm and gripper arm (each with their own skill policies and skill world models) were trained fully separately: at no point in training did the system see this full, compositional trajectory. Based on the state of the environment, the system successfully inferred that directly lifting the cube is not an option, and planned a composite trajectory using the complementary strengths of the two arms. The planner did not find this trajectory in memory; it derived it through active reasoning over its world models.
This approach contrasts sharply with teleoperation-based systems, which would have to cover such composite trajectories within the training data. As the set of individual skills becomes larger, the harder it is to augment the training set with useful composite trajectories, an issue that test-time reasoning systems are much less susceptible to.
(2) Subgoal Discovery
The system was able to effectively infer subgoals that must be achieved before the final objective (lifting the cube) became feasible. Our planner infers which skills should activate at each step, and the corresponding world model predicts the resulting state. By visualizing those predictions, we can see the inferred subgoals explicitly, providing a direct window into the system's test-time reasoning process.
(3) Self-Correction and Replanning
Finally, we observed that the system dynamically hands responsibility back and forth between the two tools/skills based on the current state of the environment. This is not a pre-programmed fallback but the planner continuously re-evaluating its world model predictions against observed reality and reassigning control when the two diverge.
The clearest example is when the system first attempts to pick the cube immediately, then realizes it cannot (because of an obstacle), then switches to the stick tool until the cube is free and within range of the gripper. In some cases we saw multiple handoffs when the gripper slipped and pushed the cube away, handing the task back to the stick arm until the system was confident the gripper could complete the lift. This adaptive replanning loop is precisely the kind of behavior that test-time cognition makes possible, and it is one that no amount of additional training data could fully anticipate.