How much useful behaviour can a robot learn from nine demonstrations? I wanted to try a small, understandable starting point before reaching for a larger model or a long training run.
I built a pick-and-place experiment using a wrist camera and visual imitation through nearest neighbours, or VINN. The idea is straightforward: look at the current scene, find similar moments in previous demonstrations, and use the actions associated with those moments to decide what to do next.
The demonstrations stay available as examples. That makes it possible to inspect where an action came from and think deliberately about which new experiences might help.
Seeing the right context
One of my first instincts was to isolate the target object. If the robot needs to pick something up, surely a cleaner view of that object should help.
In practice, the surroundings carried information I needed. The relationship between the gripper, object, and destination helped explain what stage of the task the robot had reached. Removing that context could make visually neat inputs less useful for choosing an action.
It changed how I think about representations. The useful information is whatever helps explain the interaction, including parts of the scene that initially look like background.
A demonstration has a direction
The most interesting failures were easy to understand once I looked at the retrieved examples. At the start of a run, the robot could match a stationary moment in a demonstration and keep choosing to wait. Similar views during approach and retreat could also suggest opposite movements, producing a rocking behaviour.
Those failures were reminders that visual similarity alone does not describe progress. The same scene can call for a different action depending on what has just happened and what the robot is trying to complete.
Giving retrieval a sense of task phase and forward progress helped make the behaviour more coherent. Careful demonstration boundaries mattered too: a recording includes the operator’s pauses and resets as well as the intended skill.
What nine demonstrations taught me
The robot handled modest changes in object placement around a constrained bench setup. That was encouraging for such a small dataset. The wrist camera also made its limitations visible: judging depth and maintaining clearance were harder than finding a visually similar example.
What I liked most was how much the experiment exposed. I could follow an unhelpful action back to a demonstration, recognise missing context, and collect another example with a specific purpose. The policy became a useful way to investigate the data and the task together.
I see small retrieval-based policies as valuable baselines for robot learning. They offer a quick way to ask whether the observations contain the information a task needs and whether the demonstrations cover the situations the robot encounters. Nine demonstrations gave me both a working prototype and a much clearer picture of what to improve next.
Watch the original demonstration · Related: Remote workstation