A robot can understand a request and still need permission to act on it. This prototype builds on Metacognition’s ZEOS, which supplies the runtime and action-gating mechanism. I wanted to explore what that distinction could look like inside a vision-language-action system: a model that connects what a robot sees and what we ask it to do with physical movement.
This project is a proof of concept for putting an explicit decision boundary between a model’s proposed intent and the policy that produces motion. The interesting question for me was whether those responsibilities could remain separate as the models around them change.
From understanding to action
The prototype uses a frozen Gemma backbone to interpret an image and instruction. Its representations provide context for a learned flow-matching action head, while a separate semantic proposal describes the intended action in a form that a decision system can evaluate.
ZEOS enforces the permission step. Approval allows the original representations to reach the action head; a veto stops the cycle before the head runs. The semantic proposal is what gets evaluated. The model’s internal representations remain the context used to generate movement.
I like this separation because it makes a difficult system easier to question. What did the model propose? Why was that proposal allowed? Did the policy actually run? Those questions should have answers I can inspect.
Keep the decision layer replaceable
I used a simple rule-based guard for this experiment. It gave me a clear way to test the boundary before introducing more sophisticated judgement.
That guard is an interchangeable part of the design. A system such as TypeSafe’s Jev, OpenAI’s Decisions API, or a custom implementation could provide decisions through the same interface. Those are possible extensions, rather than components already integrated into this prototype.
The broader idea is to improve how permission is decided without rebuilding the manipulation policy every time.
Seeing the boundary work
The experiment exercised both paths: approval ran the action head, while rejection produced no action output. A separate Isaac Sim replay made the difference visible as an arm moving in the allowed case and holding position in the vetoed case.
This establishes an execution boundary. It does not establish that every approved trajectory fulfils the proposal or succeeds at the physical task. Connecting intent, permission, and successful manipulation is the next question I want to explore.
For me, the useful outcome is a modular starting point for that work: a way to change the reasoning, decision system, or action policy while keeping the relationship between them understandable.
Explore the code · Built with ZEOS, Gemma, and Isaac Sim.