Compose Multi‑Step Robotic Tasks with Confidence
If you can describe a task clearly, your robot can likely do it. That’s the promise of Gemini Robotics 1.5 and ER 1.5—models built for grounded, embodied reasoning that turn high‑level intent into reliable, multi‑step action plans in the physical world. Below are 25 proven prompt templates—organized by intent—that help you compose robust, multi‑step workflows for real‑world robotics.
Style note: Practical & solution‑oriented. Each template includes the structure, recommended guardrails, and optional variables. Replace placeholders like {OBJECT}, {LOCATION}, {POLICY}, and {CONSTRAINTS} with your context.
How to Use These Templates
- Start with a high‑level goal, then enumerate steps with sensor checks and recovery behavior.
- Include constraints: safety, speed/precision, environment assumptions, and fallback strategies.
- Provide state feedback channels (e.g., vision success criteria, force/torque thresholds).
- Prefer declarative goals over brittle step-by-step micromanagement; let the model plan and adapt.
By the way, if you orchestrate prompts, logs, and iterations across a team, a side panel assistant like Sider.AI can help you draft, test, and refine prompts alongside your docs and code, keeping context visible as you iterate on your robotic skills and procedures . Section A — Planning & Grounding (Foundations)
- Task Blueprint (Goal → Constraints → Plan → Checks)
- Prompt
"You are controlling a mobile manipulator.
Goal: {GOAL}.
Environment: {DESCRIPTION}; known objects: {OBJECT_LIST}.
Constraints: {CONSTRAINTS}.
Output: 1) Assumptions to verify, 2) Ordered plan with perception/action steps, 3) Safety checks per step, 4) Recovery behaviors, 5) Termination conditions and success metrics."
- Use when: Converting a high‑level goal into an operational plan with guardrails.
- Perception‑First Plan with Uncertainty Quantification
- Prompt
"Before acting, build an observation model. Identify required observations, confidence thresholds, and edge cases for {GOAL}. Output JSON:
{ observations:. For broader prompt and agent patterns, Google’s Gemini cookbook and agent guides are helpful references.
Example: End‑to‑End Prompt for a Kitchen Task
Goal: Prepare a simple salad and pack it to‑go.
Prompt
"You are controlling a 7‑DOF mobile manipulator with a parallel gripper and tool‑changer.
Goal: Prepare and pack a salad with lettuce, tomato, cucumber, and dressing.
Environment: Kitchen island with sink, cutting board, chef knife, salad bowl, lunch container. Humans may be present.
Constraints: No blades within 0.5 m of humans. Knife edge always sheathed unless cutting. Gripper force ≤ 15 N. Slice thickness 3–4 mm. Surfaces sanitized.
Output:
- Assumptions to verify (tools, ingredients, lighting),
- Plan in phases (wash → prep → cut → assemble → pack),
- Per-step safety checks (vision/force),
- Error recovery (regrasp, re‑locate, re‑clean),
- Success metrics (visual confirmation of even slices; container sealed; area clean),
- Log schema and photos before/after."
What you’ll get: A long‑horizon, safety‑aware procedure with perception gates, tool handling rules, and clear success criteria.
Closing Thoughts
Great robotics prompts read like checklists from aviation: clear goals, measurable gates, and planned escapes. Use these 25 templates as building blocks, then refine them with logs from real runs. As Gemini Robotics 1.5 and ER 1.5 continue bringing agentic planning into the physical world, your prompts are the difference between a good demo and dependable daily operations.
FAQ
Q1:What is Gemini Robotics 1.5 / ER 1.5 used for?
They are embodied reasoning models that let robots perceive, plan, and act across complex, multi‑step tasks in the physical world—like picking items, preparing food, or facility operations. They emphasize grounding, safety, and adaptive planning.
Q2:How do I write prompts for multi‑step robotic tasks?
State the goal, environment, and constraints. Ask for assumptions to verify, safety checks, recovery behaviors, and success metrics. Let the model plan steps while you enforce policies and thresholds.
Q3:Can these prompts handle uncertainty and errors?
Yes. Include confidence thresholds, error signatures, and fallback branches. Designing state machines with nominal, low‑confidence, and failure paths improves reliability in unstructured settings.
Q4:Do I need to specify exact trajectories?
Usually no. Provide high‑level objectives, clear constraints (force, clearance, speed), and verification gates. The model can generate trajectories consistent with those constraints.
Q5:Where can I find official docs and examples?
See Google DeepMind’s Gemini Robotics pages and the developer overview for ER 1.5, plus the Gemini cookbook and agent guides for broader prompting and agent patterns.
