evaluate

Robot policy evaluation plan template

Real-robot evaluations are slow, so they tend to be small and informal, and small informal evaluations are easy to fool yourself with. Write this plan before running the policy: what question the evaluation answers, which trials, how success is judged and by whom, and how results will be reported with their uncertainty.

Who fills it: The person training the policy, reviewed by someone who didn't train it.

Other formats

When to use it

  • Comparing two checkpoints, two datasets, or a policy against a baseline on hardware.
  • Deciding whether a policy is ready for a pilot deployment.
  • Reporting results in a paper or to a customer, where others need to trust the number.

When something else fits better

  • Quick smoke tests during development. Use the trial log alone and don't report those numbers as results.

Preview

policy-evaluation-plan.md

1.Policy under test

1.1Checkpoint

Model checkpoint ID and training run.

Why this fieldA result is only reproducible if the exact weights are named.
1.2Training data version

Dataset version and any filtering applied.

Why this fieldConnects evaluation results back to data decisions, which is how you learn which data helped.
1.3Observation and action configuration

Inputs, action representation, control rate.

Why this fieldMust match the spec's deployment observations. A mismatch here explains many lab-to-site drops.

2.Questions

2.1Question this evaluation answers

One or two specific questions.

Example: Does adding fingertip tactile input improve insertion success on held-out connectors?

Why this fieldThe question decides the trials. 'How good is it?' leads to a set of trials that answers nothing in particular.
2.2Decision it informs

What you will do differently depending on the answer.

Why this fieldIf no result would change a decision, the evaluation can be smaller.

3.Trial design

3.1Conditions

The conditions tested: environments, objects, variation factors. Mark which are held out from training.

Why this fieldResults on training conditions measure fit; held-out conditions measure generalization. Report them separately.
3.2Initial conditions

How each trial's starting state is set: a fixed list, or randomized from a seed.

Why this fieldChoosing start states by hand during the evaluation tends to drift toward ones the policy handles.
3.3Trials per condition

Number of trials per condition and per policy, and the reasoning.

Why this fieldSuccess rates from a handful of trials have wide confidence intervals. Compute the interval you'd get (e.g. a Wilson interval) for the planned count before running, and check it is narrow enough to answer the question.
3.4Baselines

What the policy is compared with, run under identical conditions.

Example: Previous checkpoint; vision-only variant; scripted controller.

Why this fieldComparisons run on different days or setups confound the policy with the setup.
3.5Order and blinding

How trials of different policies are interleaved, and whether the evaluator knows which policy is running.

Why this fieldInterleaving controls for drift (lighting, wear, battery). Blinding the person judging success removes a known source of bias.

4.Success criteria

4.1Success definition

The same observable criterion as in the data requirements spec.

Why this fieldIf training labels and evaluation use different definitions, you are measuring a different task.
4.2Partial credit

Stages that count as partial success, if any.

Example: Grasped (0.33), lifted (0.66), placed (1.0).

Why this fieldPartial credit shows where a policy fails, but report full success separately so it can't hide a low completion rate.
4.3Time limit and interventions

Maximum trial time, and whether any human intervention counts as failure.

Why this fieldUnbounded retries and resets inflate success. Decide what counts before the first trial.
4.4Judge

Who decides the outcome, and how disagreements are resolved.

Why this fieldA second reviewer on recorded video for a sample of trials catches inconsistent judging.

5.Ground-truth references

5.1Reference measurements

What independent measurements are recorded during trials.

Example: External video for every trial; wrist F/T for peak force; final object pose from a fixed camera.

Why this fieldSuccess alone hides how a policy succeeds. A measured reference (force, pose, time) shows whether it is gentle, precise or lucky.
5.2Human reference

Human demonstrations of the same trials, if recorded, for comparing force profiles or timing.

Why this fieldWhere demonstrations included touch or force sensing, comparing the policy's contact profile with the human's is a direct check on whether it learned the physical part of the skill.

6.Safety

6.1Stop conditions

Force, speed or workspace limits that trigger a stop, and who can press the stop.

Why this fieldUntested policies do unexpected things. Stops protect people and hardware, and every stop is logged as an outcome.
6.2People in the workcell

Who may be present during trials, and where.

Why this fieldFollow your site's robot safety rules. The plan makes them explicit for each evaluation.

7.Hardware test log

Fill one row per trial in the trial log (download below). Never delete a row; mark invalid trials as invalid with a reason.

7.1Log location

Where the trial log and videos are stored.

Why this fieldResults tables are only as trustworthy as the log they were computed from.
7.2Invalid trials

What makes a trial invalid (e.g. setup error, hardware fault) and whether it is re-run.

Why this fieldDeciding upfront prevents quietly discarding failures as 'invalid'.

8.Analysis and reporting

8.1Metrics

Success rate with confidence interval per condition, failure-mode breakdown, time to complete.

Why this fieldAn interval shows whether two policies actually differ. A failure breakdown shows what to fix next.
8.2Comparison rule

What counts as one policy beating another.

Why this fieldSet before the results come in, so the conclusion isn't chosen after seeing the data.
8.3Sign-off

Who reviews the result and when.

Why this fieldA second reader catches the most common errors: wrong checkpoint, wrong split, mislabelled trials.

Copy or download

policy-evaluation-plan.md

The whole template with prompts, examples and the reason for each field.

# Robot policy evaluation plan

Real-robot evaluations are slow, so they tend to be small and informal, and small informal evaluations are easy to fool yourself with. Write this plan before running the policy: what question the evaluation answers, which trials, how success is judged and by whom, and how results will be reported with their uncertainty.

Who fills it: The person training the policy, reviewed by someone who didn't train it.

How to use this file: write your answer after each "Answer:". The "Why" notes explain each field for reviewers; keep them or delete them once the document is agreed.

## 1. Policy under test

### 1.1 Checkpoint

Model checkpoint ID and training run.

Answer: 

> Why: A result is only reproducible if the exact weights are named.

### 1.2 Training data version

Dataset version and any filtering applied.

Answer: 

> Why: Connects evaluation results back to data decisions, which is how you learn which data helped.

### 1.3 Observation and action configuration

Inputs, action representation, control rate.

Answer: 

> Why: Must match the spec's deployment observations. A mismatch here explains many lab-to-site drops.

## 2. Questions

### 2.1 Question this evaluation answers

One or two specific questions.

Example: Does adding fingertip tactile input improve insertion success on held-out connectors?

Answer: 

> Why: The question decides the trials. 'How good is it?' leads to a set of trials that answers nothing in particular.

### 2.2 Decision it informs

What you will do differently depending on the answer.

Answer: 

> Why: If no result would change a decision, the evaluation can be smaller.

## 3. Trial design

### 3.1 Conditions

The conditions tested: environments, objects, variation factors. Mark which are held out from training.

Answer: 

> Why: Results on training conditions measure fit; held-out conditions measure generalization. Report them separately.

### 3.2 Initial conditions

How each trial's starting state is set: a fixed list, or randomized from a seed.

Answer: 

> Why: Choosing start states by hand during the evaluation tends to drift toward ones the policy handles.

### 3.3 Trials per condition

Number of trials per condition and per policy, and the reasoning.

Answer: 

> Why: Success rates from a handful of trials have wide confidence intervals. Compute the interval you'd get (e.g. a Wilson interval) for the planned count before running, and check it is narrow enough to answer the question.

### 3.4 Baselines

What the policy is compared with, run under identical conditions.

Example: Previous checkpoint; vision-only variant; scripted controller.

Answer: 

> Why: Comparisons run on different days or setups confound the policy with the setup.

### 3.5 Order and blinding

How trials of different policies are interleaved, and whether the evaluator knows which policy is running.

Answer: 

> Why: Interleaving controls for drift (lighting, wear, battery). Blinding the person judging success removes a known source of bias.

## 4. Success criteria

### 4.1 Success definition

The same observable criterion as in the data requirements spec.

Answer: 

> Why: If training labels and evaluation use different definitions, you are measuring a different task.

### 4.2 Partial credit

Stages that count as partial success, if any.

Example: Grasped (0.33), lifted (0.66), placed (1.0).

Answer: 

> Why: Partial credit shows where a policy fails, but report full success separately so it can't hide a low completion rate.

### 4.3 Time limit and interventions

Maximum trial time, and whether any human intervention counts as failure.

Answer: 

> Why: Unbounded retries and resets inflate success. Decide what counts before the first trial.

### 4.4 Judge

Who decides the outcome, and how disagreements are resolved.

Answer: 

> Why: A second reviewer on recorded video for a sample of trials catches inconsistent judging.

## 5. Ground-truth references

### 5.1 Reference measurements

What independent measurements are recorded during trials.

Example: External video for every trial; wrist F/T for peak force; final object pose from a fixed camera.

Answer: 

> Why: Success alone hides how a policy succeeds. A measured reference (force, pose, time) shows whether it is gentle, precise or lucky.

### 5.2 Human reference

Human demonstrations of the same trials, if recorded, for comparing force profiles or timing.

Answer: 

> Why: Where demonstrations included touch or force sensing, comparing the policy's contact profile with the human's is a direct check on whether it learned the physical part of the skill.

## 6. Safety

### 6.1 Stop conditions

Force, speed or workspace limits that trigger a stop, and who can press the stop.

Answer: 

> Why: Untested policies do unexpected things. Stops protect people and hardware, and every stop is logged as an outcome.

### 6.2 People in the workcell

Who may be present during trials, and where.

Answer: 

> Why: Follow your site's robot safety rules. The plan makes them explicit for each evaluation.

## 7. Hardware test log

Fill one row per trial in the trial log (download below). Never delete a row; mark invalid trials as invalid with a reason.

### 7.1 Log location

Where the trial log and videos are stored.

Answer: 

> Why: Results tables are only as trustworthy as the log they were computed from.

### 7.2 Invalid trials

What makes a trial invalid (e.g. setup error, hardware fault) and whether it is re-run.

Answer: 

> Why: Deciding upfront prevents quietly discarding failures as 'invalid'.

## 8. Analysis and reporting

### 8.1 Metrics

Success rate with confidence interval per condition, failure-mode breakdown, time to complete.

Answer: 

> Why: An interval shows whether two policies actually differ. A failure breakdown shows what to fix next.

### 8.2 Comparison rule

What counts as one policy beating another.

Answer: 

> Why: Set before the results come in, so the conclusion isn't chosen after seeing the data.

### 8.3 Sign-off

Who reviews the result and when.

Answer: 

> Why: A second reader catches the most common errors: wrong checkpoint, wrong split, mislabelled trials.

Use it with