Robot training data requirements spec template
Write this before the first recording session. It turns "we need demonstrations of the task" into numbers a collection team can hit and a reviewer can check: which streams, at what rate, synchronized how tightly, in which environments, and how many episodes. Each field carries a note on why it is there, so the spec can be argued with rather than just filled in.
Who fills it: The technical lead for the dataset, with the person who will train the policy and the person who runs collection.
When to use it
- You are starting a new dataset for a manipulation or mobile-manipulation policy.
- Collection is about to move from one person's bench to a team, a vendor or a second site.
- A trained policy underperforms and you need to write down what the data was supposed to contain.
When something else fits better
- You are deciding which sensors to use at all: fill the sensing modality selection worksheet first, then bring its answers here.
- You are documenting a dataset that already exists for other people: use the dataset datasheet.
Preview
1.Task definition
Define the task as the policy will face it at deployment, not as it looks in a demo video.
A short, unique name used in file names and episode metadata.
Example: bin_pick_soft_pouch
One sentence: the start state, the action, and the end state.
Example: From a bin of 5–15 mixed soft pouches, pick one pouch and place it flat in the tote within 20 s.
An observable condition someone other than the operator can judge from the recording.
Example: Pouch fully inside tote, not folded, gripper open and clear of tote at episode end.
Robot model, end effector, degrees of freedom, and any mounted sensors.
Example: 6-DoF arm, parallel-jaw gripper with fingertip tactile pads, wrist force/torque sensor.
What the policy outputs and how often.
Example: Delta end-effector pose (6) + gripper width (1) at 10 Hz.
The sensors the policy will actually have on the robot in production.
Example: Wrist RGB, head RGB-D, joint states, fingertip tactile. No EMG, no gloves.
Situations the dataset deliberately does not cover.
Example: Torn pouches, pouches heavier than 1 kg, human in the workcell.
2.Modalities and sampling rates
List every stream that will be recorded, including the ones only used during collection.
| Stream | Sensor | Rate | Units / resolution | Frame | At deployment? |
|---|---|---|---|---|---|
| wrist_rgb | Wrist camera | Camera's native frame rate | Width × height, 8-bit RGB | camera_wrist | Yes |
| joint_state | Robot controller | Controller's state rate | rad, rad/s | base | Yes |
| ft_wrist | Wrist force/torque sensor | Fast enough to resolve contact onset | N, N·m | tool0 | Yes |
| tactile_left | Fingertip tactile pad | Fast enough to resolve slip | As the sensor reports it | finger_left | Yes / No |
| emg_forearm | Forearm EMG band | The band's native rate | µV or normalized | n/a | No |
Fill the table above: one row per stream, with sensor model, rate, units, coordinate frame, and whether it exists at deployment.
For each stream, why this rate: the fastest event it must resolve.
Example: Tactile at the sensor's native rate because slip onset is the event we label; RGB at the camera's native rate because motion blur, not frame rate, limits us.
How streams will be resampled or windowed for the policy, if differently from capture.
Example: All streams aligned to 10 Hz action steps; tactile and F/T summarized over each 100 ms window.
3.Timing and synchronization
Which clock every timestamp comes from, and how devices are synchronized to it.
Example: Host monotonic clock; cameras hardware-triggered; tactile and EMG timestamped on arrival with measured offset.
The unit and type used everywhere.
Example: Integer nanoseconds since episode start, plus absolute start time in UTC.
The maximum allowed offset between each pair of streams that are used together.
Example: A maximum offset in milliseconds for each pair: actions ↔ joint state, camera ↔ joint state, tactile and F/T ↔ joint state.
How the offset is measured in every session.
Example: Operator taps the fingertip on a fixed plate at session start and end; the tap must appear in tactile, F/T and video within tolerance.
4.Calibration
How and how often they are calibrated, and where the calibration files live.
Example: Checkerboard intrinsics per camera at install; extrinsics via hand-eye calibration weekly and after any bump.
When force and tactile sensors are zeroed.
Example: Tare with gripper open and unloaded at the start of every session and after any collision.
Events that force recalibration before collection continues.
Example: Camera moved, sensor swapped, sync check out of tolerance, collision.
5.Data source
Teleoperation, kinesthetic teaching, human demonstration with wearable sensors, scripted, autonomous rollouts, or a mix (with shares).
Example: Human demonstrations with wearable sensing for contact phases; teleoperation on the target robot for the full task.
How many people demonstrate, and what they must be able to do.
Example: Four operators, each passing a 10-episode qualification run before their data counts.
6.Environments and variation
Each place data is collected, with an ID and whether it is used for training or held out.
Example: lab_a (train), lab_b (train), customer_site_1 (held-out).
What changes between episodes, and the range of each.
Example: Lighting (3 settings), bin fill level (5–15 items), pouch pose (random), distractor objects (0–3).
The objects used, split into train and held-out.
Example: Train: 24 pouch SKUs. Held-out: 6 SKUs never shown in training.
7.Volume targets
How many episodes the pilot collects before the full target is set.
Example: Enough episodes to train one small policy end to end, across 2 environments and 2 operators.
Target episodes for each cell of the variation grid.
Example: The same target for every (environment × lighting × fill level) cell, so no cell stays empty.
Total episodes and hours, and the date it should be reached.
Whether failed attempts are kept and labelled, and how they are used.
Example: Kept, labelled with failure mode; excluded from behaviour cloning, used for success classifiers and analysis.
8.Acceptance and quality
Automatic checks every episode must pass.
Example: All required streams present; dropped frames below the spec's threshold; sync offset within tolerance; episode length within bounds.
What share of episodes a person reviews, and what they look for.
Example: A fixed random share of each operator's episodes per day: task followed, success label correct, no PII in frame.
9.Storage and versioning
Container format for raw and training-ready data.
Example: Raw: MCAP per session. Training: converted to the training framework's format per dataset version.
How sessions, episodes and dataset versions are named.
Example: {site}_{date}_{session}/ep_{0001}; dataset versions as semver with a changelog.
Who can access raw data, and how long it is kept.
10.Sign-off
One owner, the reviewers, and the date this version was approved.
Copy or download
The whole template with prompts, examples and the reason for each field.
# Robot training data requirements spec
Write this before the first recording session. It turns "we need demonstrations of the task" into numbers a collection team can hit and a reviewer can check: which streams, at what rate, synchronized how tightly, in which environments, and how many episodes. Each field carries a note on why it is there, so the spec can be argued with rather than just filled in.
Who fills it: The technical lead for the dataset, with the person who will train the policy and the person who runs collection.
How to use this file: write your answer after each "Answer:". The "Why" notes explain each field for reviewers; keep them or delete them once the document is agreed.
## 1. Task definition
Define the task as the policy will face it at deployment, not as it looks in a demo video.
### 1.1 Task name
A short, unique name used in file names and episode metadata.
Example: bin_pick_soft_pouch
Answer:
> Why: The name becomes a key in every episode record. Changing it later breaks joins between episodes, labels and evaluation logs.
### 1.2 Task statement
One sentence: the start state, the action, and the end state.
Example: From a bin of 5–15 mixed soft pouches, pick one pouch and place it flat in the tote within 20 s.
Answer:
> Why: A start and end state make episode boundaries unambiguous, which annotators and the success label both depend on.
### 1.3 Success criterion
An observable condition someone other than the operator can judge from the recording.
Example: Pouch fully inside tote, not folded, gripper open and clear of tote at episode end.
Answer:
> Why: If success is judged by feel, labels drift between labelers and between data collection and evaluation. Use the same criterion in the policy evaluation plan.
### 1.4 Embodiment
Robot model, end effector, degrees of freedom, and any mounted sensors.
Example: 6-DoF arm, parallel-jaw gripper with fingertip tactile pads, wrist force/torque sensor.
Answer:
> Why: Data collected on, or retargeted to, a different embodiment needs a mapping step. Writing the target down makes that gap visible early.
### 1.5 Action space
What the policy outputs and how often.
Example: Delta end-effector pose (6) + gripper width (1) at 10 Hz.
Answer:
> Why: The recorded actions must be convertible to this representation at this rate, or the data cannot supervise the policy directly.
### 1.6 Observations available at deployment
The sensors the policy will actually have on the robot in production.
Example: Wrist RGB, head RGB-D, joint states, fingertip tactile. No EMG, no gloves.
Answer:
> Why: Streams recorded only during collection (EMG, a glove, motion capture) can still be valuable as training targets or auxiliary signals, but the plan for using them has to be explicit.
### 1.7 Out of scope
Situations the dataset deliberately does not cover.
Example: Torn pouches, pouches heavier than 1 kg, human in the workcell.
Answer:
> Why: Saying what is excluded stops collection from drifting into edge cases and tells evaluators what not to test.
## 2. Modalities and sampling rates
List every stream that will be recorded, including the ones only used during collection.
| Stream | Sensor | Rate | Units / resolution | Frame | At deployment? |
| --- | --- | --- | --- | --- | --- |
| wrist_rgb | Wrist camera | Camera's native frame rate | Width × height, 8-bit RGB | camera_wrist | Yes |
| joint_state | Robot controller | Controller's state rate | rad, rad/s | base | Yes |
| ft_wrist | Wrist force/torque sensor | Fast enough to resolve contact onset | N, N·m | tool0 | Yes |
| tactile_left | Fingertip tactile pad | Fast enough to resolve slip | As the sensor reports it | finger_left | Yes / No |
| emg_forearm | Forearm EMG band | The band's native rate | µV or normalized | n/a | No |
### 2.1 Stream list
Fill the table above: one row per stream, with sensor model, rate, units, coordinate frame, and whether it exists at deployment.
Answer:
> Why: A stream without units or a frame is hard to use a year later. The deployment column separates policy inputs from supervision-only signals.
### 2.2 Rate rationale
For each stream, why this rate: the fastest event it must resolve.
Example: Tactile at the sensor's native rate because slip onset is the event we label; RGB at the camera's native rate because motion blur, not frame rate, limits us.
Answer:
> Why: Rates are often set to whatever the driver defaults to. Tying each to an event makes undersampling (missed contact transients) and oversampling (storage cost) both arguable.
### 2.3 Rate at training time
How streams will be resampled or windowed for the policy, if differently from capture.
Example: All streams aligned to 10 Hz action steps; tactile and F/T summarized over each 100 ms window.
Answer:
> Why: Capturing fast and training slow is common. Writing the reduction down keeps the training pipeline and the labels consistent.
## 3. Timing and synchronization
### 3.1 Clock source
Which clock every timestamp comes from, and how devices are synchronized to it.
Example: Host monotonic clock; cameras hardware-triggered; tactile and EMG timestamped on arrival with measured offset.
Answer:
> Why: Mixed clocks are the most common reason contact signals line up with the wrong video frame.
### 3.2 Timestamp unit
The unit and type used everywhere.
Example: Integer nanoseconds since episode start, plus absolute start time in UTC.
Answer:
> Why: Floats in seconds lose precision over long sessions; a single unit avoids silent conversions.
### 3.3 Synchronization tolerance
The maximum allowed offset between each pair of streams that are used together.
Example: A maximum offset in milliseconds for each pair: actions ↔ joint state, camera ↔ joint state, tactile and F/T ↔ joint state.
Answer:
> Why: Set the tolerance from the shortest event you need to align (a contact onset, a release). If streams drift more than that event lasts, labels attach to the wrong moment.
### 3.4 Sync check method
How the offset is measured in every session.
Example: Operator taps the fingertip on a fixed plate at session start and end; the tap must appear in tactile, F/T and video within tolerance.
Answer:
> Why: A tolerance nobody measures is a hope. A physical event visible in every stream gives a number per session.
## 4. Calibration
### 4.1 Camera intrinsics and extrinsics
How and how often they are calibrated, and where the calibration files live.
Example: Checkerboard intrinsics per camera at install; extrinsics via hand-eye calibration weekly and after any bump.
Answer:
> Why: Policies that use 3D information inherit every calibration error. Each episode must reference the calibration it was recorded under.
### 4.2 Force and tactile tare
When force and tactile sensors are zeroed.
Example: Tare with gripper open and unloaded at the start of every session and after any collision.
Answer:
> Why: Force/torque and many tactile sensors drift with temperature and load history; untared offsets look like contact.
### 4.3 Recalibration triggers
Events that force recalibration before collection continues.
Example: Camera moved, sensor swapped, sync check out of tolerance, collision.
Answer:
> Why: Explicit triggers stop operators from carrying on with a known-bad setup to hit a daily target.
## 5. Data source
### 5.1 Collection method
Teleoperation, kinesthetic teaching, human demonstration with wearable sensors, scripted, autonomous rollouts, or a mix (with shares).
Example: Human demonstrations with wearable sensing for contact phases; teleoperation on the target robot for the full task.
Answer:
> Why: Each method has a different embodiment gap, throughput and cost. Mixing is fine if every episode records which method produced it.
### 5.2 Operators
How many people demonstrate, and what they must be able to do.
Example: Four operators, each passing a 10-episode qualification run before their data counts.
Answer:
> Why: One operator gives consistent but narrow data; several give variety but need a qualification bar. Either way, operator ID goes in every episode.
## 6. Environments and variation
### 6.1 Environments
Each place data is collected, with an ID and whether it is used for training or held out.
Example: lab_a (train), lab_b (train), customer_site_1 (held-out).
Answer:
> Why: A held-out environment is the cheapest honest test of generalization. Decide it before collection, not after results are in.
### 6.2 Variation factors
What changes between episodes, and the range of each.
Example: Lighting (3 settings), bin fill level (5–15 items), pouch pose (random), distractor objects (0–3).
Answer:
> Why: Policies generalize along the axes the data varies on. Listing them turns 'diverse data' into a grid you can check coverage against.
### 6.3 Object set
The objects used, split into train and held-out.
Example: Train: 24 pouch SKUs. Held-out: 6 SKUs never shown in training.
Answer:
> Why: Held-out objects measure what the policy learned about the category rather than the specific items.
## 7. Volume targets
### 7.1 Pilot size
How many episodes the pilot collects before the full target is set.
Example: Enough episodes to train one small policy end to end, across 2 environments and 2 operators.
Answer:
> Why: Published dataset sizes vary by orders of magnitude between tasks. A pilot plus a small training run shows how performance moves with data for your task, which is a better basis for the target than another team's number.
### 7.2 Episodes per condition
Target episodes for each cell of the variation grid.
Example: The same target for every (environment × lighting × fill level) cell, so no cell stays empty.
Answer:
> Why: A single total hides empty cells. Per-cell targets make coverage gaps visible while collection is still running.
### 7.3 Total target
Total episodes and hours, and the date it should be reached.
Answer:
> Why: Converts the plan into operator-hours and storage, so budget and schedule can be checked.
### 7.4 Failures kept
Whether failed attempts are kept and labelled, and how they are used.
Example: Kept, labelled with failure mode; excluded from behaviour cloning, used for success classifiers and analysis.
Answer:
> Why: Failed episodes are expensive to collect and useful for success detection and recovery behaviour. Deleting them loses that.
## 8. Acceptance and quality
### 8.1 Acceptance checks
Automatic checks every episode must pass.
Example: All required streams present; dropped frames below the spec's threshold; sync offset within tolerance; episode length within bounds.
Answer:
> Why: Automated checks at ingest catch broken sessions the same day, while they can still be re-recorded.
### 8.2 Human review
What share of episodes a person reviews, and what they look for.
Example: A fixed random share of each operator's episodes per day: task followed, success label correct, no PII in frame.
Answer:
> Why: Some problems (wrong object, mislabelled success) only a person sees. A fixed sample rate keeps review from being skipped under deadline.
## 9. Storage and versioning
### 9.1 Format
Container format for raw and training-ready data.
Example: Raw: MCAP per session. Training: converted to the training framework's format per dataset version.
Answer:
> Why: Keeping raw data in a lossless, timestamped container means the training format can change without re-collecting.
### 9.2 Naming and versioning
How sessions, episodes and dataset versions are named.
Example: {site}_{date}_{session}/ep_{0001}; dataset versions as semver with a changelog.
Answer:
> Why: Models must be traceable to the exact data version they were trained on.
### 9.3 Retention and access
Who can access raw data, and how long it is kept.
Answer:
> Why: Video and biosignals of people may be personal data. Retention and access rules must match what operators consented to.
## 10. Sign-off
### 10.1 Owner and reviewers
One owner, the reviewers, and the date this version was approved.
Answer:
> Why: A spec without an owner stops being updated the first time collection deviates from it.