evaluate

Robot dataset datasheet template

A datasheet is what a person reads before deciding to train on your data. The structure follows the questions in "Datasheets for Datasets" (Gebru et al., Communications of the ACM, 2021), adapted for robot data: embodiment, streams, synchronization and the gap between how the data was collected and where a policy will run.

Who fills it: The dataset owner, at each release.

When to use it

  • Releasing a dataset publicly or to another team.
  • Freezing a dataset version that models will be trained and evaluated on.
  • Handing a dataset to a customer or partner who wasn't involved in collecting it.

When something else fits better

  • Collection hasn't started: write the data requirements spec. The datasheet describes what you got, not what you planned.

Preview

dataset-datasheet.md

1.Motivation

1.1Purpose

Why the dataset was created and what gap it fills.

Why this fieldTells readers the problems it was designed for, which are the ones it is most likely to work on.
1.2Creators and funding

Who created it and who funded it.

Why this fieldLets readers judge possible conflicts of interest and know whom to contact.

2.Composition

2.1What an episode is

One attempt at which task, from start to end state.

Why this fieldReaders need the unit before they can interpret any count.
2.2Counts

Episodes, hours, successes and failures, per task, environment and collection method.

Why this fieldTotals hide imbalance. A breakdown shows what the data covers well and where it is thin.
2.3Embodiment

Robot(s) and end effector(s), or human hands with which wearable sensors.

Why this fieldDetermines how directly the data can supervise a given robot.
2.4Streams

Each stream with modality, device, rate, units and frame. Link to the annotation schema.

Why this fieldLets readers check the data contains the signals their policy needs.
2.5Labels

Which labels exist (success, segments, contact events) and how they were produced.

Why this fieldAutomatic and human labels have different error profiles. Readers need to know which they are using.
2.6Splits

Recommended train, validation and held-out splits, and what is held out (environments, objects, operators).

Why this fieldShared splits make results comparable across users of the dataset.
2.7Personal data

Whether episodes contain faces, voices, biosignals or other personal data, and how they were handled.

Why this fieldUsers need to know their obligations before they download.

3.Collection process

3.1Method

Teleoperation, kinesthetic, human demonstration with wearables, scripted or autonomous, with shares.

Why this fieldThe method shapes the behaviour in the data: operator style, speed, recovery strategies.
3.2Protocol

Link the collection protocol version(s) used.

Why this fieldLets readers see exactly how sessions were run.
3.3Synchronization

Clock source and the measured sync offsets across sessions (range).

Why this fieldMeasured offsets tell users how precisely contact events line up with video and actions.
3.4Operators and consent

How many operators, how they were selected and trained, and how consent was obtained.

Why this fieldOperator variety affects behaviour diversity; consent terms limit allowed uses.
3.5Time frame and sites

When and where data was collected.

Why this fieldHardware revisions and environments change over time; dates help explain shifts.

4.Preprocessing and cleaning

4.1What was done

Filtering, resampling, cropping, anonymization, episodes removed and why.

Why this fieldEvery transformation is a decision a user may want to undo. Say whether raw data is available.

5.Uses

5.1Intended uses

The tasks and model types the dataset was built for.

Why this fieldSets expectations: where the creators expect it to work.
5.2Uses to avoid

Uses the data is unsuitable for or not permitted for.

Example: Not suitable for tasks with human contact; not licensed for identifying operators.

Why this fieldStated limits prevent misuse and save users from wasted training runs.
5.3Results so far

Policies trained on it and their evaluation results, linked to evaluation plans.

Why this fieldReal results, with their evaluation setup, are the best evidence of what the data supports.

6.Limits and known issues

6.1Coverage gaps

Conditions, objects or behaviours the data doesn't cover.

Why this fieldEvery dataset has gaps. Listing them is more useful than an impression of completeness.
6.2Known errors

Known labeling errors, sync problems or sensor faults, and which episodes they affect.

Why this fieldLets users filter affected episodes rather than discover the problem in their results.
6.3Embodiment gap

Differences between the embodiment in the data and target robots.

Why this fieldThe most common reason robot data underperforms on a new platform.

7.Distribution and maintenance

7.1License and access

License, access conditions and where to download.

Why this fieldUsers must know what they may do with it before they invest in it.
7.2Versioning and changelog

Current version and what changed from the previous one.

Why this fieldResults must cite a version; a changelog explains why numbers moved.
7.3Contact and errata

How to report problems and where corrections are published.

Why this fieldA maintained dataset improves; an unmaintained one accumulates known issues nobody hears about.

Copy or download

dataset-datasheet.md

The whole template with prompts, examples and the reason for each field.

# Robot dataset datasheet

A datasheet is what a person reads before deciding to train on your data. The structure follows the questions in "Datasheets for Datasets" (Gebru et al., Communications of the ACM, 2021), adapted for robot data: embodiment, streams, synchronization and the gap between how the data was collected and where a policy will run.

Who fills it: The dataset owner, at each release.

How to use this file: write your answer after each "Answer:". The "Why" notes explain each field for reviewers; keep them or delete them once the document is agreed.

## 1. Motivation

### 1.1 Purpose

Why the dataset was created and what gap it fills.

Answer: 

> Why: Tells readers the problems it was designed for, which are the ones it is most likely to work on.

### 1.2 Creators and funding

Who created it and who funded it.

Answer: 

> Why: Lets readers judge possible conflicts of interest and know whom to contact.

## 2. Composition

### 2.1 What an episode is

One attempt at which task, from start to end state.

Answer: 

> Why: Readers need the unit before they can interpret any count.

### 2.2 Counts

Episodes, hours, successes and failures, per task, environment and collection method.

Answer: 

> Why: Totals hide imbalance. A breakdown shows what the data covers well and where it is thin.

### 2.3 Embodiment

Robot(s) and end effector(s), or human hands with which wearable sensors.

Answer: 

> Why: Determines how directly the data can supervise a given robot.

### 2.4 Streams

Each stream with modality, device, rate, units and frame. Link to the annotation schema.

Answer: 

> Why: Lets readers check the data contains the signals their policy needs.

### 2.5 Labels

Which labels exist (success, segments, contact events) and how they were produced.

Answer: 

> Why: Automatic and human labels have different error profiles. Readers need to know which they are using.

### 2.6 Splits

Recommended train, validation and held-out splits, and what is held out (environments, objects, operators).

Answer: 

> Why: Shared splits make results comparable across users of the dataset.

### 2.7 Personal data

Whether episodes contain faces, voices, biosignals or other personal data, and how they were handled.

Answer: 

> Why: Users need to know their obligations before they download.

## 3. Collection process

### 3.1 Method

Teleoperation, kinesthetic, human demonstration with wearables, scripted or autonomous, with shares.

Answer: 

> Why: The method shapes the behaviour in the data: operator style, speed, recovery strategies.

### 3.2 Protocol

Link the collection protocol version(s) used.

Answer: 

> Why: Lets readers see exactly how sessions were run.

### 3.3 Synchronization

Clock source and the measured sync offsets across sessions (range).

Answer: 

> Why: Measured offsets tell users how precisely contact events line up with video and actions.

### 3.4 Operators and consent

How many operators, how they were selected and trained, and how consent was obtained.

Answer: 

> Why: Operator variety affects behaviour diversity; consent terms limit allowed uses.

### 3.5 Time frame and sites

When and where data was collected.

Answer: 

> Why: Hardware revisions and environments change over time; dates help explain shifts.

## 4. Preprocessing and cleaning

### 4.1 What was done

Filtering, resampling, cropping, anonymization, episodes removed and why.

Answer: 

> Why: Every transformation is a decision a user may want to undo. Say whether raw data is available.

## 5. Uses

### 5.1 Intended uses

The tasks and model types the dataset was built for.

Answer: 

> Why: Sets expectations: where the creators expect it to work.

### 5.2 Uses to avoid

Uses the data is unsuitable for or not permitted for.

Example: Not suitable for tasks with human contact; not licensed for identifying operators.

Answer: 

> Why: Stated limits prevent misuse and save users from wasted training runs.

### 5.3 Results so far

Policies trained on it and their evaluation results, linked to evaluation plans.

Answer: 

> Why: Real results, with their evaluation setup, are the best evidence of what the data supports.

## 6. Limits and known issues

### 6.1 Coverage gaps

Conditions, objects or behaviours the data doesn't cover.

Answer: 

> Why: Every dataset has gaps. Listing them is more useful than an impression of completeness.

### 6.2 Known errors

Known labeling errors, sync problems or sensor faults, and which episodes they affect.

Answer: 

> Why: Lets users filter affected episodes rather than discover the problem in their results.

### 6.3 Embodiment gap

Differences between the embodiment in the data and target robots.

Answer: 

> Why: The most common reason robot data underperforms on a new platform.

## 7. Distribution and maintenance

### 7.1 License and access

License, access conditions and where to download.

Answer: 

> Why: Users must know what they may do with it before they invest in it.

### 7.2 Versioning and changelog

Current version and what changed from the previous one.

Answer: 

> Why: Results must cite a version; a changelog explains why numbers moved.

### 7.3 Contact and errata

How to report problems and where corrections are published.

Answer: 

> Why: A maintained dataset improves; an unmaintained one accumulates known issues nobody hears about.

Use it with