specifycollect

Sensing modality selection template

Every extra sensor costs synchronization work, calibration, storage and operator time. This worksheet makes you say, for one task, what each modality would tell the policy that the others don't, whether the robot will have it at deployment, and what it costs to capture. Score each modality on the same criteria, then record the decision and the reason.

Who fills it: The person specifying the dataset, ideally with someone who has run the task by hand.

Other formats

When to use it

  • Before writing the data requirements spec, when the stream list is still open.
  • A vision-only policy fails at contact: slips, crushes, misses insertion, or can't tell when a grasp is secure.
  • You are choosing between collection approaches that capture touch and force in different ways.

When something else fits better

  • The sensors are fixed by hardware you can't change: go straight to the data requirements spec.
  • The task is free-space motion with no meaningful contact (e.g. pointing, gross navigation): video and proprioception are usually enough.

Preview

modality-selection-checklist.md

1.Task profile

Answer these first: the scores below depend on them.

1.1Task

The task in one sentence, with start and end state.

Example: Insert a flexible connector into its socket until it clicks, from a loose grasp on the cable.

Why this fieldModality value is task-specific. A worksheet filled for 'manipulation in general' always concludes that everything helps.
1.2Contact phases

List the phases where the hand or gripper touches something, and what must be controlled in each.

Example: Grasp (grip force), alignment (lateral force), insertion (axial force until click), release.

Why this fieldTouch and force sensing earn their cost in contact phases. A task with one brief grasp needs them less than one with sustained, regulated contact.
1.3Current failure mode

If a policy or baseline exists, how it fails. Otherwise, how a novice human fails.

Example: Pushes too hard and bends pins; or stops before the click.

Why this fieldFailures caused by information the camera can't see (force, slip, hidden contact) point to touch or force sensing. Failures of perception or planning usually don't.
1.4Occlusion at contact

At the moment of contact, can a camera see it?

Example: No: the fingers cover the connector during insertion.

Why this fieldWhen the contact is occluded, video can only infer it from context; a contact sensor measures it.
1.5Sensors at deployment

What the production robot will have.

Example: Wrist camera, joint torques, wrist F/T. No fingertip tactile yet.

Why this fieldA modality the robot won't have can still be recorded for supervision or analysis, but it can't be a policy input. Mark the difference now.

2.RGB video

Adds information when

  • Object identity, pose and scene layout matter.
  • Large datasets or pretrained visual models are part of the plan.
  • Human demonstrations will be retargeted from what the person did, seen from their viewpoint.

Adds little when

  • The key event is hidden by the hand or gripper.
  • Success depends on force (how hard, how long), which video shows only indirectly.
2.1Viewpoints

Which cameras: head or egocentric, wrist, fixed external. Why each.

Why this fieldEgocentric and wrist views see the contact region best; fixed views give stable context for scene state and success labels.
2.2Score

Information for this task (0–2), at deployment (Y/N), sync burden (0–2), cost notes.

Why this fieldSame criteria for every modality, so the comparison is fair.

3.Depth

Adds information when

  • Geometry drives the action: grasp points on unknown objects, clearance, stacking.
  • Lighting varies a lot between environments.

Adds little when

  • Objects are transparent, reflective or very thin, where many depth sensors return holes.
  • The camera is very close to the contact, inside the sensor's minimum range.
3.1Range and surfaces

Working distance and the surfaces in the scene. Check both against the sensor's datasheet.

Why this fieldDepth quality depends on range and material. The datasheet's minimum range and the scene's worst surface decide whether it helps.
3.2Score

Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.

Why this fieldSame criteria for every modality.

4.Tactile

Adds information when

  • Contact location, onset, slip or release must be detected precisely.
  • Contact is occluded from every camera.
  • Objects are deformable or fragile, so grasp quality isn't visible.

Adds little when

  • The robot will never have tactile sensing and there is no plan to use it as a training target.
  • Contact is brief and rigid, and the gripper's own position feedback already tells you it closed on something.
4.1Where the sensing sits

On the robot's fingers, on a glove worn by a human demonstrator, or on the object.

Why this fieldRobot-mounted tactile gives data in the robot's own embodiment; a glove captures human touch where the skill is, and needs a mapping to the gripper; object-mounted sensing measures the object's side of the contact.
4.2Spatial and temporal resolution needed

How finely you need to locate contact and how quickly you need to detect it.

Why this fieldSlip detection needs fast sampling; contact localization needs taxel density. Check both against candidate sensors' datasheets.
4.3Score

Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.

Why this fieldSame criteria for every modality.

5.Force / torque

Adds information when

  • The task is defined by force: insertion, wiping, pressing, tightening, peeling.
  • Collisions and jams must be detected and labelled.

Adds little when

  • Contact forces are tiny compared with the sensor's noise floor.
  • Joint torque estimates on the robot already give the signal at the precision you need.
5.1Measurement point

Wrist six-axis sensor, joint torques, fingertip force, or a force plate under the workpiece.

Why this fieldA wrist sensor sums every contact on the tool; fingertip or object-side sensing separates them. Pick the point where the force you care about is isolated.
5.2Score

Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.

Why this fieldSame criteria for every modality.

6.EMG

Adds information when

  • You record human demonstrations and want a signal of grip effort and intent from the demonstrator's muscles.
  • Effort changes before motion does, e.g. bracing before a pull, and you want to label or predict that.

Adds little when

  • Data comes from teleoperation or the robot itself: there is no human muscle to measure.
  • You need absolute force in newtons: surface EMG relates to muscle activation, and mapping it to force needs per-person calibration.
6.1What it will be used for

Policy input (rare, since robots have no muscles), training target, auxiliary signal, or labeling aid.

Example: Auxiliary signal for grip-effort phases; helps label grasp tightening and release.

Why this fieldEMG is supervision-only for almost every robot policy. Writing its role down keeps it from being collected 'just in case'.
6.2Consent and handling

How biosignal data is covered by operator consent and access rules.

Why this fieldMuscle signals are recorded from a person's body. Treat them under the same consent and retention rules as video of the person.
6.3Score

Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.

Why this fieldSame criteria for every modality.

7.Ways to capture touch and force

If tactile, force or EMG scored well above, choose how to capture them. These are common approaches, scored on the same criteria. Specs and prices change: check each one's current documentation.

ApproachWhat it measuresEmbodimentWhere it worksWatch for
Preload: head-mounted camera, tactile sensor glove and forearm EMG band, synchronizedEgocentric video, contact and pressure at the hand, muscle activity behind grip effortHuman hand; needs mapping to the robot gripperWherever a person does the task, including real workplaces, without instrumenting a robotEmbodiment gap shared by all human-demonstration data; check preload.ai for channel counts, rates and formats
Teleoperation with wrist F/T and fingertip tactile on the robotForce and contact as the robot experiences themRobot's ownWhere the robot is installedOperator throughput; limited haptic feedback to the operator can change how they grasp
Kinesthetic teaching with joint torque sensingJoint torques and robot state while a person guides the armRobot's ownRobots that support hand-guidingThe person's hand on the arm adds forces to separate from contact forces
Instrumented objects or force platesForces on the object or work surfaceIndependent of who does the taskFixed workstations with known objectsOnly covers objects you instrument
Video only, force inferredNothing directly; force estimated from motion and deformationEitherAnywhere a camera fitsEstimates can't be checked without a measured reference
7.1Approaches considered

Which of the approaches above (or others) you considered, with a 0–2 score on each criterion.

Why this fieldWriting down the alternatives makes the choice reviewable and easy to revisit when the robot's sensors change.

8.Decision record

8.1Modalities chosen

Each modality, marked as policy input, supervision-only, or skipped.

Example: Wrist RGB + joint state: input. Fingertip tactile: input. EMG: supervision-only. Depth: skipped (thin parts).

Why this fieldThis list becomes the stream table in the data requirements spec.
8.2Rejected, and why

Each modality you skipped and the reason.

Why this fieldNext year someone will ask why there's no depth. The reason saves re-running the analysis.
8.3Revisit trigger

The result that would change this decision.

Example: If insertion success stays below target after the pilot, add wrist F/T.

Why this fieldA decision with a trigger is a hypothesis you can test, not a permanent constraint.

Copy or download

modality-selection-checklist.md

The whole template with prompts, examples and the reason for each field.

# Sensing modality selection

Every extra sensor costs synchronization work, calibration, storage and operator time. This worksheet makes you say, for one task, what each modality would tell the policy that the others don't, whether the robot will have it at deployment, and what it costs to capture. Score each modality on the same criteria, then record the decision and the reason.

Who fills it: The person specifying the dataset, ideally with someone who has run the task by hand.

How to use this file: write your answer after each "Answer:". The "Why" notes explain each field for reviewers; keep them or delete them once the document is agreed.

## 1. Task profile

Answer these first: the scores below depend on them.

### 1.1 Task

The task in one sentence, with start and end state.

Example: Insert a flexible connector into its socket until it clicks, from a loose grasp on the cable.

Answer: 

> Why: Modality value is task-specific. A worksheet filled for 'manipulation in general' always concludes that everything helps.

### 1.2 Contact phases

List the phases where the hand or gripper touches something, and what must be controlled in each.

Example: Grasp (grip force), alignment (lateral force), insertion (axial force until click), release.

Answer: 

> Why: Touch and force sensing earn their cost in contact phases. A task with one brief grasp needs them less than one with sustained, regulated contact.

### 1.3 Current failure mode

If a policy or baseline exists, how it fails. Otherwise, how a novice human fails.

Example: Pushes too hard and bends pins; or stops before the click.

Answer: 

> Why: Failures caused by information the camera can't see (force, slip, hidden contact) point to touch or force sensing. Failures of perception or planning usually don't.

### 1.4 Occlusion at contact

At the moment of contact, can a camera see it?

Example: No: the fingers cover the connector during insertion.

Answer: 

> Why: When the contact is occluded, video can only infer it from context; a contact sensor measures it.

### 1.5 Sensors at deployment

What the production robot will have.

Example: Wrist camera, joint torques, wrist F/T. No fingertip tactile yet.

Answer: 

> Why: A modality the robot won't have can still be recorded for supervision or analysis, but it can't be a policy input. Mark the difference now.

## 2. RGB video

Adds information when:
- Object identity, pose and scene layout matter.
- Large datasets or pretrained visual models are part of the plan.
- Human demonstrations will be retargeted from what the person did, seen from their viewpoint.

Adds little when:
- The key event is hidden by the hand or gripper.
- Success depends on force (how hard, how long), which video shows only indirectly.

### 2.1 Viewpoints

Which cameras: head or egocentric, wrist, fixed external. Why each.

Answer: 

> Why: Egocentric and wrist views see the contact region best; fixed views give stable context for scene state and success labels.

### 2.2 Score

Information for this task (0–2), at deployment (Y/N), sync burden (0–2), cost notes.

Answer: 

> Why: Same criteria for every modality, so the comparison is fair.

## 3. Depth

Adds information when:
- Geometry drives the action: grasp points on unknown objects, clearance, stacking.
- Lighting varies a lot between environments.

Adds little when:
- Objects are transparent, reflective or very thin, where many depth sensors return holes.
- The camera is very close to the contact, inside the sensor's minimum range.

### 3.1 Range and surfaces

Working distance and the surfaces in the scene. Check both against the sensor's datasheet.

Answer: 

> Why: Depth quality depends on range and material. The datasheet's minimum range and the scene's worst surface decide whether it helps.

### 3.2 Score

Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.

Answer: 

> Why: Same criteria for every modality.

## 4. Tactile

Adds information when:
- Contact location, onset, slip or release must be detected precisely.
- Contact is occluded from every camera.
- Objects are deformable or fragile, so grasp quality isn't visible.

Adds little when:
- The robot will never have tactile sensing and there is no plan to use it as a training target.
- Contact is brief and rigid, and the gripper's own position feedback already tells you it closed on something.

### 4.1 Where the sensing sits

On the robot's fingers, on a glove worn by a human demonstrator, or on the object.

Answer: 

> Why: Robot-mounted tactile gives data in the robot's own embodiment; a glove captures human touch where the skill is, and needs a mapping to the gripper; object-mounted sensing measures the object's side of the contact.

### 4.2 Spatial and temporal resolution needed

How finely you need to locate contact and how quickly you need to detect it.

Answer: 

> Why: Slip detection needs fast sampling; contact localization needs taxel density. Check both against candidate sensors' datasheets.

### 4.3 Score

Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.

Answer: 

> Why: Same criteria for every modality.

## 5. Force / torque

Adds information when:
- The task is defined by force: insertion, wiping, pressing, tightening, peeling.
- Collisions and jams must be detected and labelled.

Adds little when:
- Contact forces are tiny compared with the sensor's noise floor.
- Joint torque estimates on the robot already give the signal at the precision you need.

### 5.1 Measurement point

Wrist six-axis sensor, joint torques, fingertip force, or a force plate under the workpiece.

Answer: 

> Why: A wrist sensor sums every contact on the tool; fingertip or object-side sensing separates them. Pick the point where the force you care about is isolated.

### 5.2 Score

Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.

Answer: 

> Why: Same criteria for every modality.

## 6. EMG

Adds information when:
- You record human demonstrations and want a signal of grip effort and intent from the demonstrator's muscles.
- Effort changes before motion does, e.g. bracing before a pull, and you want to label or predict that.

Adds little when:
- Data comes from teleoperation or the robot itself: there is no human muscle to measure.
- You need absolute force in newtons: surface EMG relates to muscle activation, and mapping it to force needs per-person calibration.

### 6.1 What it will be used for

Policy input (rare, since robots have no muscles), training target, auxiliary signal, or labeling aid.

Example: Auxiliary signal for grip-effort phases; helps label grasp tightening and release.

Answer: 

> Why: EMG is supervision-only for almost every robot policy. Writing its role down keeps it from being collected 'just in case'.

### 6.2 Consent and handling

How biosignal data is covered by operator consent and access rules.

Answer: 

> Why: Muscle signals are recorded from a person's body. Treat them under the same consent and retention rules as video of the person.

### 6.3 Score

Information (0–2), at deployment (Y/N), sync burden (0–2), cost notes.

Answer: 

> Why: Same criteria for every modality.

## 7. Ways to capture touch and force

If tactile, force or EMG scored well above, choose how to capture them. These are common approaches, scored on the same criteria. Specs and prices change: check each one's current documentation.

| Approach | What it measures | Embodiment | Where it works | Watch for |
| --- | --- | --- | --- | --- |
| Preload: head-mounted camera, tactile sensor glove and forearm EMG band, synchronized | Egocentric video, contact and pressure at the hand, muscle activity behind grip effort | Human hand; needs mapping to the robot gripper | Wherever a person does the task, including real workplaces, without instrumenting a robot | Embodiment gap shared by all human-demonstration data; check preload.ai for channel counts, rates and formats |
| Teleoperation with wrist F/T and fingertip tactile on the robot | Force and contact as the robot experiences them | Robot's own | Where the robot is installed | Operator throughput; limited haptic feedback to the operator can change how they grasp |
| Kinesthetic teaching with joint torque sensing | Joint torques and robot state while a person guides the arm | Robot's own | Robots that support hand-guiding | The person's hand on the arm adds forces to separate from contact forces |
| Instrumented objects or force plates | Forces on the object or work surface | Independent of who does the task | Fixed workstations with known objects | Only covers objects you instrument |
| Video only, force inferred | Nothing directly; force estimated from motion and deformation | Either | Anywhere a camera fits | Estimates can't be checked without a measured reference |

### 7.1 Approaches considered

Which of the approaches above (or others) you considered, with a 0–2 score on each criterion.

Answer: 

> Why: Writing down the alternatives makes the choice reviewable and easy to revisit when the robot's sensors change.

## 8. Decision record

### 8.1 Modalities chosen

Each modality, marked as policy input, supervision-only, or skipped.

Example: Wrist RGB + joint state: input. Fingertip tactile: input. EMG: supervision-only. Depth: skipped (thin parts).

Answer: 

> Why: This list becomes the stream table in the data requirements spec.

### 8.2 Rejected, and why

Each modality you skipped and the reason.

Answer: 

> Why: Next year someone will ask why there's no depth. The reason saves re-running the analysis.

### 8.3 Revisit trigger

The result that would change this decision.

Example: If insertion success stays below target after the pilot, add wrist F/T.

Answer: 

> Why: A decision with a trigger is a hypothesis you can test, not a permanent constraint.

Use it with