Robot sensing stack reference
The sensing stack decides what your robot training data can ever contain. A model can't learn force, contact or effort that was never measured. This page lists the layers, what each adds, and the ways to capture touch and force, judged on the same criteria.
The layers
| Layer | Captures | Adds information for | Limits |
|---|---|---|---|
| Vision (RGB) | What the scene looks like from a head-mounted, wrist or fixed camera. | Object identity, pose and layout; the largest pool of pretrained models. | Can't see occluded contact; shows force only indirectly. |
| Depth | Distance per pixel from a stereo, structured-light or time-of-flight sensor. | Geometry for grasp selection, clearance and placement. | Holes on transparent, reflective or thin objects; a minimum range that close-up wrist views can fall inside. |
| Proprioception | Joint positions, velocities and the commanded actions. | The action labels behaviour cloning needs. Always record it on a robot. | For human demonstrations it must be estimated (hand pose) and retargeted. |
| Force / torque | Forces and torques at the wrist, joints, fingertips, or under the workpiece. | How hard, along which axis: insertion, pressing, wiping, jams. | A wrist sensor sums every contact on the tool; it drifts and needs taring. |
| Tactile | Contact location, pressure and its changes on a fingertip, pad or glove. | Contact onset, slip and release, especially when the camera can't see the contact. | Sensors vary widely in resolution, rate and durability: check the datasheet against the event you need. |
| EMG | Electrical activity of the demonstrator's forearm muscles. | Grip effort and its timing during human demonstrations, often before motion shows it. | Only exists for human demonstrations; relates to activation, not newtons, without per-person calibration. |
Ways to capture touch and force
Each approach is described on the same criteria: what it measures, whose embodiment the data is in, where it works, and what to watch for. Specifications and prices change, so check each one's current documentation before you decide.
Preload
A head-mounted camera, a tactile sensor glove and a forearm EMG band, recorded together, capturing video, hand contact and grip effort in one stream set.
| Embodiment | Human hand; mapped to the robot's gripper for training. |
|---|---|
| Where it works | Wherever a person does the task, including real workplaces, without a robot in the loop. |
| Watch for | The embodiment gap every human-demonstration method has. For channel counts, rates and data formats, check preload.ai. |
Teleoperation with robot-mounted F/T and tactile
An operator drives the robot while its wrist F/T sensor and fingertip tactile sensors record contact.
| Embodiment | The robot's own. |
|---|---|
| Where it works | Where the robot is installed. |
| Watch for | Throughput per operator-hour; limited force feedback to the operator changes how they grasp. |
Kinesthetic teaching with joint torques
A person guides the arm by hand; joint torque sensing records the interaction.
| Embodiment | The robot's own. |
|---|---|
| Where it works | Robots that support hand-guiding. |
| Watch for | The person's own push on the arm has to be separated from contact forces. |
Instrumented objects or force plates
Sensors in the object or under the work surface measure forces on it.
| Embodiment | Independent of who does the task. |
|---|---|
| Where it works | Fixed stations with a known set of objects. |
| Watch for | Covers only the objects you instrument. |
Video only, force inferred
Force and contact estimated from motion and deformation in video.
| Embodiment | Either. |
|---|---|
| Where it works | Anywhere a camera fits. |
| Watch for | Nothing is measured, so estimates can't be checked without a measured reference. |
Next step
Fill the sensing modality selection worksheet for your task, then carry the chosen streams into the data requirements spec, with their rates and synchronization tolerance.