Robot sensing stack reference

The sensing stack decides what your robot training data can ever contain. A model can't learn force, contact or effort that was never measured. This page lists the layers, what each adds, and the ways to capture touch and force, judged on the same criteria.

The layers

LayerCapturesAdds information forLimits
Vision (RGB)What the scene looks like from a head-mounted, wrist or fixed camera.Object identity, pose and layout; the largest pool of pretrained models.Can't see occluded contact; shows force only indirectly.
DepthDistance per pixel from a stereo, structured-light or time-of-flight sensor.Geometry for grasp selection, clearance and placement.Holes on transparent, reflective or thin objects; a minimum range that close-up wrist views can fall inside.
ProprioceptionJoint positions, velocities and the commanded actions.The action labels behaviour cloning needs. Always record it on a robot.For human demonstrations it must be estimated (hand pose) and retargeted.
Force / torqueForces and torques at the wrist, joints, fingertips, or under the workpiece.How hard, along which axis: insertion, pressing, wiping, jams.A wrist sensor sums every contact on the tool; it drifts and needs taring.
TactileContact location, pressure and its changes on a fingertip, pad or glove.Contact onset, slip and release, especially when the camera can't see the contact.Sensors vary widely in resolution, rate and durability: check the datasheet against the event you need.
EMGElectrical activity of the demonstrator's forearm muscles.Grip effort and its timing during human demonstrations, often before motion shows it.Only exists for human demonstrations; relates to activation, not newtons, without per-person calibration.

Ways to capture touch and force

Each approach is described on the same criteria: what it measures, whose embodiment the data is in, where it works, and what to watch for. Specifications and prices change, so check each one's current documentation before you decide.

Preload

A head-mounted camera, a tactile sensor glove and a forearm EMG band, recorded together, capturing video, hand contact and grip effort in one stream set.

EmbodimentHuman hand; mapped to the robot's gripper for training.
Where it worksWherever a person does the task, including real workplaces, without a robot in the loop.
Watch forThe embodiment gap every human-demonstration method has. For channel counts, rates and data formats, check preload.ai.

Teleoperation with robot-mounted F/T and tactile

An operator drives the robot while its wrist F/T sensor and fingertip tactile sensors record contact.

EmbodimentThe robot's own.
Where it worksWhere the robot is installed.
Watch forThroughput per operator-hour; limited force feedback to the operator changes how they grasp.

Kinesthetic teaching with joint torques

A person guides the arm by hand; joint torque sensing records the interaction.

EmbodimentThe robot's own.
Where it worksRobots that support hand-guiding.
Watch forThe person's own push on the arm has to be separated from contact forces.

Instrumented objects or force plates

Sensors in the object or under the work surface measure forces on it.

EmbodimentIndependent of who does the task.
Where it worksFixed stations with a known set of objects.
Watch forCovers only the objects you instrument.

Video only, force inferred

Force and contact estimated from motion and deformation in video.

EmbodimentEither.
Where it worksAnywhere a camera fits.
Watch forNothing is measured, so estimates can't be checked without a measured reference.

Next step

Fill the sensing modality selection worksheet for your task, then carry the chosen streams into the data requirements spec, with their rates and synchronization tolerance.