Visionless Soft Robotic Grasping by Touch and RL

Grasping in the dark

A camera-guided gripper can't grasp in the dark. A rigid gripper that cannot see either crushes what it holds or lets it slip.

The robot is a two-finger soft silicone gripper on a 6-DOF metal arm. It finds an unknown object by touch, rebuilds the object's shape as a 3D model, and grasps it with a reinforcement learning policy and everything runs on a Raspberry Pi 5.

How it works

The gripper closes on the object again and again, rotating the wrist between closures. Each time a fingertip force sensor crosses its contact threshold, the system records the wrist angle and how far apart the fingers are. RBR rebuilds the object from those pairs (below)

flowchart LR
    A[Rotate wrist,<br>close gripper] --> B{Touch?}
    B -- no --> A
    B -- yes --> C[Record θ, w]
    C -- next angle --> A
    C -- N angles done --> D[RBR mesh<br>and URDF]
    D --> E[RL grasp,<br>lift, hold]

The pipeline has three stages:

1. Physical probing. The arm sweeps through N wrist angles and records an (angle, width, sensor side) triplet at every contact.
2. Geometry. Radial Boundary Reconstruction (RBR) turns those triplets into a closed mesh with mass and inertia, then exports it as URDF.
3. Policy. An actor-critic network trained in the Genesis physics engine reads the reconstructed outline and picks the wrist angle for a stable force-closure grasp.

Hardware

The soft fingers

The parallel gripper and its mounts were modeled in CAD and 3D printed. Each finger is cast from silicone rubber in a 3D printed D-shaped mold. The compound is mixed with its hardener, injected by syringe to keep air out, and left to cure for 6 hours at room temperature.

The finished fingers are Shore A 30, 125 mm long, and taper from 24 mm at the base to 14 mm at the tip so that strain spreads evenly when the tendon pulls.

A flex sensor runs along each finger's spine to read its curvature. A force sensing resistor sits on the inner contact face to detect the moment of touch. Together they tell the system both that it touched something and how wide the object is at that angle.

Circuit design

Everything talks over one I2C bus at 400 kHz. The Raspberry Pi 5 is the bus master. The ADS1115 ADC and the PCA9685 motor driver share its SDA, SCL and ground lines.

Sensing. Each flex sensor and FSR forms a voltage divider with a 47 kΩ fixed resistor, powered from the Pi's clean 3.3 V logic rail. The four divider junctions feed the ADC's A0–A3 inputs. At 16 bits, the ADC resolves 65,536 steps, enough to pick up slight tactile deformations and small changes in bend.

Power isolation. The servos draw heavy current during a grasp, and their inductive spikes would brown out or corrupt the logic side. So a separate 5 V 10 A supply feeds only the motor driver's V+ rail. The Pi, the ADC and the driver's logic stay on 3.3 V with a common ground.

Radial Boundary Reconstruction

RBR turns a few dozen touches into a watertight 3D model without any training.

Each contact gives a wrist angle θ, a finger opening w and the side of the sensor that fired. The radial distance to the boundary is half the opening, plus a correction for how much the soft fingers squash:

r = w / 2 + δ_soft
left sensor  → vertex ( r·cos θ,  r·sin θ)
right sensor → vertex (−r·cos θ, −r·sin θ)

The vertices are sorted by angle into a closed outline and triangulated with ear clipping. The outline is extruded into side walls and capped top and bottom. Mass and inertia come from the closed mesh via the divergence theorem, and the result is written out as a URDF with collision and visual geometry, ready for the simulator.

The gripper itself was modeled in Fusion and its URDF checked for collision geometry before it went into simulation.

Learning to grasp

The grasp is framed as a single-step contextual bandit. The policy sees the reconstructed outline, the object's height, the wrist roll and the gripper opening. It outputs a wrist angle and an opening. The squeeze itself is fixed in the stable region, so learning focuses on alignment. The reward combines alignment, how tightly the fingers sit at the periphery, and a success bonus when the object lifts clear.

The actor-critic network was trained with PPO in RSL_RL, running thousands of episodes in parallel in the Genesis physics engine.

On the robot, the trained policy runs through ONNX Runtime on a Raspberry Pi 5 with a PREEMPT_RT kernel, inside a 100 Hz control loop.

Safety and control

• A hardware override sits outside the policy. If any tactile reading reaches 18 N, the PWM driver's power is cut immediately to protect the object and the fingers.
• Before every grasp, the system spends 2.5 seconds re-zeroing the sensor baselines with the gripper fully open, to cancel the silicone's drift.
• A Pygame interface jogs every joint degree by degree, so the physical start pose matches the simulated one exactly. An exponential cross-fade hands control between operator and policy without sudden PWM jumps.

Testing in the dark

Testing used a zero-visibility workspace with no lighting and no optical sensing of any kind.

A single grasp runs in two rounds of probing, then the grasp, a hold and the release.

Limits and next steps

• Shape. RBR assumes the object is a vertical extrusion of its outline. That works well for prisms, cylinders and convex shapes, but not for cavities or strongly irregular forms. Active Z-axis sweeps would let it build a true 3D point cloud.
• Speed. Probing takes seconds per object, which is too slow for high-speed sorting.
• Drift. Long, high-tension use makes the silicone drift, which forces regular recalibration. An unsupervised loop that tunes δ_soft as the material ages would remove those pauses.
• Friction. Silicone-on-surface friction is still hard to model in simulation, and that gap occasionally triggers false slip penalties on the real robot.
• Resolution. A continuous tactile skin in place of the discrete FSR pads would sense shear and fine slip, and allow in-hand rotation.