An ultrasound wristband built at MIT tracks all five fingers and the palm in real time, and a wearer has used it to make a robotic hand play a simple tune on a piano. The device images the muscles, tendons and ligaments of the wrist and passes the resulting hand positions to a robot or a virtual environment wirelessly. The team's results appear in a paper in Nature Electronics.
The work is led by Xuanhe Zhao, the Uncas and Helen Whitaker Professor of Mechanical Engineering at MIT, and Gengxi Lu. It is a form of ultrasound wristband hand tracking that does not rely on cameras or sensor gloves.
How the wristband reads the hand
The wristband carries an ultrasound sticker the size of a smartwatch, along with onboard electronics about as small as a cellphone. Zhao's group has been developing ultrasound stickers for some time. They are miniaturised versions of the transducers used in doctors' offices, paired with a hydrogel that sticks safely to skin.
The principle rests on the anatomy of the wrist. "The tendons and muscles in your wrist are like strings pulling on puppets, which are your fingers," Lu says. Imaging the state of those strings indicates the state of the hand.
The fingers and thumb have 22 degrees of freedom, meaning distinct ways of extending or angling. The researchers found regions in the black-and-white ultrasound images that correlate with each one. Changes in one region relate to thumb extension, for example, while changes in another track movement of the index finger.
Training the software
To build those links, a volunteer wore the wristband and moved the hand through many positions while multiple cameras recorded the gestures. Matching changes in image regions to the camera-recorded hand positions let the team label each region with its degree of freedom.
Doing that translation continuously and in real time is beyond what people can do by hand, so the team trained an AI algorithm on the labelled images. They tested it on a new set of ultrasound images, and it correctly predicted the corresponding hand gestures.
The team then tested the device on eight volunteers with different hand and wrist sizes. The volunteers formed gestures and grasps, including the signs for all 26 letters of American Sign Language. They also held a tennis ball, a plastic bottle, a pair of scissors and a pencil. In each case the wristband tracked and predicted the position of the hand.
Demonstrations with a robot hand and a screen
In the robot tests, the wristband acted as a wireless controller for a simple commercial robotic hand. A volunteer went through the motions of playing a keyboard, and the robot copied them in real time to play a tune on a piano. The same robot mimicked finger taps to play a desktop basketball game.
The team also paired the wristband with a simple computer program. Pinching and grasping gestures zoomed in and out on an object on screen and moved it in a smooth, continuous fashion.
Zhao plans to shrink the hardware further and train the software on more gestures from volunteers with a wider range of hand sizes and shapes. The group is already collecting hand motion data from more users. It envisions a large dataset that could be used to train humanoid robots in dexterity tasks such as certain surgical procedures. The stated goal is a wearable tracker anyone can use to control humanoid robots or virtual objects.
What this means for the field
The comparison with existing methods is the clearest reason to take the work seriously. Camera systems are impractical to set up for each application and are prone to visual obstruction. Sensor gloves can limit natural hand motion and sensation. Electrical signals from wrist or forearm muscles are affected by environmental noise and can tell whether a thumb and index finger are pinched or apart, but not much of the path in between. Ultrasound sees the tissue that moves the fingers, and the demonstrations showed continuous output rather than a few discrete states. A wrist-worn band also leaves the fingers free to touch real objects, which the object-holding tests suggest it can tolerate.
The data argument may matter as much as the control argument. Teaching humanoid robots dexterity requires large volumes of human hand motion. A band that can be worn during ordinary tasks could gather that data without cameras or gloves. That is a plausible route, though the team describes it as an aim, not something it has done.
Several points remain unproven. The tests involved eight volunteers, while the team itself says it needs data from many more hand sizes and shapes. The robot used was a simple commercial hand, and the tasks were a short piano tune and a desktop basketball game. Those show the tracking works as a controller. They do not show it can drive a humanoid through precision work such as surgery. The source gives no figures for tracking error or latency, so the claim of precision cannot be compared numerically with gloves or electrical sensing. The hardware also still needs to shrink. Zhao calls it the most advanced way to track dexterous hand motion, but that rests on his group's own results, and independent testing at larger scale would settle it.