He is wearing a huge headgear, holding a heavy controller in both hands, and his arms are not easily flapping in the air. In the virtual game, his hand clenched the handle and turned into a Doraemon ball. His flexible fingers were useless in the magnificent and fantastic new world.
Chinese scientists from Cornell University and University of Wisconsin Madison recently jointly created a “3D Bracelet” – fingertrak.
if you are an animator, you should know how complicated it is to capture a human hand. There are countless possibilities for each bending and stretching of each joint of the hand.
fingertrak’s greatest feature is the use of a new technology to accurately predict the position of each finger – thermal sensor tracking.
they all have a low resolution infrared camera at the tip, pointing to the side of the finger. Although the resolution is low, only 32 x 24 pixels, they can accurately track the movement of the wearer’s hand.
through the depth neural network specially designed in the system, the position of 20 finger joints can be accurately predicted according to the wrist contour.
because the device no longer captures the finger position directly through the camera, but combines thermal imaging and machine learning technology to “invisibly” acquire it. Places that can’t be captured by the camera and subtle movements that can’t be detected will be “seen.”.
in the test of 11 subjects, the results showed that the recognition angle error was 6.46 ° to 8.06 ° and the more uniform the background, the lower the error rate.
the fingertrak prototype in the study is the operation of connecting raspberry PI. But the researchers say future upgrades will be able to connect to smartphones, VR glasses and even wireless connectivity.
fingertrak will be able to monitor our health like a smart watch, but what’s better than a smart watch is that it can communicate your health as long as you do it.
this is especially useful for the elderly, especially through how they use their hands in their daily life to monitor early signs of diseases such as Parkinson’s disease and Alzheimer’s disease.
in terms of human-computer interaction, you can also use fingertrak to let the robots at home imitate human actions and serve us better.
after all, if the current sign language translation technology does not require users to wear gloves or use cameras, both of which are relatively troublesome. Fingertrak will make sign language translation direct, fast and humanized.
However, fingertrak is still in its early stage of research, and it may take several years for its commercialization. After all, the actual use of any of the scenarios mentioned above is not simple.
for example, if you want to match the VR / AR application and replace the handle controller in the device, you must first consider the problem of precision and the coordination between the fingers.
after all, in reality, without a finger and a little offset, everything in our hands may fall to the ground, so if the accuracy and reality are slightly biased, the real experience in the virtual world will be disordered.
However, the researchers say that the price of the components used in the device is very affordable, and the data has been able to achieve a high degree of accuracy, which is a good thing before commercialization in the future.
besides, Xiaohe is still in its infancy and virtual reality is still in its infancy. The emergence of fingertrak has brought us new possibilities as well as its great potential in the future.
the common wearable devices in our life can monitor hand movement by using sensors such as gyroscopes and accelerometers, such as lifting the wrist, and the screen will automatically light up.
fingers are an important part of expressing human feelings. In fact, the research and development of hand tracking technology has gone a long way, and various emerging technologies appear every year.
for example, deep sensing cameras or infrared sensors scientists at Purdue University use depth sensing cameras to capture hand movements, then use deep learning networks to solve millions of gestures, and then display them accurately in the virtual world.
some people also use motion sensing Gloves – researchers at UCLA have unveiled a glove like device that converts sign language into speech dialogue and plays it on a mobile phone through a scalable sensor array assisted by machine learning.
there is even the use of electromagnetic sensors – members of oculus research department developed an electromagnetic sensor system in 2015. Through the electromagnet installed on the user’s fingertips and four magnetometers to track fingertip movement, people can stop on the way to VR games and choose to play the piano or draw with their fingertips.
but they all have the same disadvantages: either the equipment is too large and bulky, or it needs high-performance PC support and response lag, and most of them can’t be commercially available.
in recent years, with the development of AR / VR and other fields, equipment comfort and visual fidelity are upgrading. Researchers have been developing more natural solutions than handheld controllers, so hand tracking technology is also updated rapidly.
for example, Sony released the prototype of “the next generation VR game controller that can track finger movement with high precision” in May this year. This grip can determine finger movement according to electrode induction, and can already track the subtle movements of fingers. It can make fist clenching and bending fingers, which can be compared with “Ye” at the top and “666” at the bottom. You can choose from 11 positions.
in the research simulation, players can pick up the building blocks in the virtual space with their hands and build them one by one. However, the finger gesture recognition is still limited. Many people can’t compare the following gesture, and the device can’t recognize it:
for example, in February this year, researchers at the University of Washington developed an auraring called auraring The ring uses a new electromagnetic tracking system, which can carry out high-resolution optical tracking. People only need to cover a 3D printed wire coil on the index finger, plus a wrist strap full of sensors. The finger can write, tap, flick, squeeze objects in the virtual world.
light is light, small is small, but its problem is that the ring is an extension of the wrist strap’s sensing ability. If you want to operate in the virtual world, I’m afraid you have to wear a whole hand.
in VR market, among Sony, oculus and HTC as representatives, oculus also uses the new technology of “gesture tracking” on the head mounted device oculus quest. Its unique place is that it can realize hand tracking without any physical objects.
this is all based on the four external cameras on the top of oculus quest. It can not only scan the surrounding space, confirm the player’s position, but also refine and analyze the movement of each finger when you look at the finger, and extend it to the virtual world.
oculus quest has achieved an all-in-one design separated from mobile phones and computers. Combined with hand tracking technology, it can make players feel more immersed. It may also be the beginning of Facebook ar headgear, bringing real-world objects into the VR world.
However, due to the limitations of computer vision and positioning technology, oculus Quest’s gesture application will still be limited in practice.
looking back, although fingertrak’s design needs to be equipped with a “Bracelet”, the biggest advantage is that the “Bracelet” is light, compact and free to move.
Zuckerberg firmly believes that VR and AR will become the next generation of major computing platforms after mobile devices in 10 years, directly changing our work and life.
Gartner also pointed out that driven by 5g and AI, VR and AR will make great changes in user experience from now to 2028 – users’ perception of the digital world and the way users interact with the digital world.
this means that our interaction with the digital world is no longer limited to the two-dimensional screen, but to give full play to human’s complete ability in VR and AR, making digital and reality completely seamless integration.
the VR head display in the future can realize 200 degree field of view, retinal resolution, variable focal length and vivid audio, while ar glasses are “super fashionable”, which can completely replace smart phones and give the wearer the ability of perception.
this also requires the same level of technology upgrading, including optical system, interface, touch, virtual avatar, ergonomics, etc., and hand tracking is one of the most important links.