Behind-the-scenes: TrueMotion

By Colin Brown and Abel Gonzalez Garcia, Computer Vision Scientists

Starting a new physical therapy program, staying motivated, and sticking with it can be a real challenge. At Hinge Health, we know there’s a high bar for us to earn the trust of those whom we aim to treat for their musculoskeletal conditions. We're focused on building a virtual, personalized exercise experience that is clinically-validated and rigorous yet simple-to-use to deliver the best outcomes. One of our most impactful technology innovations is our TrueMotion™ computer vision technology. 

Imagine having your own personal physical therapist with you at all times providing instant exercise feedback. For people using Hinge Health, this is now a reality through our patented TrueMotion technology, built by our engineers, using the latest computer vision and AI technologies, and observing the highest standards of privacy. TrueMotion works right from the Hinge Health app on your phone for helpful and accurate guidance while you do your exercises.

Building a digital motion capture studio to read the human body 

Our journey to build TrueMotion began with the acquisition of an innovative start-up called wrnch in 2021. Leveraging technology used in visual storytelling and video game development, wrnch had spent years developing a platform for digitizing human motion used from videos using deep learning and human pose estimation - essentially, a digital version of the motion capture suits actors wear to film special effect scenes. From this foundation, Hinge Health built a system that uses computer vision to identify up to 87 reference points on the human body which are processed by AI models to recognize the exercises which our physical therapists and clinicians prescribe. The computer vision platform provides people with guidance and feedback, all through the camera on your smartphone. 

This concept video shows an AI-tracked avatar exhibiting different training movements in our virtual studio.

Training our AI models to decipher real-world movement 

People don’t come in one shape or size, so we trained our technology to recognize a diverse range of human bodies which all move in idiosyncratic ways. We also needed to consider people doing their exercise therapy in all kinds of environments - indoors, outdoors, in low light, with their dog walking by, etc. - and who may be dressed in a wide variety of clothing. 

So we designed our AI models specifically to work well with this varied subject and movement data. The AI model first takes video and image data from smartphone cameras, looking for exercise patterns. We then augment (i.e. "mess up") the training data to replicate the imperfect video that the model may receive from people using the product in the real world. We’ve built a proprietary data pipeline to quickly take in video and images and put out altered versions that we can use for training. For instance, we may rotate the image, flip it, or make it blurry. That way, for example, when someone in a low-light environment and dark clothing uses our TrueMotion technology, the AI model will be able to recognize and process that image. In addition, our TrueMotion engineering team has built a state-of-the-art, 3D virtual capture studio to generate an infinite variety of highly-accurate training images and video for training our AI models.

Putting TrueMotion in people's pockets

Now, how do our AI models come to life for people with musculoskeletal pain? The simple instructions in the Hinge Health app guide you to place your smartphone on the floor in front of you and start exercising. No external hardware, including special tablets or sensors, are required. The technology generates a 3D replica of the user’s skeleton as they exercise, detecting movements and giving real-time feedback. If they perform an exercise correctly, they will be reassured, in real time. If their form has room for improvement, they will hear and see that feedback on their phone. We aim to provide the user with encouragement and feedback so that they get the most out of their treatment. Most importantly, we prioritized security and privacy in developing TrueMotion, ensuring users' data is handled with the utmost care and in line with the highest industry standards for data protection. 

What’s Next? 

We’re dedicated to improving experiences for all Hinge Health users and expect that our deep investments in TrueMotion computer vision engineering and AI will deliver the best clinical outcomes. We’re excited about applying the latest technologies to transform healthcare and make everyone’s lives better through movement and exercise. 

We hope we’ve been able to spark your interest in Hinge Health and in our TrueMotion and other technologies. Learn more about our products and services at www.hingehealth.com and join us in our mission to help the world move beyond pain.