Improving future VR avatars with data from TikTok videos

Yasamin Jafarian, a doctoral student in computer science and engineering, has over the past year harnessed videos for the frame-by-frame building blocks she uses to build realistic 3D avatars of real people.

Finding many of today’s 3D avatars cartoonish, he wants to replace them using machine learning and artificial intelligence (AI), to generate more realistic avatars for use in future virtual reality environments. To that end, it trains artificial intelligence computers to understand visual data through images and videos.

Training algorithms with TikTok videos

The movie industry produces realistic avatars for movies or video games through CGI (computer generated imagery). But the industry can afford to take thousands of photos of artists.

“The problem with film technology is that it is not accessible to everyone”Jafarian commented, referring to the related technical aspects. “I wanted to create the same opportunity for the average person to be able to use their phone camera to create a 3D avatar of themselves.“, he also said.

Jafarian intended to design an algorithm that needed only a photo or video of a person to generate a realistic avatar. That required a large dataset of videos to “to train” the algorithm. TikTok dance videos, which often feature a single person showing the full length of her body in multiple poses, filled the bill.

After viewing some 1,000 TikTok videos, Jafarian chose 340 for his data set, each 10 to 15 seconds long. At 30 frames per second, they reached more than 100,000 images of people dancing.

So far, he has successfully used his algorithm to generate a 3D avatar of a person from the front view. Jafarian plans to keep refining the algorithm until he can generate a complete person’s body using just a few views. She hopes that real people will one day use the technology to interact in virtual social spaces online, and not just through Zoom.

“We can have virtual environments, using virtual reality glasses like Oculus, for example, where we can see each other and interact with each other”says the researcher. “If we can make those digital avatars realistic, those interactions would be deeper and more interesting.”he added.

The improvement of the technology behind the avatars can have a transversal scope, from augmented reality experiences that allow us to try on clothes without leaving home, to even more complex virtual reality experiences.