|

Facial Feature Enhancement for Immersive Real-Time Avatar-Based Sign Language Communication using Personalized CNNs

Kristoffer Waldow, Arnulph Fuhrmann and Daniel Roth
In:Proceedings of 31th IEEE Virtual Reality Conference (VR ’24), Orlando, USA

Abstract:

Facial recognition is crucial in sign language communication. Especially for virtual reality and avatar-based communication, increased facial features have the potential to integrate the deaf and hard-of-hearing community to improve speech comprehension and empathy. But, current methods lack precision in capturing nuanced expressions. To address this, we present a real-time solution that utilizes personalized Convolutional Neural Networks (CNNs) to capture intricate facial details, such as tongue movement and individual puffed cheeks. Our system’s classification models offer easy expansion and integration into existing facial recognition systems via UDP network broadcasting.

Preprint:

Link to preprint version

Ähnliches

Interaktive 3D Visualisierung

Inzwischen ermöglichen neue Algorithmen und hoch entwickelte GPUs das Rendering mit physikalisch basierten Modellen, wodurch die Darstellungsqualität realistisch geworden ist. Gleichzeitig kann die Geometrie mittels Echtzeitsimulation dynamisch verformt werden, so dass der Anwender sich nicht nur in der 3D Szene interaktiv bewegen, sondern diese auch sofort verändern kann.

Perceptual Comparison of Four Upscaling Algorithms for Low-Resolution Rendering for Head-mounted VR Displays

When rendering images in real-time, shading pixels is a comparatively expensive
operation. Especially for head-mounted displays, where separate images are rendered for
each eye and high frame rates need to be achieved. Upscaling algorithms are one possibility
of reducing the pixel shading costs. Four basic upscaling algorithms are implemented in a
VR rendering system, with a subsequent user study on subjective image quality. We find
that users preferred methods with a better contrast preservation.