Addressing Deaf or Hard-of-Hearing People in Avatar-Based Mixed Reality Collaboration Systems

Kristoffer Waldow and Arnulph Fuhrmann

In: Proceedings of 27th IEEE Virtual Reality Conference (VR ’20), Atlanta, USA

The abstract overview of our MASR-engine (Mixed Reality Audio Speech Recognition) embedded in our previously published MR remote collaboration system. Speech is recorded via the integrated microphone of the HMD and further processed to text, that is sent over the network to be visualized on a remote client. This enables deaf or hard-of-hearing people to participate in MR remote scenarios.

Abstract

Automatic Speech Recognition (ASR) technologies can be used to address people with auditory disabilities by integrating them in an interpersonal communication via textual visualization of speech. Especially in avatar-based Mixed Reality (MR) remote collaboration systems, speech is an important additional modality and allows natural human interaction. Therefore, we propose an easy to integrate ASR and textual visualization extension for an avatar-based MR remote collaboration system that visualizes speech via spatial floating speech bubbles. In a small pilot study, we achieved word accuracy of our extension of 97% by measuring the widely used word error rate.

Paper

Preprint

Video

Demo Video auf Youtube

Ähnliches

Localization Service Using Sparse Visual Information Based on Recent Augmented Reality Platforms

Localization Service Using Sparse Visual Information Based on Recent Augmented Reality Platforms

The ability to localize a device or user precisely within a known space, would allow many use cases on the context of location-based augmented reality. We propose a localization service based on sparse visual information using ARCore, a state-of-the-art augmented reality platform for mobile devices.

An Evaluation of Smartphone-Based Interaction in AR for Constrained Object Manipulation

An Evaluation of Smartphone-Based Interaction in AR for Constrained Object Manipulation

In Augmented Reality, interaction with the environment can be achieved with a number of different approaches. In current systems, the most common are hand and gesture inputs. However experimental applications also integrated smartphones as intuitive interaction devices and demonstrated great potential for different tasks. One particular task is constrained object manipulation, for which we conducted a user study. In it we compared standard gesture-based approaches with a touch-based interaction via smartphone. We found that a touch-based interface is significantly more efficient, although gestures are being subjectively more accepted. From these results we draw conclusions on how smartphones can be used to realize modern interfaces in AR.

SVEn – Shared Virtual Environment

SVEn – Shared Virtual Environment

This paper presents a system for a shared virtual experience which was developed within a student project. The main idea is to have two or more persons at different locations, who can interact with each other in the same virtual environment. In order to realize this idea every person is motion-captured and wears a head-mounted display (HMD). The virtual environment is rendered with the Unity game engine and the tracked positions are updated via the internet. The virtual environment developed in this project is highly immersive and users felt a strong sense of presence.