Facebook Researchers Develop Invisible Finger-Tracking Keyboard
Using AI training and statistical word prediction, researchers at Facebook Reality Labs (FRL) are developing an invisible finger tracking keyboard that could be used in combination with Virtual Reality headsets like Oculus Quest.
One of the most basic dilemmas surrounding AR/VR tech has been how to make text input in VR a fast, comfortable and also a familiar and intuitive experience. During the scientific conference “User Interface Software Technology” held last week, Facebook Reality Labs unveiled a new research into hand tracking which aims to introduce touch typing to Augmented Reality and Virtual Reality users without the use of a physical keyboard.
Oculus Quest already has basic hand tracking that enables users to navigate the system User Interface, browse the web and also play any supported games such as Waltz of the Wizard: Extended Edition (2019) that does not rely on the Touch controllers but enables users to use their hands to manipulate objects or cast spells.
While these hand tracking use-cases might be useful and interesting, they are still in their infancy as far as their potential implementation in VR goes. There is a still a huge gap to be filled when it comes to the provision of natural VR input methods. Facebook researchers have been working on a hand tracking system that could be put into even more useful use-cases and their latest development is aimed at solving the problem of text input in AR/VR headsets.
The Facebook researchers are using an AI-based method which predicts the output of the letters based on the finger movements. The Artificial Intelligence model was trained with data consisting of motion tracking recordings from 20 typists who typed their movement data on an actual physical model using special gloves with a ten-finger system.
To achieve automatic word corrections, the researchers also used statistical techniques from speech and word correction such as those used in smartphones. This is how the system is able to catch typing errors that may emanate from the hands moving slightly on the table surface.
The Facebook system relies on a motion model to predict what people are intending to type irrespective of the erratic motion which might occur when a user types on a flat surface.
Facebook said its tech is capable of isolating individual fingers as well as their trajectories as these fingers reach for the keys. This kind of tech doesn’t even exist for existing touch pad services such as tablets and smartphones.
FRL stated that their new approach leverages hand motion from a marker-based hand tracking system as input and the decodes the motion directly into the text that the user is intending to type. The system is still in the research phase but the research demonstrates the potential for hand tracking applications in productivity scenarios such as typing faster on any surface.
One of the biggest challenges to be overcome with this kind of technology is the “erratic” typing patterns displayed by users. Because the researchers did not have the benefit of haptic feedback, they relied on other predictive fields in Artificial Intelligence to explore the difficulty of guessing where a user’s fingers would logically head to next. FRL also stated that its researchers leveraged the statistical decoding techniques from automatic speech recognition, in effect, replacing the phonemes for hand motion to predict the keystrokes.
This was used in concert with a language model to predict what people intended to type in spite of ambiguous or “erratic” hand motions. By leveraging this new method, typists using the system averaged 73 words per minute and had a 2.4% uncorrected error rate using merely their hands and a flat surface and were also able to achieve a similar speed and accuracy as that of typist who is using an actual physical keyboard.
The FRL researchers performed the research on typing with hands through the use of a high-precision external hand-tracking system. This was likely used to prove that the use of hand movements alone was sufficient to reliably predict intended typing before attempting to adapt the system for use with less accurate hand-tracking data such as what is capable with the Oculus Quest or Quest 2 headset. Hand tracking on Quest is still a new and ongoing research frontier and will likely get better with time. The image below shows how Facebook was able to grapple with tracking a single hand across multiple camera views while also seamlessly transitioning between them.
The insights into hand tracking will undoubtedly prepare Facebook for the next generation of Augmented Reality headsets. The next-gen AR headset might take the form of ‘always on’ kind of standalone Augmented Reality headsets that users might wear anywhere including in the car, at home, at work and anywhere else and which they can only take off when they need to recharge it. For Facebook, therefore, the use of Quest 2 as a test bed for AR interactions seems like a very logical step.
No Upper Cases Letters
The FRL system still has some limitations. For example, it currently supports only the lowercase letters on the virtual keyboard. So, in spite of the promise, FRL’s AI software is still a far cry away from being a replacement to a physical keyboard.
The next steps for the researchers would be to expand the keyboard function in order to realize the same typing precision with the tracking cameras integrated in Oculus Quest. Whether and when this will happen isn’t clear from the research paper. According to the Facebook scientists, this project is still in its early research phase but it shows the potential of hand tracking for productivity-based use-cases.
Until Facebook’s virtual VR keyboard is ready, Facebook is currently providing an interim solution: the real keyboards (by Logitech) can be digitally displayed and operated in Virtual Reality via tracking. Facebook announced this feature alongside Infinite Office in September with aim of making this a real productivity package for those who work virtually in VR.
Source: Research Paper
Related Posts
https://virtualrealitytimes.com/2020/10/25/facebook-researchers-develop-invisible-finger-tracking-keyboard/https://virtualrealitytimes.com/wp-content/uploads/2020/10/Facebook-Invisible-Hand-Tracking-Keyboard-600×361.jpghttps://virtualrealitytimes.com/wp-content/uploads/2020/10/Facebook-Invisible-Hand-Tracking-Keyboard-150×90.jpgTechnologyUsing AI training and statistical word prediction, researchers at Facebook Reality Labs (FRL) are developing an invisible finger tracking keyboard that could be used in combination with Virtual Reality headsets like Oculus Quest. One of the most basic dilemmas surrounding AR/VR tech has been how to make text input in…Sam OchanjiSam Ochanjisochanji@yahoo.comAdministratorVirtual Reality Times
