A real-time silent speech system for voice restoration after total laryngectomy

Abstract

Background and aim

Individuals who have lost their voice following a laryngectomy as a treatment for cancer will inevitably struggle with their daily communication. Unfortunately, the current methods for speaking after laryngectomy all have limitations, either because of the poor acoustics generated by these methods or because they are potentially harmful. The aim of this work is thus to explore an alternative method for post-laryngectomy voice restoration in which the movement of the intact articulators is captured and then converted into audible speech using machine learning techniques.

Materials and methods

To demonstrate the feasibility of speech generation from captured articulator movement, 6 healthy adults were recruited. For each subject, both the speech acoustics and the subject's articulator movements were recorded simultaneously. Articulator movements were captured using a technique known as permanent magnet articulography (PMA), in which small magnets are attached to the articulators (typically tongue and lips) and the magnetic field generated by the magnets is captured with sensors located close to the mouth. Deep artificial neural networks were then used to model the mapping between the sensor data and the speech acoustics, thus, enabling the synthesis of speech from captured articulatory data.

Results

The proposed silent speech system is able to generate speech that sounds natural, resembles the subject's own voice and is fairly intelligible (up to 92% intelligibility for some speakers on a phonetically-rich corpus).

Conclusions

With further research, the proposed system could in future be a real option to restore lost voice after laryngectomy.

Resumen

Antecedentes y objetivo

Aquellas personas que han perdido su voz después de una laringectomía se ven limitadas irremediablemente en su comunicación diaria. A pesar de existir en la actualidad métodos para recuperar el habla tras la laringectomía, todos ellos presentan limitaciones. El objetivo de este trabajo es explorar un método alternativo para hablar tras la laringectomía, en el que el movimiento de los órganos de la voz se transforma en una señal acústica utilizando técnicas de aprendizaje automático.

Materiales y métodos

En esta investigación participaron 6 adultos sanos. Para cada sujeto se grabó tanto su voz como los movimientos de sus labios y lengua. Los movimientos de los órganos del habla fueron capturados usando una técnica conocida como Articulografía de Imán Permanente (PMA), en la cual pequeños imanes se colocan sobre estos órganos y el campo magnético generado por los imanes se captura usando unos sensores sensibles al campo magnético. Se utilizaron redes neuronales artificiales profundas para modelar la transformación entre los datos de los sensores y la acústica de la voz.

Resultados

El sistema de habla silenciosa propuesto es capaz de generar voz que suena natural, se asemeja a la propia voz del sujeto y es inteligible (hasta un 92% de inteligibilidad para algunos sujetos).

Conclusiones

El sistema propuesto podría ser en el futuro una opción viable para restaurar la voz tras una laringectomía total.