Smart eyewear (augmented reality glasses, smart frames, and audio spectacles) is rapidly emerging as the next major personal computing platform. However, transitioning advanced audio capabilities from traditional form factors (like smartphones or over-ear headphones) to a lightweight pair of glasses introduces unprecedented acoustic and hardware challenges.This workshop explores the intersection of spatial audio, speech enhancement, and resource-constrained edge computing.The scope covers algorithmic and architectural solutions tailored to the unique realities of eyewear:

• Egocentric Array Processing: Managing extreme acoustic shadowing from the user's head and optimizing tightly spaced microphone arrays localized around the temples;
• Sensor Fusion: Combining acoustic microphones with alternative sensors (e.g., bone-conduction sensors and accelerometers) for robust speech extraction in high-noise environments;
• Severe Resource Deficits: Developing ultra-low-latency algorithms capable of running within microwatt power constraints;
• Acoustics & Transduction: Mitigating severe wind noise across the frames and managing micro-loudspeaker playback to ensure acoustic transparency (hear-through) without feedback, own-voice amplification, and interferences.

Experts from both academia and industry will provide a comprehensive overview of audio processing for smart eyewear, covering the state of the art and the field's challenges from both practical and theoretical perspectives.

Program:


Monday, Sept. 7
1:30 PM - 2:00 PM: Compact Microphone and Loudspeaker Array Design for AI Smart Glasses - Prof. Jingdong Chen
2:00 PM - 2:30 PM: Hybrid Model-Based and Learning-Based Approaches for Speech Enhancement in Hearables - Prof. Simon Doclo
2:30 PM - 3:00 PM: Audio Processing for Augmented Reality Glasses - Dr. Ivan Tashev
3:00 PM - 3:30 PM: Coffee Break
3:30 PM - 4:00 PM: How do we Change the (Hearing) World? - Dr. Marcus Anders Holmberg
4:00 PM - 4:30 PM: From Hearing Challenges to Everyday Adoption: Clinical Validation of Nuance Audio Plus Glasses - Maayan Koenigstein Priel

Venue: Politecnico di Milano - Polo Territoriale di Cremona

Organization:
Prof. Israel Cohen - Technion & Polimi
Dr. Niccolò Antonello - EssilorLuxottica

AI-powered smart glasses are emerging as a new generation of consumer devices, requiring compact and efficient microphone and loudspeaker arrays for seamless voice communication and human-AI speech interaction. These arrays must accurately capture the wearer's speech while delivering high-quality personal audio with minimal acoustic leakage, ensuring both robust performance in noisy environments and user privacy. In this talk, I will present our recent work on compact superarrays for speech acquisition and high-gain loudspeaker arrays for personal audio in smart glasses. The proposed superarray combines a small number of omnidirectional and directional microphones to achieve highly directional beamforming despite the limited array aperture. I will also introduce compact loudspeaker arrays that generate spatially focused sound fields, delivering clear audio to the wearer while substantially reducing sound leakage. The talk will highlight the underlying design principles, signal processing techniques, and representative experimental results.

In assistive listening devices, speech quality and speech understanding may be severely degraded by background noise, competing speakers, and acoustic feedback. Hence, robust speech enhancement and feedback cancellation algorithms are required that are online-capable and perform well in dynamic acoustic scenarios. In this talk, I will present recent advances in model-based and deep learning-based approaches for multi-microphone speech enhancement and feedback cancellation in hearables. First, I will present hybrid speech enhancement algorithms in which quantities required by a model-based enhancement stage are estimated by a learning-based stage. This includes deep multi-frame MVDR beamforming, where I will show how structured temporal and spatio-temporal representations, as well as spatial regularization, can improve the interpretability of the learned quantities. Second, I will present deep learning-based own voice reconstruction for hearables equipped with an outer and an in-ear microphone, jointly addressing noise reduction and bandwidth extension by exploiting phoneme-dependent own voice transfer characteristics. Third, I will discuss deep feedback cancellation for hearing aids, focusing on an in-the-loop training strategy that improves robustness at high amplification gains. Together, these contributions illustrate how combining acoustic modeling with deep learning can lead to interpretable, robust, and real-time-capable speech processing algorithms for future assistive listening devices.

This talk will start by outlining the typical hardware configuration of augmented reality (AR) glasses and discussing key design tradeoffs. It will review the main audio processing blocks, as well as the audio consumers and generators in core scenarios. Examples will span classic statistical signal processing, discriminative AI methods, and generative AI-based approaches. The talk will conclude with a discussion of the role of domain experts in the modern AI era. It will be illustrated with examples from the design of HoloLens and HoloLens 2 and audio processing work from the Audio and Acoustics Research Group at Microsoft Research.

No matter how we look at the data, hearing impairment remains a woefully undertreated handicap. We are also learning more and more that untreated hearing loss is a health issue with consequences that extend well beyond hearing, speech understanding, and the other outcomes we have traditionally measured. Looking carefully at the data, there have been improvements over the past years. However, progress has been slow. This talk will focus on the range of potential limitations to treatment—technical, psychological, and beyond—as well as a range of potential ways to improve the treatment uptake. What types of research can we learn from beyond the hearing and audiology fields? What does that mean for products, and in particular for future signal processing strategies? The challenge ahead is not only to make hearing devices perform better in the laboratory, but to make them more valuable and relevant in everyday life. This may require new ways of thinking about signal processing, product design, and the role of hearing care itself.

Millions of adults experience hearing difficulties yet do not adopt traditional hearing aids, often due to stigma, perceived need, or barriers to access. Nuance Audio Glasses represent a new category of wearable technology that combines auditory support with familiar eyewear, potentially offering a more approachable pathway to hearing assistance. This presentation will review the evidence-generation journey behind Nuance Audio Plus Glasses, focusing on both controlled laboratory assessment and real-world user experience. Clinical studies evaluated speech understanding in noise using standardized speech perception measures, while ecological and patient-reported outcome methodologies captured benefit during everyday listening situations. Participants used the technology in their daily environments, providing insight into communication performance, listening effort, usability, and overall acceptance. The presentation will discuss key lessons learned from translating audio-processing innovation into a consumer wearable product and explore the role of smart eyewear in expanding access to hearing support for adults with mild-to-moderate hearing difficulties. Results demonstrate improved speech understanding in challenging acoustic environments, accompanied by positive real-world outcomes and high user acceptance. The findings highlight the importance of evaluating emerging hearing technologies not only through traditional laboratory metrics but also through evidence reflecting everyday use and user behavior.