Multimodal (audio-visual) speech and video understanding
Yi Wang
PhD Student · University of Edinburgh, United Kingdom
Short Bio
I am a PhD student at the University of Edinburgh, supervised by Prof. Peter Bell and Dr. Hao Tang. I expect to graduate in spring or summer 2028. Recently, I have also worked closely with Dr. Tianzi Wang. My current research focuses on multimodal speech representation analysis, instruction optimization, and multimodal agents for video understanding.
Prior to that, I obtained an MPhil degree at The Chinese University of Hong Kong, supervised by Prof. Xunying Liu and Prof. Helen Meng, during which I focused on Automatic Neurocognitive Disorder Detection. I received my BA and MEng degrees in Information Engineering from the University of Cambridge.
- Multimodal agents and audio-visual reasoning
- Audio-visual speech representation learning
- Neurocognitive disorder detection
Selected Publications
-
Designing Textual Interfaces to Enhance LLM Reasoning on Audio-Visual Question Answering
Under review at ICASSP 2027
Investigated LLM-based instruction self-improvement for multimodal agent systems in audio-visual video understanding.
-
Efficient Agentic Audio-Visual Question Answering through Toolset Pruning and Modality Consolidation
Under review at ACL Rolling Review
Investigated how toolset design and modality consolidation affect the efficiency of agentic audio-visual reasoning, reducing latency and token consumption by approximately 70% while maintaining competitive performance.
arXiv forthcoming -
Rethinking Discrete Speech Representation Tokens for Accent Generation
arXiv preprint, 2026
Quantitatively analysed how source self-supervised speech representations and codebook sizes affect the accent information encoded in discrete speech tokens.
-
The Role of Audio-Visual Integration in the Time Course of Phonetic Encoding in Self-Supervised Speech Models
Interspeech 2025
Studied the temporal dynamics of phonetic encoding and showed that audio-visual self-supervised models do not fully capture natural audio-visual asynchrony (between lip movement and speech audio).
-
Exploiting Prompt Learning with Pre-trained Language Models for Alzheimer’s Disease Detection
ICASSP 2023
Investigated prompt-based fine-tuning with interpretable linguistic features for transformer-based automatic Alzheimer’s Disease detection.
-
Exploring Linguistic Feature and Model Combination for Speech Recognition Based Automatic AD Detection
Interspeech 2022
Developed BERT- and RoBERTa-based systems for automatic Alzheimer’s Disease detection, achieving 93.75% accuracy on the ADReSS dataset.
* Equal contribution.