Multimodal (audio-visual) speech and video understanding

Yi Wang

PhD Student · University of Edinburgh, United Kingdom

Short Bio

I am a PhD student at the University of Edinburgh, supervised by Prof. Peter Bell and Dr. Hao Tang. I expect to graduate in spring or summer 2028. Recently, I have also worked closely with Dr. Tianzi Wang. My current research focuses on multimodal speech representation analysis, instruction optimization, and multimodal agents for video understanding.

Prior to that, I obtained an MPhil degree at The Chinese University of Hong Kong, supervised by Prof. Xunying Liu and Prof. Helen Meng, during which I focused on Automatic Neurocognitive Disorder Detection. I received my BA and MEng degrees in Information Engineering from the University of Cambridge.

  • Multimodal agents and audio-visual reasoning
  • Audio-visual speech representation learning
  • Neurocognitive disorder detection

Selected Publications

  1. Designing Textual Interfaces to Enhance LLM Reasoning on Audio-Visual Question Answering

    Yi Wang, Tianzi Wang, Tong Lei, Dahan Wang, Peter Bell

    Under review at ICASSP 2027

    Investigated LLM-based instruction self-improvement for multimodal agent systems in audio-visual video understanding.

  2. Efficient Agentic Audio-Visual Question Answering through Toolset Pruning and Modality Consolidation

    Yi Wang, Tianzi Wang, Peter Bell

    Under review at ACL Rolling Review

    Investigated how toolset design and modality consolidation affect the efficiency of agentic audio-visual reasoning, reducing latency and token consumption by approximately 70% while maintaining competitive performance.

  3. Rethinking Discrete Speech Representation Tokens for Accent Generation

    Jinzuomu Zhong*, Yi Wang*, Korin Richmond, Peter Bell

    arXiv preprint, 2026

    Quantitatively analysed how source self-supervised speech representations and codebook sizes affect the accent information encoded in discrete speech tokens.

  4. The Role of Audio-Visual Integration in the Time Course of Phonetic Encoding in Self-Supervised Speech Models

    Yi Wang, Oli Danyi Liu, Peter Bell

    Interspeech 2025

    Studied the temporal dynamics of phonetic encoding and showed that audio-visual self-supervised models do not fully capture natural audio-visual asynchrony (between lip movement and speech audio).

  5. Exploiting Prompt Learning with Pre-trained Language Models for Alzheimer’s Disease Detection

    Yi Wang, Jiajun Deng, Tianzi Wang, Bo Zheng, Shoukang Hu, Xunying Liu, Helen Meng

    ICASSP 2023

    Investigated prompt-based fine-tuning with interpretable linguistic features for transformer-based automatic Alzheimer’s Disease detection.

  6. Exploring Linguistic Feature and Model Combination for Speech Recognition Based Automatic AD Detection

    Yi Wang, Tianzi Wang, Zi Ye, Lingwei Meng, Shoukang Hu, Xixin Wu, Xunying Liu, Helen Meng

    Interspeech 2022

    Developed BERT- and RoBERTa-based systems for automatic Alzheimer’s Disease detection, achieving 93.75% accuracy on the ADReSS dataset.

* Equal contribution.