Research

The primary objective of my research is to develop robust multi-modal emotion recognition (MMER) systems that assess emotional distress in real-time, specifically within the context of mental health and suicide prevention. By incorporating multiple data modalities—such as raw speech signals, text, and visual cues—this research provides a comprehensive approach to understanding and detecting emotional states, facilitating early intervention for individuals at risk.

Multi-Modal Emotion Recognition for Crisis Intervention

MMER represents a significant advancement over single-channel analysis, enabling a fuller capture of emotional cues across various input forms. In this research, speech, text, and visual data are integrated to create a holistic representation of emotional states. Speech signals, which contain essential prosodic and paralinguistic cues, are particularly valuable in capturing distress in real-time phone calls. Text data further enriches this context by identifying keywords and sentiments expressed by the speaker, while visual cues, when available, add an additional layer of insight through facial expressions or gestures.

This multi-faceted approach is tailored for high-stakes scenarios, such as crisis intervention and emergency call centers, where comprehensive emotional understanding can lead to more effective support and life-saving actions. The integration of these diverse data streams ensures that even subtle expressions of emotional distress are detected, providing a more reliable and accurate assessment of the speaker’s state.

Speech Emotion Recognition through Advanced Deep Learning Architectures

A core component of my work involves designing sophisticated deep learning architectures to analyze raw speech signals. Unlike traditional SER systems that depend on handcrafted features, my aim is to process raw waveforms directly, allowing for the extraction of fine-grained emotional features. I have developed hybrid CNN-GRU architectures that combine convolutional neural networks (CNNs) for capturing local features with gated recurrent units (GRUs) to model long-term dependencies within the speech signal. This design enables the system to identify both immediate and cumulative emotion patterns, creating a dynamic and accurate understanding of the speaker’s emotional state.

Wavelet-based transformations further enhance these models by capturing information across both time and frequency domains. Through multi-resolution wavelet analysis, the models achieve precise localization of critical emotional features, even in noisy environments or with variable-length audio segments. This approach has shown to outperform traditional models, offering a highly accurate solution for detecting distress signals in real time.

Model Explainability and Ethical Considerations in AI

A crucial component of my research is ensuring that the AI models are not only accurate but also transparent and interpretable. Explainable AI (XAI) techniques are integral to this work, providing insights into how specific features contribute to the emotion predictions made by the models. Using iterative feature boosting and Shapley values, the framework continuously refines feature sets to retain only the most impactful information for emotion recognition. This feedback mechanism not only enhances model performance but also ensures that the decision-making process is accessible to mental health professionals.

Transparency in AI is essential for applications in sensitive areas such as mental health, where ethical considerations play a central role. By making model predictions interpretable, I aim to build trust and ensure that AI tools can be safely integrated into crisis intervention and support systems, empowering professionals to make informed decisions based on the model’s insights.