How2comm: Communication-Efficient and Collaboration-Pragmatic Multi-Agent Perception

Dingkang Yang$^\star$, Kun Yang$^\star$, Yuzheng Wang, ..., Peng Zhai$^\dagger$, Lihua Zhang$^\dagger$

Project

• Multi-agent collaborative perception has recently received widespread attention as an emerging application in driving scenarios. We propose How2comm, a collaborative perception framework that seeks a trade-off between perception performance and communication bandwidth.

What2comm: Towards Communication-efficient Collaborative Perception via Feature Decoupling

Kun Yang$^\star$, Dingkang Yang$^\star$, Jingyu Zhang, ...

• We propose What2comm, a communication-efficient multiagent collaborative perception framework. Our framework outperforms previous approaches on real-world and simulated datasets by addressing various collaboration interferences, including communication noises, transmission delay, and localization errors, in an end-to-end manner.

AIDE: A Vision-Driven Multi-View, Multi-Modal, Multi-Tasking Dataset for Assistive Driving Perception

Dingkang Yang, Shuai Huang, Zhi Xu, ..., Lihua Zhang$^\dagger$

Project | ArXiv

• We propose an AssIstive Driving pErception dataset (AIDE) to facilitate further research on the vision-driven driver monitoring systems. AIDE captures rich information inside and outside the vehicle from several drivers in realistic driving conditions.

Spatio-Temporal Domain Awareness for Multi-Agent Collaborative Perception

Kun Yang$^\star$, Dingkang Yang$^\star$, Jingyu Zhang, ...

Project | ArXiv

• Multi-agent collaborative perception as a potential application for vehicle-to-everything communication could significantly improve the perception performance of autonomous vehicles over single-agent perception. We propose SCOPE, a novel collaborative perception framework that aggregates the spatio-temporal awareness characteristics across on-road agents in an end-to-end manner.

Context De-confounded Emotion Recognition

Dingkang Yang, Zhaoyu Chen, Yuzheng Wang, ..., Lihua Zhang$^\dagger$

Project | ArXiv

• We are the first to investigate the adverse context bias of the datasets in the context-aware emotion recognition task from the causal inference perspective and identify that such bias is a confounder, which misleads the models to learn the spurious correlation. In this case, we propose a contextual causal intervention module based on the backdoor adjustment to de-confound the confounder and exploit the true causal effect for model training.

Emotion Recognition for Multiple Context Awareness

Dingkang Yang$^\star$, Shuai Huang$^\star$, Shunli Wang, ..., Lihua Zhang$^\dagger$

Project | Supplementary | Data

• We present a context-aware emotion recognition framework that combines four complementary contexts.

Towards Robust Multimodal Sentiment Analysis under Uncertain Signal Missing

Mingcheng Li, Dingkang Yang, Lihua Zhang$^\dagger$

• We propose a robust multimodal missing signal framework to handle the problem of uncertain signal missing for the multimodal sentiment analysis task and can be generalized to other multimodal patterns.

Target and Source Modality Co-reinforcement for Emotion Understanding from Asynchronous Multimodal Sequences

Dingkang Yang, Yang Liu, Can Huang, ..., Peng Zhai$^\dagger$, Lihua Zhang$^\dagger$

• Inspired by the human perception paradigm, we propose a target and source modality co-reinforcement approach to achieve sufficient crossmodal interaction and fusion at different granularities.

Disentangled Representation Learning for Multimodal Emotion Recognition

Dingkang Yang, Shuai Huang, Haopeng Kuang, Yangtao Du, Lihua Zhang$^\dagger$

• We propose a feature-disentangled multimodal emotion recognition method, which learns the common and private feature representations for each modality.

Learning Modality-specific and -agnostic Representations for Asynchronous Multimodal Language Sequences

Dingkang Yang, Haopeng Kuang, Shuai Huang, Lihua Zhang$^\dagger$

• We propose a multimodal fusion approach for learning modality-specific and modality-agnostic representations to refine multimodal representations and leverage the complementarity across different modalities.

Contextual and Cross-modal Interaction for Multi-modal Speech Emotion Recognition

Dingkang Yang, Shuai Huang, Yang Liu, Lihua Zhang$^\dagger$

• We propose a multimodal speech emotion recognition method based on interaction awareness.