University of Pennsylvania
zihaoden@seas.upenn.edu
[Github] [Google Scholar]
I am a graduate student at University of Pennsylvania with a major in Computer and Information Science. Before that, I received my Bachelor of Science in Computer Science from Carnegie Mellon University, where I was supervised by Prof.Min Xu from Xu lab and Prof.Louis-Philippe Morency from the MultiComp Lab, with a research focus on the foundations of multimodal machine learning and their applications.
Multimodal ML: Large multimodal AI systems lie in the core of the current machine learning research, as the models must be capable of processing diverse multimodal signals to be adopted and deployed in more domains. I study the fundamental problems in multimodal machine learning, including data heterogeneity and interactions, modality alignment and fusion, and scalable multimodal foundation models
Self-supervised and Multimodal Representation Learning: representation learning, as a fundamental research problem that closely relates to the
downstream task performances of any ML models, is inherently complex when heterogeneous data sources are present. Incorporating methods from
information theories and statistical analysis, my research focuses on developing representations that are inclusive of semantic meanings of all data and
the information necessary for a broader range of downstream tasks. Relevant Publications:
Model Explainability, Safety, and Visualization: The black-box nature of most deep leanring models has significantly hindered their applications to a
wider range of high-stake domains where model safety and explainability are of top priority. In order to deploy intelligent multimodal systems with
data from healthcare, finance, etc., my research concerns about designing faithful and interpretable explanations for black-box multimodal models. This involves
developing custom ways of presenting or visualizing explanations for different modality inputs, algorithms that explain and analyze cross-modal contribution of
features, and faithful and interpretable designs that unveils the underlying decision workflow of the models. Relevant publications:
ML in Medicine and Healthcare: I have great interests in developing reliable healthcare models that can assist the diagnosis of diseases, surgeries, and health monitoring.
We are working on very interesting projects like generating rigorous reports for thyroid nodules based on multi-sensory examination data and also edge devices that run times-series models
to monitor the activities and actions of animals. Some common considerations include explainability, robustness, and training with scarse data/domain adaptation.
MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response
Zihao Deng*, Yinghao Ma*, Yudong Liu, Rongchen Guo, Ge Zhang, Wenhu Chen, Wenhao Huang, Emmanouil Benetos
NAACL 2024 [arxiv].
Factorized Contrastive Learning: Going Beyond Multi-view Redundancy
Paul Pu Liang*, Zihao Deng*, Martin Ma*, James Zou, Louis-Philippe Morency, Ruslan Salakhutdinov
NeurIPS 2023 [arxiv].
Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework
Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, Ruslan Salakhutdinov, Louis-Philippe Morency
NeurIPS 2023 [arxiv].
MultiViz: An Analysis Benchmark for Visualizing and Understanding Multimodal Models
Paul Pu Liang, Yiwei Lyu, Gunjan Chhablani, Nihal Jain, Zihao Deng, Xingbo Wang, Louis-Philippe Morency, Ruslan Salakhutdinov
ICLR 2023 [arxiv].
Dime: Fine-grained interpretations of multimodal models via disentangled local explanations
Yiwei Lyu, Paul Pu Liang, Zihao Deng, Ruslan Salakhutdinov, Louis-Philippe Morency
AIES 2022 [arxiv].
Cryo-shift: Reducing domain shift in cryo-electron subtomograms with unsupervised domain adaptation and randomization
Hmrishav Bandyopadhyay, Zihao Deng, Leiting Ding, Sinuo Liu, Mostofa Rafid Uddin, Xiangrui Zeng, Sima Behpour, Min Xu
Bioinformatics 2021 [arxiv].
(* denotes equal contribution)