Ye Zhu

Postdoc Researcher

Ph.D. (she/her)

Princeton University

I am Ye Zhu (竺烨 in Chinese), currently a postdoc research associate in the Computer Science Department at Princeton University, working with Prof. Olga Russakovsky at the VisualAI lab. My main research focuses on Multimodality Generation, with interdisciplinary research interests in Machine Learning for Physics.

Prior to the postdoc, I received my Ph.D. in Computer Science from Illinois Institute of Technology, USA; and my M.S. and B.S. from Shanghai Jiao Tong University (SJTU), China. I completed my partial education within the French engineering system, which included three years of intensive mathematics and physics training as an undergraduate. Subsequently, I pursued studies at École Polytechnique (X), France and obtained my French Engineering Diploma (Diplôme d’Ingénieur in French).

I serve as a regular reviewer for NeurIPS, ICLR, ICML, CVPR, ECCV, ICCV, and SIGGRAPH. I also hold French language diplomas in both DALF and TCF C1, and was a part-time French translator for the European scientific magazine Science & Vie for 8 years.

news

Feb 25, 2024	I am recently giving a themed talk on the Tuning-free Paradigm for Versatile Applications with Diffusion Models at academia and industrial labs.
Feb 15, 2024	I am co-organizing the first Responsible Generative AI (ReGenAI) workshop at CVPR 24!
Jan 15, 2024	Our Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation (COW) paper accepted to ICLR 2024.
Sep 30, 2023	Our BoundaryDiffusion paper accepted to NeurIPS 2023.
Sep 2, 2023	Finish my Ph.D defense and join Princeton CS as a postdoc researcher.

selected publications

Boundary Guided Learning-Free Semantic Control with Diffusion Models

Ye Zhu, Yu Wu, Zhiwei Deng, Olga Russakovsky, and Yan Yan

In Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS), 2023

Bib Page PDF Code

@inproceedings{zhu2023boundary,
  title = {Boundary Guided Learning-Free Semantic Control with Diffusion Models},
  author = {Zhu, Ye and Wu, Yu and Deng, Zhiwei and Russakovsky, Olga and Yan, Yan},
  booktitle = {Thirty-seventh Conference on Neural Information Processing Systems (NeurIPS)},
  year = {2023},
}

Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation

Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan

In The Eleventh International Conference on Learning Representations (ICLR), 2023

Bib Page PDF Code

@inproceedings{zhu2023discrete,
  title = {Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation},
  author = {Zhu, Ye and Wu, Yu and Olszewski, Kyle and Ren, Jian and Tulyakov, Sergey and Yan, Yan},
  booktitle = {The Eleventh International Conference on Learning Representations (ICLR)},
  year = {2023},
}

Denoising Diffusion Probabilistic Models to Predict the Density of Molecular Clouds

Duo Xu, Jonathan C Tan, Chia-Jung Hsu, and Ye Zhu

The Astrophysical Journal, 2023

Bib PDF

@article{xu2023denoising,
  title = {Denoising Diffusion Probabilistic Models to Predict the Density of Molecular Clouds},
  author = {Xu, Duo and Tan, Jonathan C and Hsu, Chia-Jung and Zhu, Ye},
  journal = {The Astrophysical Journal},
  volume = {950},
  number = {2},
  pages = {146},
  year = {2023},
  publisher = {IOP Publishing},
}

Preprint

Vision+ X: A Survey on Multimodal Learning in the Light of Data

Ye Zhu, Yu Wu, Nicu Sebe, and Yan Yan

arXiv preprint arXiv:2210.02884, 2022

Bib PDF

@article{zhu2022vision+,
  title = {Vision+ X: A Survey on Multimodal Learning in the Light of Data},
  author = {Zhu, Ye and Wu, Yu and Sebe, Nicu and Yan, Yan},
  journal = {arXiv preprint arXiv:2210.02884},
  year = {2022},
}

Quantized GAN for Complex Music Generation from Dance Videos

Ye Zhu, Kyle Olszewski, Yu Wu, Panos Achlioptas, Menglei Chai, Yan Yan, and Sergey Tulyakov

In European Conference on Computer Vision (ECCV), 2022

Bib Page PDF Code

@inproceedings{zhu2022quantized,
  title = {Quantized GAN for Complex Music Generation from Dance Videos},
  author = {Zhu, Ye and Olszewski, Kyle and Wu, Yu and Achlioptas, Panos and Chai, Menglei and Yan, Yan and Tulyakov, Sergey},
  booktitle = {European Conference on Computer Vision (ECCV)},
  pages = {182--199},
  year = {2022},
  organization = {Springer},
}

CV/ML

Saying the Unseen: Video Descriptions via Dialog Agents

Ye Zhu, Yu Wu, Yi Yang, and Yan Yan

IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022

Bib PDF

@article{9468337,
  author = {Zhu, Ye and Wu, Yu and Yang, Yi and Yan, Yan},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)},
  title = {Saying the Unseen: Video Descriptions via Dialog Agents},
  year = {2022},
  volume = {44},
  number = {10},
  pages = {7190-7204},
  doi = {10.1109/TPAMI.2021.3093360},
}

CV/ML

Describing unseen videos via multi-modal cooperative dialog agents

Ye Zhu, Yu Wu, Yi Yang, and Yan Yan

In European Conference on Computer Vision (ECCV), 2020

Bib PDF Code

@inproceedings{zhu2020describing,
  title = {Describing unseen videos via multi-modal cooperative dialog agents},
  author = {Zhu, Ye and Wu, Yu and Yang, Yi and Yan, Yan},
  booktitle = {European Conference on Computer Vision (ECCV)},
  pages = {153--169},
  year = {2020},
  organization = {Springer},
}