I am a postdoctoral researcher at Max Planck Institute for Informatics with Prof. Christian Theobalt. I received my Ph.D. degree of Computer Science & Engineering from Nanyang Technological University, Singapore in 2021, supervised by Prof. Shijian Lu. I also worked as research intern at Alibaba DAMO Academy. I obtained my bachelor degree in Communication Engineering from University of Electronic Science and Technology of China.
Coinciding with the dictum of “What I cannot create, I do not understand” by Richard Feynman, my research goals are developing & understanding intelligence emerged from the visual generation & rendering process (i.e., Generative AI). With Neural Rendering and Generative Models as general learning machine, the detailed research directions include:
∙ Render-to-learn, e.g., gauge (Neural Gauge Fields), camera pose (VMRF), semantic fields (3D-OVS).
∙ Generate-to-learn, e.g., correspondence (UNITE), representation (IQ-VAE), alignment (SF-GAN).
∙ AI-generated content and data source, e.g., 2D data (GA-DAN), 3D-aware data (EMLight).
∙ [09, 2023] One paper is accetped to IJCV, one is accepted to NeurIPS 2023.
∙ [08, 2023] One paper about Generative AI is accetped to TPAMI 2023.
∙ [07, 2023] Two papers are accetped to ICCV 2023.
∙ [03, 2023] Three papers are accetped to CVPR 2023.
∙ [01, 2023] Our Neural Gauge Fields is accepted to ICLR 2023.
∙ [12, 2022] We are organizing workshop on Generative Models for Computer Vision at CVPR 2023.
∙ [09, 2022] One paper is accepted to NeurIPS 2022.
∙ [07, 2022] Two papers are accepted to ECCV 2022 (1 Oral).
∙ [03, 2022] Three papers are accepted to CVPR 2022 (1 Oral).
General Neural Gauge Fields
Fangneng Zhan, Lingjie Liu, Adam Kortylewski, Christian Theobalt
International Conference on Learning Representations (ICLR), 2023
Paper | Project | Code
A general paradigm and framework for learning gauge transformations in neural fields.
Multimodal Image Synthesis and Editing: The Generative AI
Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Shijian Lu, Lingjie Liu, Adam Kortylewski, Christian Theobalt, Eric Xing
IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023
Paper | Project | Code
3D Open-vocabulary Segmentation with Foundation Models
Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, Shijian Lu
Advances in Neural Information Processing Systems (NeurIPS), 2023
Paper | Code
WaveNeRF: Wavelet-based Generalizable Neural Radiance Fields
A Deeper Analysis of Volumetric Relightiable Fields
Pramod Rao, Mallikarjun B R, Gereon Fox, Tim Weyrich, Bernd Bickel, Hanspeter Pfister, Wojciech Matusik, Fangneng Zhan, Ayush Tewari, Christian Theobalt, Mohamed Elgharib
International Journal of Computer Vision (IJCV), 2023 (Invited Paper)
Paper | Project
StyleRF: Zero-shot 3D Style Transfer of Neural Radiance Fields
Kunhao Liu, Fangneng Zhan, Yiwen Chen, Jiahui Zhang, Yingchen Yu, Abdulmotaleb El Saddik, Shijian Lu, Eric Xing
IEEE Conference on Computer Vision and Pattern Recognition (CVPR)，2023
Paper | Project | Code
VMRF: View Matching Neural Radiance Fields
Jiahui Zhang, Fangneng Zhan, Rongliang Wu, Yingchen Yu, Wenqing Zhang, Song Bai, Xiaoqin Zhang, Shijian Lu
ACM International Conference on Multimedia (ACM MM), 2022
Auto-regressive Image Synthesis with Integrated Quantization
Bi-level Feature Alignment for Versatile Image Translation and Manipulation
Marginal Correspondence for Conditional Image Generation
Modulated Contrast for Versatile Image Synthesis
GMLight: Lighting Estimation via Geometric Distribution Approximation
Sparse Needlets for Lighting Estimation with Spherical Transport Loss
EMLight: Lighting Estimation via Spherical Distribution Approximation
WaveFill: A Wavelet-based Generation Network for Image Inpainting
Unbalanced Feature Transport for Exemplar-based Image Translation
Fangneng Zhan, Yingchen Yu, Kaiwen Cui, Gongjie Zhang, Shijian Lu, Jianxiong Pan, Changgong Zhang, Feiying Ma, Xuansong Xie, Chunyan Miao
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021
Paper | Project | Code | Supple
Diverse Image Inpainting with Bidirectional and Autoregressive Transformers
Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, Chunyan Miao
ACM International Conference on Multimedia (ACM MM), 2021 (Oral Presentation)
Paper | Code
ESIR: End-To-End Scene Text Recognition via Iterative Image Rectification
Verisimilar Image Synthesis for Accurate Detection and Recognition of Texts in Scenes
Projects & Datasets
Virtual Object Relighting (VOR) Dataset for Lighting Estimation Evaluation
We create an evaluation dataset consisting of 3D scenes to conduct virtual object insertion & rendering in Blender. The lighting estimaiton performance is evaluated by using the predicted illumination map as the environment light in Blender.
Dataset | Paper | Code
Real time Scene Text Detection and Recognition System
We design a real time system for the end-to-end detection and recognition of scene texts in videos. The texts in one frame is first localized with a detection model, then a recognition model is employed to recognize the croped scene text images.
Code | Video
Workshops & Tutorials & Talks:
∙ Organizer, CVPR 2023 Workshop: Generative Models for Computer Vision.
∙ Speaker, The AI Talks invited talks: On the Gauge Transformation of Neural Fields.
Program Committee Member:
∙ AAAI 2022, 2021, 2020
Conference & Journal Reviewer:
∙ ICML2022, ICLR 2022, NeurIPS 2021, 2020, CVPR 2022, 2021, 2020, 2019, ICCV 2021, 2019, ECCV 2020, BMVC 2021, 2020, ACCV 2020, WACV 2022, 2021
∙ TPAMI, TIP, TMM, Pattern Recognition, Neurocomputing