Biography

I am a postdoctoral researcher at Max Planck Institute for Informatics with Prof. Christian Theobalt. I received my Ph.D. degree of Computer Science & Engineering from Nanyang Technological University, Singapore in 2021, supervised by Prof. Shijian Lu. I also worked as research intern at Alibaba DAMO Academy. I obtained my bachelor degree in Communication Engineering from University of Electronic Science and Technology of China.
My research interests are Generative Models and Neural Rendering (e.g., NeRF, 3D GAN, Diffusion Models). Aligning with the dictum of “What I cannot create, I do not understand” by Richard Feynman, my research goals are (1) Developing algorithms to synthesize and reconstruct (create) the visual world and (2) Exploring how visual analysis (understand) can benifit from the advance of visual synthesis.

News

∙ [01, 2023] Our Neural Gauge Fields is accetped to ICLR 2023.
∙ [12, 2022] Our workshop Generative Models for Computer Vision is accetped to CVPR 2023.
∙ [09, 2022] One paper is accepted to NeurIPS 2022.
∙ [07, 2022] Our survey paper Multimodal Image Synthesis and Editing is updated.
∙ [07, 2022] Two papers are accepted to ECCV 2022 (1 Oral).
∙ [06, 2022] Two papers are accepted to ACM MM 2022.
∙ [03, 2022] Three papers are accepted to CVPR 2022 (1 Oral).
∙ [01, 2022] One paper is accepted to TIP.

Research Overview














Publications

General Neural Gauge Fields 🔥🔥🔥

Fangneng Zhan, Lingjie Liu, Adam Kortylewski, Christian Theobalt
International Conference on Learning Representations (ICLR), 2023
Paper | Project(Soon) | Code(Soon)
A general and unified framework for performing gauge transformations in neural fields.


Multimodal Image Synthesis and Editing: A Survey

Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Shijian Lu, Lingjie Liu, Adam Kortylewski, Christian Theobalt, Eric Xing
Under Submission, 2022
Paper | Project


Auto-regressive Image Synthesis with Integrated Quantization

Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Kaiwen Cui, Changgong Zhang, Shijian Lu
European Conference on Computer Vision (ECCV), 2022 (Oral Presentation)
Paper | Code


Bi-level Feature Alignment for Versatile Image Translation and Manipulation

Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Kaiwen Cui, Aoran Xiao, Shijian Lu, Chunyan Miao
European Conference on Computer Vision (ECCV), 2022
Paper | Code


VMRF: View Matching Neural Radiance Fields

Jiahui Zhang, Fangneng Zhan, Rongliang Wu, Yingchen Yu, Wenqing Zhang, Song Bai, Xiaoqin Zhang, Shijian Lu
ACM International Conference on Multimedia (ACM MM), 2022
Paper


Towards Counterfactual Image Manipulation via CLIP

Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jiahui Zhang, Shijian Lu, Miaomiao Cui, Xuansong Xie, Xiansheng Hua, Chunyan Miao
ACM International Conference on Multimedia (ACM MM), 2022
Paper | Code


Marginal Contrastive Correspondence for Guided Image Generation

Fangneng Zhan, Yingchen Yu, Rongliang Wu, Jiahui Zhang, Shijian Lu, Changgong Zhang
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022 (Oral Presentation)
Paper | Code | Supple


Modulated Contrast for Versatile Image Synthesis

Fangneng Zhan, Jiahui Zhang, Yingchen Yu, Rongliang Wu, Shijian Lu
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022
Paper | Code | Supple


Unbalanced Feature Transport for Exemplar-based Image Translation

Fangneng Zhan, Yingchen Yu, Kaiwen Cui, Gongjie Zhang, Shijian Lu, Jianxiong Pan, Changgong Zhang, Feiying Ma, Xuansong Xie, Chunyan Miao
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021
Paper | Code | Supple


WaveFill: A Wavelet-based Generation Network for Image Inpainting

Yingchen Yu, Fangneng Zhan, Shijian Lu, Jianxiong Pan, Feiying Ma, Xuansong Xie, Chunyan Miao
IEEE International Conference on Computer Vision (ICCV), 2021 (Oral Presentation)
Paper | Code


Sparse Needlets for Lighting Estimation with Spherical Transport Loss

Fangneng Zhan, Changgong Zhang, Wenbo Hu, Shijian Lu, Feiying Ma, Xuansong Xie, Ling Shao
IEEE International Conference on Computer Vision (ICCV), 2021
Paper | Code | Dataset | Supple


EMLight: Lighting Estimation via Spherical Distribution Approximation

Fangneng Zhan, Changgong Zhang, Yingchen Yu, Yuan Chang, Shijian Lu, Feiying Ma, Xuansong Xie
AAAI Conference on Artificial Intelligence (AAAI), 2021
Paper | Code | Dataset | Supple


GMLight: Lighting Estimation via Geometric Distribution Approximation

Fangneng Zhan, Yingchen Yu, Changgong Zhang, Rongliang Wu, Wenbo Hu, Shijian Lu, Feiying Ma, Xuansong Xie, Ling Shao
IEEE Transactions on Image Processing (TIP), 2022
Paper | Code | Dataset | Supple


Diverse Image Inpainting with Bidirectional and Autoregressive Transformers

Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, Chunyan Miao
ACM International Conference on Multimedia (ACM MM), 2021 (Oral Presentation)
Paper | Code


ESIR: End-To-End Scene Text Recognition via Iterative Image Rectification

Fangneng Zhan, Shijian Lu
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019
Paper | Code | Supple


Verisimilar Image Synthesis for Accurate Detection and Recognition of Texts in Scenes

Fangneng Zhan, Shijian Lu, Chuhui Xue
European conference on computer vision (ECCV), 2018
Paper | Code | Dataset | Supple


Projects & Datasets

Virtual Object Relighting (VOR) Dataset for Lighting Estimation Evaluation

We create an evaluation dataset consisting of 3D scenes to conduct virtual object insertion & rendering in Blender. The lighting estimaiton performance is evaluated by using the predicted illumination map as the environment light in Blender.
Dataset | Paper | Code


Real time Scene Text Detection and Recognition System

We design a real time system for the end-to-end detection and recognition of scene texts in videos. The texts in one frame is first localized with a detection model, then a recognition model is employed to recognize the croped scene text images.
Code | Video


Academic Service

Program Committee Member:
∙ AAAI 2022, 2021, 2020
Conference & Journal Reviewer:
∙ ICML2022, ICLR 2022, NeurIPS 2021, 2020, CVPR 2022, 2021, 2020, 2019, ICCV 2021, 2019, ECCV 2020, BMVC 2021, 2020, ACCV 2020, WACV 2022, 2021
∙ TPAMI, TIP, TMM, Pattern Recognition, Neurocomputing