Sida Peng (彭思达)

Tenure-track Assistant Professor

3D Vision Group
School of Software Technology
Zhejiang University

Email: pengsida at zju dot edu dot cn

[Google Scholar][GitHub]

Short Bio

I am an assistant professor (ZJU-100 Young Professor) at Zhejiang University. I received my Ph.D. degree from College of Computer Science and Technology at Zhejiang University in 2023, supervised by Prof. Xiaowei Zhou and Prof. Hujun Bao, and obtained my bachelor degree in Information Engineering from Zhejiang University in 2018. I received the 2020 CCF-CV Excellent Young Researcher Award and was selected as the 2022 Apple Scholar in AI/ML. I have been maintaining a handbook on learning research, which have received more than 5900 stars on GitHub.

I work closely with Prof. Xiaowei Zhou. We are looking for students, postdocs and research assistants. Please apply to our lab here if you are interested in working with us.

News

[Show more]

Research Summary

I like to build useful systems and solve fundamental research problems that arise from important challenges when developing systems. I also like to explore the capability boundaries of new techniques on my target systems.

Volumetric Video
Driving Simulator
Egocentric AI

Preprints

Deep Generative Models on 3D Representations: A Survey
Zifan Shi*, Sida Peng*, Yinghao Xu*, Yiyi Liao, Yujun Shen (* equal contribution)
[Paper][Project Page]

Publications (* denotes equal contribution, and † denotes the corresponding author)

2024

Dyn-E: Local Appearance Editing of Dynamic Neural Radiance Fields
Yinji Shentu, Shangzhan Zhang, Mingyue Xu, Qing Shuai, Tianrun Chen, Sida Peng, Xiaowei Zhou
ChinaGraph 2024 Best Paper.
[Paper][Project Page]
Representing Long Volumetric Video with Temporal Gaussian Hierarchy
Zhen Xu, Yinghao Xu, Zhiyuan Yu, Sida Peng, Jiaming Sun, Hujun Bao, Xiaowei Zhou
SIGGRAPH Asia 2024 Journal Track.
World-Grounded Human Motion Recovery via Gravity-View Coordinates
Zehong Shen, Huaijin Pi, Yan Xia, Zhi Cen, Sida Peng†, Zechen Hu, Hujun Bao, Ruizhen Hu, Xiaowei Zhou
SIGGRAPH Asia 2024 Conference Track.
[Paper][Project Page][Code]
Modeling Emitters in Indoor Scenes for Inverse Rendering
Sida Peng, Xi Chen, Chenxu Zhou, Chengfei Lv, Zhe Wang, Gang Yu, Xiaowei Zhou
JCAD 2024. Best Paper of CCF CAD&CG 2024.
[Paper]
Street Gaussians for Modeling Dynamic Urban Scenes
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, Sida Peng†
ECCV 2024.
[Paper][Project Page][Code]
IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under Unknown Illumination
Xi Chen, Sida Peng, Dongchen Yang, Yuan Liu, Bowen Pan, Chengfei Lv, Xiaowei Zhou
ECCV 2024.
[Paper][Project Page][Code]
SAM-guided Graph Cut for 3D Instance Segmentation
Haoyu Guo*, Hu Zhu*, Sida Peng, Yuang Wang, Yujun Shen, Ruizhen Hu, Xiaowei Zhou
ECCV 2024.
[Paper][Project Page][Code]
TELA: Text to Layer-wise 3D Clothed Human Generation
Junting Dong, Qi Fang, Zehuan Huang, Xudong Xu, Jingbo Wang, Sida Peng, Bo Dai
ECCV 2024.
[Paper]
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation
Yinghao Xu, Zifan Shi, Yifan Wang, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, Gordon Wetzstein
ECCV 2024.
[Paper][Project Page]
GVGEN: Text-to-3D Generation with Volumetric Representation
Xianglong He, Junyi Chen, Sida Peng, Di Huang, Yangguang Li, Xiaoshui Huang, Chun Yuan, Wanli Ouyang, Tong He
ECCV 2024.
[Paper][Project Page]
UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation
Zexiang Liu, Yangguang Li, Youtian Lin, Xin Yu, Sida Peng, Yan-Pei Cao, Xiaojuan Qi, Xiaoshui Huang, Ding Liang, Wanli Ouyang
ECCV 2024.
[Paper][Project Page]
CPT-VR: Improving Surface Rendering via Closest Point Transform with View-Reflection Appearance
Zhipeng Hu, Yongqiang Zhang, Chen Liu, Lincheng Li, Sida Peng, Xiaowei Zhou, Changjie Fan, Xin Yu
ECCV 2024.
[Paper]
MaPa: Text-driven Photorealistic Material Painting for 3D Shapes
Shangzhan Zhang, Sida Peng, Tao Xu, Yuanbo Yang, Tianrun Chen, Nan Xue, Yujun Shen, Hujun Bao, Ruizhen Hu, Xiaowei Zhou
SIGGRAPH 2024 Conference Track.
[Paper][Project Page]
Neural 3D Scene Reconstruction with Indoor Planar Priors
Xiaowei Zhou, Haoyu Guo, Sida Peng, Yuxi Xiao, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao
TPAMI 2024.
[Paper][Supplementary Material][Project Page][Code]
4K4D: Real-Time 4D View Synthesis at 4K Resolution
Zhen Xu, Sida Peng, Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujun Bao, Xiaowei Zhou
CVPR 2024.
[Paper][Project Page][Code]
Relightable and Animatable Neural Avatar from Sparse-View Video
Zhen Xu, Sida Peng†, Chen Geng, Linzhan Mou, Zihan Yan, Jiaming Sun, Hujun Bao, Xiaowei Zhou
CVPR 2024 Highlight.
[Paper][Project Page][Code]
Detector-Free Structure from Motion
Xingyi He, Jiaming Sun, Yifan Wang, Sida Peng, Qixing Huang, Hujun Bao, Xiaowei Zhou
CVPR 2024. First Place in Image Matching Challenge 2023.
[Paper][Project Page][Code]
SpatialTracker: Tracking Any 2D Pixels in 3D Space
Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, Xiaowei Zhou
CVPR 2024 Highlight.
[Paper][Project Page][Code]
Efficient LoFTR: Semi-Dense Local Feature Matching with Sparse-Like Speed
Yifan Wang*, Xingyi He*, Sida Peng†, Dongli Tan, Xiaowei Zhou
CVPR 2024 Highlight.
[Paper][Project Page][Code]
Rapid 3D Model Generation with Intuitive 3D Input
Tianrun Chen, Chaotao Ding, Shangzhan Zhang, Chunan Yu, Ying Zang, Zejian Li, Sida Peng, Lingyun Sun
CVPR 2024 Highlight. [Paper]
Generating Human Motion in 3D Scenes from Text Descriptions
Zhi Cen, Huaijin Pi, Sida Peng†, Zehong Shen, Minghui Yang, Shuai Zhu, Hujun Bao, Xiaowei Zhou
CVPR 2024.
[Paper][Project Page]
PlaneStereo: Plane-Aware Multi-View Stereo
Haoyu Guo, Sida Peng, Ting Shen, Xiaowei Zhou
MIR 2024.
Neural Polynomial Gabor Fields for Macro Motion Analysis
Chen Geng, Hong-Xing Yu, Sida Peng, Xiaowei Zhou, Jiajun Wu
ICLR 2024.
[Paper]
Animatable Implicit Neural Representations for Creating Realistic Avatars from Videos
Xiaowei Zhou, Sida Peng†, Zhen Xu, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Hujun Bao
TPAMI 2024.
[Paper][Supplementary Material][Project Page][Code]
A Guide to Conducting a Research Project
Sida Peng
CCCF 2024.
[Article][Project Page][Talk]

2023

EasyVolcap: Accelerating Neural Volumetric Video Research
Zhen Xu, Tao Xie, Sida Peng, Haotong Lin, Qing Shuai, Zhiyuan Yu, Guangzhao He, Jiaming Sun, Hujun Bao, Xiaowei Zhou
SIGGRAPH Asia 2023 Technical Communications.
[Paper][Code]
Compact Neural Volumetric Video Representations with Dynamic Codebooks
Haoyu Guo, Sida Peng†, Yunzhi Yan, Linzhan Mou, Yujun Shen, Hujun Bao, Xiaowei Zhou
NeurIPS 2023.
[Paper][Code]
Benchmarking and Analyzing 3D-aware Image Synthesis with a Modularized Codebase
Qiuyu Wang, Zifan Shi, Kecheng Zheng, Yinghao Xu, Sida Peng, Yujun Shen
NeurIPS 2023 Datasets and Benchmarks Track.
[Paper][Code]
Im4D: High-Fidelity and Real-Time Novel View Synthesis for Dynamic Scenes
Haotong Lin, Sida Peng†, Zhen Xu, Tao Xie, Xingyi He, Hujun Bao, Xiaowei Zhou
SIGGRAPH Asia 2023.
[Paper][Project Page][Code]
iVS-Net: Learning Human View Synthesis from Internet Videos
Junting Dong, Qi Fang, Tianshuo Yang, Qing Shuai, Chengyu Qiao, Sida Peng†
ICCV 2023.
[Paper][Supplementary Material][Project Page]
Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models
Huaijin Pi, Sida Peng, Minghui Yang, Xiaowei Zhou, Hujun Bao
ICCV 2023.
[Paper][Project Page][Code]
Ponder: Point Cloud Pre-training via Neural Rendering
Di Huang, Sida Peng, Tong He, Xiaowei Zhou, Wanli Ouyang
ICCV 2023.
[Paper][Project Page]
Representing Volumetric Videos as Dynamic MLP Maps
Sida Peng*, Yunzhi Yan*, Qing Shuai, Hujun Bao, Xiaowei Zhou
CVPR 2023.
[Paper][Project Page][Code]
Learning Neural Volumetric Representations of Dynamic Humans in Minutes
Chen Geng*, Sida Peng*, Zhen Xu*, Hujun Bao, Xiaowei Zhou
CVPR 2023.
[Paper][Project Page][Code]
Painting 3D Nature in 2D: View Synthesis of Natural Scenes from a Single Semantic Mask
Shangzhan Zhang, Sida Peng, Tianrun Chen, Linzhan Mou, Haotong Lin, Kaicheng Yu, Yiyi Liao, Xiaowei Zhou
CVPR 2023.
[Paper][Project Page][Code]
AutoRecon: Automated 3D Object Discovery and Reconstruction
Yuang Wang, Xingyi He, Sida Peng, Haotong Lin, Hujun Bao, Xiaowei Zhou
CVPR 2023 Highlight.
[Paper][Project Page][Code]
Learning Human Mesh Recovery in 3D Scenes
Zehong Shen, Zhi Cen, Sida Peng, Qing Shuai, Hujun Bao, Xiaowei Zhou
CVPR 2023.
[Paper][Project Page][Code]
Grid-guided Neural Radiance Fields for Large Urban Scenes
Linning Xu*, Yuanbo Xiangli*, Sida Peng, Xingang Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, Dahua Lin
CVPR 2023.
[Paper][Project Page]
Neural Scene Chronology
Haotong Lin, Qianqian Wang, Ruojin Cai, Sida Peng, Hadar Elor, Xiaowei Zhou, Noah Snavely
CVPR 2023.
[Paper][Project Page][Code]
DisCoScene: Spatially Disentangled Generative Radiance Fields for Controllable 3D-aware Scene Synthesis
Yinghao Xu, Menglei Chai, Zifan Shi, Sida Peng, Ivan Skorokhodov, Aliaksandr Siarohin, Ceyuan Yang, Yujun Shen, Hsin-Ying Lee, Bolei Zhou, Sergey Tulyakov
CVPR 2023 Highlight.
[Paper][Project Page][Code]
Learning 3D-aware Image Synthesis with Unknown Pose Distribution
Zifan Shi*, Yujun Shen*, Yinghao Xu, Sida Peng, Yiyi Liao, Sheng Guo, Qifeng Chen, Dit-Yan Yeung
CVPR 2023.
[Paper][Project Page][Code]
Implicit Neural Representations with Structured Latent Codes for Human Body Modeling
Sida Peng, Chen Geng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Xiaowei Zhou, Hujun Bao
TPAMI 2023.
[Paper][Code]

2022

TotalSelfScan: Learning Full-body Avatars from Self-Portrait Videos of Faces, Hands, and Bodies
Junting Dong, Qi Fang, Yudong Guo, Sida Peng, Qing Shuai, Hujun Bao, Xiaowei Zhou
NeurIPS 2022.
[Paper][Project Page][Code]
Efficient Neural Radiance Fields for Interactive Free-viewpoint Video
Haotong Lin*, Sida Peng*, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, Xiaowei Zhou
SIGGRAPH Asia 2022.
[Paper][Project Page][Code]
MPS-NeRF: Generalizable 3D Human Rendering from Multiview Images
Xiangjun Gao, Jiaolong Yang, Jongyoo Kim, Sida Peng, Zicheng Liu, Xin Tong
TPAMI 2022.
[Paper][Project Page][Code]
NerfCap: Human Performance Capture with Dynamic Neural Radiance Fields
Kangkan Wang, Sida Peng, Xiaowei Zhou, Jian Yang, and Guofeng Zhang
TVCG 2022.
[Paper]
Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Images
Yuan Liu, Yilin Wen, Sida Peng, Cheng Lin, Xiaoxiao Long, Taku Komura, Wenping Wang
ECCV 2022.
[Paper][Project Page][Code]
Learning to Estimate Object Poses without Real Image Annotations
Haotong Lin*, Sida Peng*, Zhize Zhou, Xiaowei Zhou
IJCAI 2022.
[Paper][Code]
Novel View Synthesis of Human Interactions from Sparse Multi-view Videos
Qing Shuai, Chen Geng, Qi Fang, Sida Peng, Wenhao Shen, Hujun Bao, Xiaowei Zhou
SIGGRAPH 2022.
[Paper][Project Page][Code]
Neural 3D Scene Reconstruction with the Manhattan-world Assumption
Haoyu Guo*, Sida Peng*, Haotong Lin, Qianqian Wang, Guofeng Zhang, Hujun Bao, Xiaowei Zhou
CVPR 2022 Oral presentation.
[Paper][Supplementary Material][Project Page][Code]
3D-aware Image Synthesis via Learning Structural and Textural Representations
Yinghao Xu, Sida Peng, Ceyuan Yang, Yujun Shen, Bolei Zhou
CVPR 2022.
[Paper][Project Page][Code]
Neural Rays for Occlusion-aware Image-based Rendering
Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, Wenping Wang
CVPR 2022.
[Paper][Project Page][Code]
Towards Efficient and Photorealistic 3D Human Reconstruction: A Brief Survey
Lu Chen, Sida Peng, Xiaowei Zhou
Visual Informatics 2022.
[Paper]

2021

Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies
Sida Peng*, Junting Dong*, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, Hujun Bao
ICCV 2021.
[Paper][Supplementary Material][Project Page][Code][Extension]
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans
Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, Xiaowei Zhou
CVPR 2021 Best paper candidate.
[Paper][Supplementary Material][Project Page][Code]

2020

PVNet: Pixel-wise Voting Network for 6DoF Object Pose Estimation
Sida Peng*, Xiaowei Zhou*, Yuan Liu, Haotong Lin, Qixing Huang, Hujun Bao
TPAMI 2020.
[Paper][Code]
Deep Snake for Real-Time Instance Segmentation
Sida Peng, Wen Jiang, Huaijin Pi, Xiuli Li, Hujun Bao, Xiaowei Zhou
CVPR 2020 Oral presentation.
[Paper][Code]

2019

GIFT: Learning Transformation-Invariant Dense Visual Descriptors via Group CNNs
Yuan Liu, Zehong Shen, Zhixuan Lin, Sida Peng, Hujun Bao, Xiaowei Zhou
NeurIPS 2019.
[Paper][Supplementary Material][Project Page][Code]
PVNet: Pixel-wise Voting Network for 6DoF Estimation
Sida Peng*, Yuan Liu*, Qixing Huang, Xiaowei Zhou, Hujun Bao
CVPR 2019 Oral presentation.
[Paper][Supplementary Material][Project Page][Code]

Honors

Services

Teaching

Teaching Assistant