research-article

QuickPose: Real-time Multi-view Multi-person Pose Estimation in Crowded Scenes

Authors:

Xiaowei ZhouAuthors Info & Claims

SIGGRAPH '22: ACM SIGGRAPH 2022 Conference Proceedings

Article No.: 45, Pages 1 - 9

https://doi.org/10.1145/3528233.3530746

Published: 24 July 2022 Publication History

Abstract

This work proposes a real-time algorithm for reconstructing 3D human poses in crowded scenes from multiple calibrated views. The key challenge of this problem is to efficiently match 2D observations across multiple views. Previous methods perform multi-view matching either at the full-body level, which is sensitive to 2D pose estimation error, or at the part level, which ignores 2D constraints between different types of body parts in the same view. Instead, our approach reasons about all plausible skeleton proposals during multi-view matching, where each skeleton may consist of an arbitrary number of parts instead of being a whole body or a single part. To this end, we formulate the multi-view matching problem as mode seeking in the space of skeleton proposals and develop an efficient algorithm named QuickPose to solve the problem, which enables real-time motion capture in crowded scenes. Experiments show that the proposed algorithm achieves the state-of-the-art performance in terms of both speed and accuracy on public datasets.

Supplementary Material

Supplemental file (supp.pdf)

Download
8.07 MB

References

[1]

Vasileios Belagiannis, Sikandar Amin, Mykhaylo Andriluka, Bernt Schiele, Nassir Navab, and Slobodan Ilic. 2014a. 3D Pictorial Structures for Multiple Human Pose Estimation. In CVPR.

[2]

Vasileios Belagiannis, Sikandar Amin, Mykhaylo Andriluka, Bernt Schiele, Nassir Navab, and Slobodan Ilic. 2016. 3D Pictorial Structures Revisited: Multiple Human Pose Estimation. TPAMI (2016).

[3]

Vasileios Belagiannis, Xinchao Wang, Bernt Schiele, Pascal Fua, Slobodan Ilic, and Nassir Navab. 2014b. Multiple Human Pose Estimation with Temporally Consistent 3D Pictorial Structures. In ECCVW.

[4]

Lewis Bridgeman, Marco Volino, Jean-Yves Guillemaut, and Adrian Hilton. 2019. Multi-Person 3D Pose Estimation and Tracking in Sports. In CVPRW.

[5]

Zhe Cao, Gines Hidalgo Martinez, Tomas Simon, Shih-En Wei, and Yaser A. Sheikh. 2019. OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. TPAMI (2019).

[6]

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields. In CVPR.

[7]

He Chen, Pengfei Guo, Pengfei Li, Gim Hee Lee, and Gregory Chirikjian. 2020b. Multi-person 3D Pose Estimation in Crowded Scenes Based on Multi-view Geometry. In ECCV.

[8]

Long Chen, Haizhou Ai, Rui Chen, Zijie Zhuang, and Shuang Liu. 2020a. Cross-View Tracking for Multi-Human 3D Pose Estimation at Over 100 FPS. In CVPR.

[9]

Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun. 2018. Cascaded Pyramid Network for Multi-person Pose Estimation. In CVPR.

[10]

Dorin Comaniciu and Peter Meer. 2002. Mean shift: A robust approach toward feature space analysis. TPAMI (2002).

[11]

Junting Dong, Wen Jiang, Qixing Huang, Hujun Bao, and Xiaowei Zhou. 2019. Fast and Robust Multi-Person 3D Pose Estimation From Multiple Views. In CVPR.

[12]

Zijian Dong, Jie Song, Xu Chen, Chen Guo, and Otmar Hilliges. 2021. Shape-aware Multi-Person Pose Estimation from Multi-View Images. In ICCV.

[13]

Sara Ershadi-Nasab, Erfan Noury, Shohreh Kasaei, and Esmaeil Sanaei. 2018. Multiple human 3D pose estimation from multiview images. Multimedia Tools and Applications(2018).

[14]

Keinosuke Fukunaga and Larry D. Hostetler. 1975. The Estimation of the Gradient of a Density Function, with Applications in Pattern Recognition. IEEE Transactions on Information Theory(1975).

Digital Library

[15]

Congzhentao Huang, Shuai Jiang, Yang Li, Ziyue Zhang, Jason Traish, Chen Deng, Sam Ferguson, and Richard Yi Da Xu. 2020. End-to-end Dynamic Matching Network for Multi-view Multi-person 3D Pose Estimation. In ECCV.

[16]

Eldar Insafutdinov, Leonid Pishchulin, Bjoern Andres, Mykhaylo Andriluka, and Bernt Schiele. 2016. DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model. In ECCV.

[17]

Karim Iskakov, Egor Burkov, Victor Lempitsky, and Yury Malkov. 2019. Learnable Triangulation of Human Pose. In ICCV.

[18]

Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. 2015. Panoptic Studio: A Massively Multiview System for Social Motion Capture. In ICCV.

[19]

Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh. 2019. Panoptic Studio: A Massively Multiview System for Social Interaction Capture. TPAMI (2019).

[20]

Abdolrahim Kadkhodamohammadi and Nicolas Padoy. 2021. A generalizable approach for multi-view 3D human pose regression. Machine Vision and Applications(2021).

[21]

Muhammed Kocabas, Salih Karagoz, and Emre Akbas. 2018. MultiPoseNet: Fast Multi-Person Pose Estimation Using Pose Residual Network. In ECCV.

[22]

Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu. 2019. CrowdPose: Efficient Crowded Scenes Pose Estimation and a New Benchmark. In CVPR.

[23]

Jiahao Lin and Gim Hee Lee. 2021. Multi-View Multi-Person 3D Pose Estimation with Plane Sweep Stereo. In CVPR.

[24]

Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Srinath Sridhar, Gerard Pons-Moll, and Christian Theobalt. 2018. Single-Shot Multi-person 3D Pose Estimation from Monocular RGB. In 3DV.

[25]

Alejandro Newell, Kaiyu Yang, and Jia Deng. 2016. Stacked hourglass networks for human pose estimation. In ECCV.

[26]

Takuya Ohashi, Yosuke Ikegami, and Yoshihiko Nakamura. 2020. Synergetic reconstruction from 2D pose and 3D motion for wide-space multi-person video motion capture in the wild. Image and Vision Computing(2020).

[27]

George Papandreou, Tyler Zhu, Liang Chieh Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy. 2018. Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model. In ECCV.

[28]

Emanuel Parzen. 1962. On Estimation of a Probability Density Function and Mode. The Annals of Mathematical Statistics(1962).

[29]

Leonid Pishchulin, Eldar Insafutdinov, Siyu Tang, Bjoern Andres, Mykhaylo Andriluka, Peter Gehler, and Bernt Schiele. 2016. DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation. In CVPR.

[30]

N Dinesh Reddy, Laurent Guigues, Leonid Pishchulin, Jayan Eledath, and Srinivasa G. Narasimhan. 2021. TesseTrack: End-to-End Learnable Multi-Person Articulated 3D Pose Tracking. In CVPR.

[31]

Murray Rosenblatt. 1956. Remarks on Some Nonparametric Estimates of a Density Function. The Annals of Mathematical Statistics(1956).

[32]

Jamie Shotton, Andrew Fitzgibbon, Mat Cook, Toby Sharp, Mark Finocchio, Richard Moore, Alex Kipman, and Andrew Blake. 2011. Real-time human pose recognition in parts from single depth images. In CVPR.

[33]

Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. 2019. Deep High-Resolution Representation Learning for Human Pose Estimation. In CVPR.

[34]

Julian Tanke and Juergen Gall. 2019. Iterative Greedy Matching for 3D Human Pose Tracking from Multiple Views. In GCPR.

[35]

Roberto Tron, Xiaowei Zhou, Carlos Esteves, and Kostas Daniilidis. 2017. Fast Multi-image Matching via Density-Based Clustering. In ICCV.

[36]

Hanyue Tu, Chunyu Wang, and Wenjun Zeng. 2020. VoxelPose: Towards Multi-camera 3D Human Pose Estimation in Wild Environment. In ECCV.

[37]

Han Vanholder. 2016. Efficient Inference with TensorRT. NVIDIA.

[38]

Andrea Vedaldi and Stefano Soatto. 2008. Quick Shift and Kernel Methods for Mode Seeking. In ECCV.

[39]

Tao Wang, Jianfeng Zhang, Yujun Cai, Shuicheng Yan, and Jiashi Feng. 2021. Direct Multi-view Multi-person 3D Pose Estimation. In NeurIPS.

[40]

Size Wu, Sheng Jin, Wentao Liu, Lei Bai, Chen Qian, Dong Liu, and Wanli Ouyang. 2021. Graph-Based 3D Multi-Person Pose Estimation Using Multi-View Images. In ICCV.

[41]

Bin Xiao, Haiping Wu, and Yichen Wei. 2018. Simple Baselines for Human Pose Estimation and Tracking. In ECCV.

[42]

Andrei Zanfir, Elisabeta Marinoiu, and Cristian Sminchisescu. 2018. Monocular 3D Pose and Shape Estimation of Multiple People in Natural Scenes: The Importance of Multiple Scene Constraints. In CVPR.

[43]

Yuxiang Zhang, Liang An, Tao Yu, Xiu Li, Kun Li, and Yebin Liu. 2020. 4D Association Graph for Realtime Multi-Person Motion Capture Using Multiple Video Cameras. In CVPR.

[44]

Yuxiang Zhang, Zhe Li, Liang An, Mengcheng Li, Tao Yu, and Yebin Liu. 2021. Lightweight Multi-person Total Motion Capture Using Sparse Multi-view Cameras. In ICCV.

[45]

Jianan Zhen, Qi Fang, Jiaming Sun, Wentao Liu, Wei Jiang, Hujun Bao, and Xiaowei Zhou. 2020. SMAP: Single-Shot Multi-person Absolute 3D Pose Estimation. In ECCV.

Cited By

Yang YWang Y(2024)Real-Time Multi-View Human Pose Estimation System2024 39th Youth Academic Annual Conference of Chinese Association of Automation (YAC)10.1109/YAC63405.2024.10598814(2181-2186)Online publication date: 7-Jun-2024
https://doi.org/10.1109/YAC63405.2024.10598814
Niu ZLu KXue JQin XWang JShao L(2024)From Methods to Applications: A Review of Deep 3D Human Motion CaptureIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.342341134:11(11340-11359)Online publication date: Nov-2024
https://doi.org/10.1109/TCSVT.2024.3423411
Jiang JChen J(2024)Exploring Latent Cross-Channel Embedding for Accurate 3d Human Pose Reconstruction in a Diffusion FrameworkICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10448487(7870-7874)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10448487
Show More Cited By

Recommendations

Explicit Occlusion Reasoning for Multi-person 3D Human Pose Estimation
Computer Vision – ECCV 2022
Abstract
Occlusion poses a great threat to monocular multi-person 3D human pose estimation due to large variability in terms of the shape, appearance, and position of occluders. While existing methods try to handle occlusion with pose priors/constraints, ...
Uncalibrated multi-view multiple humans association and 3D pose estimation by adversarial learning
Abstract
Multiple human 3D pose estimation is a useful but challenging task in computer vison applications. The ambiguities in estimation of 2D and 3D poses of multiple persons can be verified by using multi-view frames, in which the occluded or self-...
Multi-person 3D Pose Estimation in Crowded Scenes Based on Multi-view Geometry
Computer Vision – ECCV 2020
Abstract
Epipolar constraints are at the core of feature matching and depth estimation in current multi-person multi-camera 3D human pose estimation methods. Despite the satisfactory performance of this formulation in sparser crowd scenes, its ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '22: ACM SIGGRAPH 2022 Conference Proceedings

July 2022

553 pages

ISBN:9781450393379

DOI:10.1145/3528233

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

NSFC

Conference

SIGGRAPH '22

Sponsor:

SIGGRAPH

SIGGRAPH '22: Special Interest Group on Computer Graphics and Interactive Techniques Conference

August 7 - 11, 2022

BC, Vancouver, Canada

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
841
Total Downloads

Downloads (Last 12 months)119
Downloads (Last 6 weeks)11

Reflects downloads up to 22 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang YWang Y(2024)Real-Time Multi-View Human Pose Estimation System2024 39th Youth Academic Annual Conference of Chinese Association of Automation (YAC)10.1109/YAC63405.2024.10598814(2181-2186)Online publication date: 7-Jun-2024
https://doi.org/10.1109/YAC63405.2024.10598814
Niu ZLu KXue JQin XWang JShao L(2024)From Methods to Applications: A Review of Deep 3D Human Motion CaptureIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.342341134:11(11340-11359)Online publication date: Nov-2024
https://doi.org/10.1109/TCSVT.2024.3423411
Jiang JChen J(2024)Exploring Latent Cross-Channel Embedding for Accurate 3d Human Pose Reconstruction in a Diffusion FrameworkICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10448487(7870-7874)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10448487
Niu ZLu KXue JWang J(2024)Skeleton Cluster Tracking for robust multi-view multi-person 3D human pose estimationComputer Vision and Image Understanding10.1016/j.cviu.2024.104059246(104059)Online publication date: Sep-2024
https://doi.org/10.1016/j.cviu.2024.104059
Lu FDong ZSong JHilliges O(2024)AvatarPose: Avatar-Guided 3D Pose Estimation of Close Human Interaction from Sparse Multi-view VideosComputer Vision – ECCV 202410.1007/978-3-031-73668-1_13(215-233)Online publication date: 1-Dec-2024
https://doi.org/10.1007/978-3-031-73668-1_13
Jiang TBillingham JMüksch SZarate JEvans NOswald MPolleyfeys MHilliges OKaufmann MSong J(2024)WorldPose: A World Cup Dataset for Global 3D Human Pose EstimationComputer Vision – ECCV 202410.1007/978-3-031-72655-2_20(343-362)Online publication date: 6-Dec-2024
https://doi.org/10.1007/978-3-031-72655-2_20
Chen JGuo YLiu ZTong MZuo MXiao K(2024)3D Pose Measurement of All Students Using Existing Cameras in the Smart Classroom: A Pilot StudyAdvances in Haptics and Virtual Reality10.1007/978-3-031-56521-2_16(193-201)Online publication date: 19-May-2024
https://doi.org/10.1007/978-3-031-56521-2_16
Shuai QYu ZZhou ZFan LYang HYang CZhou X(2023)Reconstructing Close Human Interactions from Multiple ViewsACM Transactions on Graphics10.1145/361833642:6(1-14)Online publication date: 5-Dec-2023
https://doi.org/10.1145/3618336

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents