research-article

Hierarchical and view-invariant light field segmentation by maximizing entropy rate on 4D ray graphs

Authors:

Wolfgang HeidrichAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 38, Issue 6

Article No.: 167, Pages 1 - 15

https://doi.org/10.1145/3355089.3356521

Published: 08 November 2019 Publication History

Abstract

Image segmentation is an important first step of many image processing, computer graphics, and computer vision pipelines. Unfortunately, it remains difficult to automatically and robustly segment cluttered scenes, or scenes in which multiple objects have similar color and texture. In these scenarios, light fields offer much richer cues that can be used efficiently to drastically improve the quality and robustness of segmentations.

In this paper we introduce a new light field segmentation method that respects texture appearance, depth consistency, as well as occlusion, and creates well-shaped segments that are robust under view point changes. Furthermore, our segmentation is hierarchical, i.e. with a single optimization, a whole hierarchy of segmentations with different numbers of regions is available. All this is achieved with a submodular objective function that allows for efficient greedy optimization. Finally, we introduce a new tree-array type data structure, i.e. a disjoint tree, to efficiently perform submodular optimization on very large graphs. This approach is of interest beyond our specific application of light field segmentation.

We demonstrate the efficacy of our method on a number of synthetic and real data sets, and show how the obtained segmentations can be used for applications in image processing and graphics.

Supplementary Material

ZIP File (a167-li.zip)

Supplemental files.

Download
465.49 MB

MP4 File (a167-li.mp4)

Download
148.27 MB

References

[1]

Radhakrishna Achanta, Pablo Marquez Neila, Pascal Fua, and Sabine Süsstrunk. 2018. Scale-Adaptive Superpixels. Color and Imaging Conference Final Program and Proceedings (2018).

[2]

Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. 2012. SLIC Superpixels Compared to State-of-the-Art Superpixel Methods. PAMI (2012).

[3]

Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. 2011. Contour Detection and Hierarchical Image Segmentation. PAMI (2011).

[4]

Alper Ayvaci and Stefano Soatto. 2012. Detachable Object Detection: Segmentation and Depth Ordering from Short-Baseline Video. PAMI (2012).

[5]

Sai Bi, Xiaoguang Han, and Yizhou Yu. 2015. An L1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition. TOG (2015).

[6]

Jiansheng Chen, Zhengqin Li, and Bo Huang. 2017. Linear Spectral Clustering Superpixel. TIP (2017).

[7]

Jingchun Cheng, Yi-Hsuan Tsai, Shengjin Wang, and Ming-Hsuan Yang. 2017. SegFlow: Joint Learning for Video Object Segmentation and Optical Flow. In ICCV.

[8]

Thomas M. Cover and Joy A. Thomas. 2006. Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing).

Digital Library

[9]

Donald G. Dansereau, Bernd Girod, and Gordon Wetzstein. 2019. LiFF: Light Field Features in Scale and Depth. In CVPR.

[10]

Donald G. Dansereau, Ian Mahon, Oscar Pizarro, and Stefan B Williams. 2011. Plenoptic flow: Closed-form visual odometry for light field cameras. In IROS.

[11]

Piotr Dollár and C Lawrence Zitnick. 2015. Fast edge detection using structured forests. PAMI (2015).

[12]

Pedro F. Felzenszwalb and Daniel P. Huttenlocher. 2004. Efficient Graph-Based Image Segmentation. IJCV (2004).

[13]

Brian Fulkerson, Andrea Vedaldi, and Stefano Soatto. 2009. Class segmentation and object localization with neighborhoods. In ICCV. 670--677.

[14]

Matthias Grundmann, Vivek Kwatra, Mei Han, and Irfan Essa. 2010. Efficient Hierarchical Graph Based Video Segmentation. In CVPR.

[15]

Matthieu Hog, Neus Sabater, and Christine Guillemot. 2016. Light Field Segmentation Using a Ray-Based Graph Structure. In ECCV.

[16]

Matthieu Hog, Neus Sabater, and Christine Guillemot. 2017. Super-rays for Efficient Light Field Processing. IEEE Journal of Selected Topics in Signal Processing (2017).

[17]

Katrin Honauer, Ole Johannsen, Daniel Kondermann, and Bastian Goldlücke. 2016. A Dataset and Evaluation Methodology for Depth Estimation on 4D Light Fields. In ACCV.

[18]

Aaron Isaksen, Leonard McMillan, and Steven J Gortler. 2000. Dynamically reparameterized light fields. In SIGGRAPH.

[19]

Naeemullah Khan, Marei Algarni, Anthony Yezzi, and Ganesh Sundaramoorthi. 2015. Shape-tailored local descriptors and their application to segmentation and tracking. In CVPR.

[20]

Naeemullah Khan, Byung-Woo Hong, Anthony Yezzi, and Ganesh Sundaramoorthi. 2017. Coarse-to-Fine Segmentation with Shape-Tailored Continuum Scale Spaces. In CVPR.

[21]

Andreas Krause and Daniel Golovin. 2014. Submodular function maximization.

[22]

Jure Leskovec, Andreas Krause, Carlos Guestrin, Christos Faloutsos, Jeanne Van-Briesen, and Natalie Glance. 2007. Cost-effective outbreak detection in networks. In SIGKDD.

[23]

Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In SIGGRAPH.

[24]

Nianyi Li, Jinwei Ye, Yu Ji, Haibin Ling, and Jingyi Yu. 2014. Saliency detection on light field. In CVPR.

[25]

Ming-Yu Liu, Oncel Tuzel, Srikumar Ramalingam, and Rama Chellappa. 2011. Entropy rate superpixel segmentation. In CVPR.

[26]

Ming-Yu Liu, Oncel Tuzel, Srikumar Ramalingam, and Rama Chellappa. 2014. Entropyrate clustering: Cluster analysis via maximizing a submodular function subject to a matroid constraint. PAMI (2014).

[27]

Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional networks for semantic segmentation. In CVPR.

[28]

Ehsan Miandji, Saghi Hajisharif, and Jonas Unger. 2019. A Unified Framework for Compression and Compressed Sensing of Light Fields and Light Field Videos. TOG (2019).

[29]

Hajime Mihara, Takuya Funatomi, Kenichiro Tanaka, Hiroyuki Kubo, Yasuhiro Mukaigawa, and Hajime Nagahara. 2016. 4D light field segmentation with spatial and angular consistencies. In ICCP.

[30]

Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, and Pat Hanrahan. 2005. Light field photography with a hand-held plenoptic camera. (2005).

[31]

Sylvain Paris and Frédo Durand. 2007. A Topological Approach to Hierarchical Segmentation using Mean Shift. In ICCV.

[32]

Zhao Pei, Yanning Zhang, Tao Yang, Xiuwei Zhang, and Yee-Hong Yang. 2012. A novel multi-object detection method in complex scene using synthetic aperture imaging. Pattern Recognition (2012).

[33]

David Stutz, Alexander Hermans, and Bastian Leibe. 2018. Superpixels: An evaluation of the state-of-the-art. CVIU (2018).

[34]

Michael W Tao, Sunil Hadap, Jitendra Malik, and Ravi Ramamoorthi. 2013. Depth from combining defocus and correspondence using light-field cameras. In ICCV.

[35]

Michael Van den Bergh, Xavier Boix, Gemma Roig, Benjamin de Capitani, and Luc Van Gool. 2012. Seeds: Superpixels extracted via energy-driven sampling. In ECCV.

[36]

Kartik Venkataraman, Dan Lelescu, Jacques Duparré, Andrew McMahon, Gabriel Molina, Priyam Chatterjee, Robert Mullis, and Shree Nayar. 2013. Picam: An ultra-thin high performance monolithic camera array. TOG (2013).

Digital Library

[37]

Ting-Chun Wang, Alexei A Efros, and Ravi Ramamoorthi. 2015a. Occlusion-Aware Depth Estimation Using Light-Field Cameras. In ICCV.

[38]

Wenguan Wang, Jianbing Shen, and F. Porikli. 2015b. Saliency-aware geodesic video object segmentation. In CVPR.

[39]

Sven Wanner, Christoph Straehle, and Bastian Goldluecke. 2013. Globally Consistent Multi-label Assignment on the Ray Space of 4D Light Fields. In CVPR.

[40]

Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, and Marc Levoy. 2005. High performance imaging using large camera arrays. TOG (2005).

[41]

Gaochang Wu, Belen Masia, Adrian Jarabo, Yuchen Zhang, Liangyong Wang, Qionghai Dai, Tianyou Chai, and Yebin Liu. 2017. Light Field Image Processing: An Overview. IEEE Journal of Selected Topics in Signal Processing (2017).

[42]

Yichao Xu, Hajime Nagahara, Atsushi Shimada, and Rin-ichiro Taniguchi. 2015. TransCut: Transparent Object Segmentation From a Light-Field Image. In ICCV.

[43]

Kaan Yücer, Alexander Sorkine-Hornung, Oliver Wang, and Olga Sorkine-Hornung. 2016. Efficient 3D Object Segmentation from Densely Sampled Light Fields with Applications to 3D Reconstruction. TOG (2016).

[44]

Guangming Zang, Mohamed Aly, Ramzi Idoughi, Peter Wonka, and Wolfgang Heidrich. 2018a. Super-Resolution and Sparse View CT Reconstruction. In ECCV.

[45]

Guangming Zang, Ramzi Idouchi, Ran Tao, Gilles Lubineau, Peter Wonka, and Wolfgang Heidrich. 2018b. Space-time tomography for continuously deforming objects. TOG (2018).

[46]

Guangming Zang, Ramzi Idoughi, Ran Tao, Gilles Lubineau, Peter Wonka, and Wolfgang Heidrich. 2019. Warp-and-project Tomography for Rapidly Deforming Objects. TOG (2019).

[47]

Hao Zhu, Qi Zhang, and Qing Wang. 2017. 4D Light Field Superpixel and Segmentation. In CVPR.

Cited By

Cong RSheng HYang DYang DChen RWang SCui Z(2024)End-to-End Semantic Segmentation Utilizing Multi-Scale Baseline Light FieldIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.336737034:7(5790-5804)Online publication date: Jul-2024
https://doi.org/10.1109/TCSVT.2024.3367370
Li YSjöström MKnorr SSikora T(2023)Segmentation-based Initialization for Steered Mixture of Experts2023 IEEE International Conference on Visual Communications and Image Processing (VCIP)10.1109/VCIP59821.2023.10402643(1-5)Online publication date: 4-Dec-2023
https://doi.org/10.1109/VCIP59821.2023.10402643
Hamad MConti CNunes PSoares L(2023)Hyperpixels: Flexible 4D Over-Segmentation for Dense and Sparse Light FieldsIEEE Transactions on Image Processing10.1109/TIP.2023.329052332(3790-3805)Online publication date: 2023
https://doi.org/10.1109/TIP.2023.3290523
Show More Cited By

Index Terms

Hierarchical and view-invariant light field segmentation by maximizing entropy rate on 4D ray graphs
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Computational photography
  2. Computer graphics
    1. Image manipulation
      1. Image processing

Recommendations

Joint segmentation and registration for 4D lung CT images based on Markov random field
ICIMCS '18: Proceedings of the 10th International Conference on Internet Multimedia Computing and Service

The study of segmentation and registration of lung volume in medical images has been an active area with the emergence and development of 4D CT (Computed Tomography) medical imaging techniques. Precise image segmentation and registration methods are ...
Active Contour Based Lung Field Segmentation
IHMSC '09: Proceedings of the 2009 International Conference on Intelligent Human-Machine Systems and Cybernetics - Volume 01

Conventional methods that perform lung segmentation in CT rely on a large contrast in Hounsfield units between the lung and surrounding tissues. But the segmentation of lungs affected by high density pathologies that are connected to the lung border and ...
Entropy-based automatic segmentation of bones in digital X-ray images
PReMI'11: Proceedings of the 4th international conference on Pattern recognition and machine intelligence

Bone image segmentation is an integral component of orthopedic Xray image analysis that aims at extracting the bone structure from the muscles and tissues. Automatic segmentation of the bone part in a digital X-ray image is a challenging problem because ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 38, Issue 6

December 2019

1292 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3355089

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2019

Published in TOG Volume 38, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
235
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)1

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cong RSheng HYang DYang DChen RWang SCui Z(2024)End-to-End Semantic Segmentation Utilizing Multi-Scale Baseline Light FieldIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2024.336737034:7(5790-5804)Online publication date: Jul-2024
https://doi.org/10.1109/TCSVT.2024.3367370
Li YSjöström MKnorr SSikora T(2023)Segmentation-based Initialization for Steered Mixture of Experts2023 IEEE International Conference on Visual Communications and Image Processing (VCIP)10.1109/VCIP59821.2023.10402643(1-5)Online publication date: 4-Dec-2023
https://doi.org/10.1109/VCIP59821.2023.10402643
Hamad MConti CNunes PSoares L(2023)Hyperpixels: Flexible 4D Over-Segmentation for Dense and Sparse Light FieldsIEEE Transactions on Image Processing10.1109/TIP.2023.329052332(3790-3805)Online publication date: 2023
https://doi.org/10.1109/TIP.2023.3290523
Yang XHu FYe LShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Text to SceneProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3478570(2819-2821)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3478570
Nguyen DNepal SKanhere SShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Diverse Multimedia Layout Generation with Multi Choice LearningProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475525(218-226)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475525
Feng HWang YZhou WDeng JLi HShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)DocTr: Document Image Transformer for Geometric Unwarping and Illumination CorrectionProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475388(273-281)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475388
Zhang SLi YHe YYang YZhang SShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)MageAdd: Real-Time Interaction Simulation for Scene SynthesisProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475194(965-973)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475194
Zhu SXu ZSun TKuznetsov AMeyer MJensen HSu HRamamoorthi R(2021)Hierarchical neural reconstruction for path guiding using hybrid path and photon samplesACM Transactions on Graphics10.1145/3450626.345981040:4(1-16)Online publication date: 19-Jul-2021
https://dl.acm.org/doi/10.1145/3450626.3459810
Li MKaufman DJiang C(2021)Codimensional incremental potential contactACM Transactions on Graphics10.1145/3450626.345976740:4(1-24)Online publication date: 19-Jul-2021
https://dl.acm.org/doi/10.1145/3450626.3459767
Hamad MConti CNunes PSoares L(2021)ALFO: Adaptive Light Field Over-SegmentationIEEE Access10.1109/ACCESS.2021.31143249(131147-131165)Online publication date: 2021
https://doi.org/10.1109/ACCESS.2021.3114324
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents