research-article

Public Access

Understanding and Exploiting Object Interaction Landscapes

Authors:

Suren Deepak Rajasekaran,

Yusuke Yoshiyasu,

Leonidas J. GuibasAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 36, Issue 3

Article No.: 31, Pages 1 - 14

https://doi.org/10.1145/3083725

Published: 27 June 2017 Publication History

Abstract

Interactions play a key role in understanding objects and scenes for both virtual and real-world agents. We introduce a new general representation for proximal interactions among physical objects that is agnostic to the type of objects or interaction involved. The representation is based on tracking particles on one of the participating objects and then observing them with sensors appropriately placed in the interaction volume or on the interaction surfaces. We show how to factorize these interaction descriptors and project them into a particular participating object so as to obtain a new functional descriptor for that object, its interaction landscape, capturing its observed use in a spatiotemporal framework. Interaction landscapes are independent of the particular interaction and capture subtle dynamic effects in how objects move and behave when in functional use. Our method relates objects based on their function, establishes correspondences between shapes based on functional key points and regions, and retrieves peer and partner objects with respect to an interaction.

Supplementary Material

JPG File (tog-31.jpg)

Download
18.66 KB

pirk (pirk.zip)

Supplemental movie, appendix, image and software files for, Understanding and Exploiting Object Interaction Landscapes

Download

MP4 File (tog-31.mp4)

Download
439.54 MB

References

[1]

Adobe. 2015. Mixamo. Retrieved from https://www.mixamo.com/.

[2]

R. Ali Al-Asqhar, T. Komura, and M. Geol Choi. 2013. Relationship descriptors for interactive motion adaptation. In Proceedings of ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA’13). ACM, 45--53.

Digital Library

[3]

N. Amenta, M. Bern, and M. Kamvysselis. 1998. A new Voronoi-based surface reconstruction algorithm. In Proceedings of SIGGRAPH. ACM, New York, 415--421.

Digital Library

[4]

D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis. 2005. SCAPE: Shape completion and animation of people. In Proceedings of SIGGRAPH.

Digital Library

[5]

F. G. Ashby. 1992. Multidimensional Models of Perception and Cognition. L. Erlbaum. https://books.google.com/books?id=eeQh4ESeOfsC

[6]

E. Bar-Aviv and E. Rivlin. 2006. Functional 3D object classification using simulation of embodied agent. In BMVC. British Machine Vision Association, 307--316.

[7]

I. Baran and J. Popović. 2007. Automatic rigging and animation of 3D characters. ACM Trans. Graph. 26, 3 (July 2007), 72:1--72:8.

Digital Library

[8]

S. Biasotti, A. Cerri, A. Bronstein, and M. Bronstein. 2015. Recent trends, applications, and perspectives in 3D shape similarity assessment. Comp. Graph. Forum (2015).

Digital Library

[9]

A. W. Black and P. Taylor. 1997. Automatically clustering similar units for unit selection in speech synthesis. In Eurospeech97. 601--604.

[10]

M. Caine. 1994. The design of shape interactions using motion constraints. In Proceedings of the 1994 IEEE International Conference on Robotics and Automation, Vol. 1. 366--371.

[11]

A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. 2015. ShapeNet: An information-rich 3D model repository. ArXiv e-prints (Dec. 2015).

[12]

Y.-W. Chao, Z. Wang, Y. He, J. Wang, and J. Deng. 2015. HICO: A benchmark for recognizing human-object interactions in images. ICCV (2015).

Digital Library

[13]

D.-Y. Chen, X.-P. Tian, Y.-T. Shen, and M. Ouhyoung. 2003. On visual similarity based 3D model retrieval. Comp. Graph. Forum 22, 3 (2003), 223--232.

[14]

J. Chen, X. Ge, L.-Y. Wei, B. Wang, Y. Wang, H. Wang, Y. Fei, K.-L. Qian, J.-H. Yong, and W. Wang. 2013. Bilateral blue noise sampling. ACM Trans. Graph. 32, 6, Article 216 (Nov. 2013), 11 pages.

Digital Library

[15]

S. Durrleman. 2010. Statistical Models of Currents for Measuring the Variability of Anatomical Curves, Surfaces and Their Evolution. Ph.D. dissertation. Université Nice - Sophia Antipolis, France.

[16]

M. Fisher, M. Savva, Y. Li, P. Hanrahan, and M. Niessner. 2015. Activity-centric scene synthesis for functional 3D scene modeling. ACM Trans. Graph. 34, 6, Article 179 (Oct. 2015), 13 pages.

Digital Library

[17]

R. Gal and D. Cohen-Or. 2006. Salient geometric features for partial shape matching and similarity. ACM Trans. Graph. 25, 1 (2006), 130--150.

Digital Library

[18]

H. Grabner, J. Gall, and L. Van Gool. 2011. What makes a chair a chair?. In CVPR. 1529--1536.

Digital Library

[19]

A. Gupta, A. Kembhavi, and L. S. Davis. 2009. Observing human-object interactions: Using spatial and functional compatibility for recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31, 10 (Oct. 2009), 1775--1789.

Digital Library

[20]

E. S. L. Ho, T. Komura, and C.-L. Tai. 2010. Spatial relationship preserving character motion adaptation. ACM Trans. Graph. 29, 4, Article 33 (2010), 8 pages.

Digital Library

[21]

R. Hu, O. van Kaick, B. Wu, H. Huang, A. Shamir, and H. Zhang. 2016. Learning how objects function via co-analysis of interactions. ACM Trans. Graph. 35, 4, Article 47 (July 2016), 13 pages.

Digital Library

[22]

R. Hu, C. Zhu, O. van Kaick, L. Liu, A. Shamir, and H. Zhang. 2015. Interaction context (ICON): Towards a geometric functionality descriptor. ACM Trans. Graph. 34, 4, Article 83 (2015), 12 pages.

Digital Library

[23]

H. Jiang and D. R. Martin. 2008. Finding actions using shape flows. In Computer Vision ECCV 2008, David Forsyth, Philip Torr, and Andrew Zisserman (Eds.). Lecture Notes in Computer Science, Vol. 5303. Springer, Berlin, 278--292.

Digital Library

[24]

Germany Karlsruhe Institute of Technology, Karlsruhe. 2015. KIT Whole Human Motion Database. Retrieved from https://motion-database.humanoids.kit.edu/.

[25]

V. G. Kim, S. Chaudhuri, L. Guibas, and T. Funkhouser. 2014. Shape2Pose: Human-centric shape analysis. ACM Trans. Graph. 33, 4, Article 120 (2014), 12 pages.

Digital Library

[26]

H. Laga, M. Mortara, and M. Spagnuolo. 2013. Geometry and context for semantic correspondences and functionality recognition in man-made 3D shapes. ACM Trans. Graph. 32, 5, Article 150 (2013), 16 pages.

Digital Library

[27]

C. H. Lee, A. Varshney, and D. W. Jacobs. 2005. Mesh saliency. ACM Trans. Graph. 24, 3 (July 2005), 659--666.

Digital Library

[28]

M. Leordeanu and M. Hebert. 2005. A spectral technique for correspondence problems using pairwise constraints. In Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV’05). Vol. 2. 1482--1489.

Digital Library

[29]

P. Li, B. Wang, F. Sun, X. Guo, C. Zhang, and W. Wang. 2015. Q-MAT: Computing medial axis transform by quadratic error minimization. ACM Trans. Graph. 35, 1, Article 8 (Dec. 2015), 16 pages.

Digital Library

[30]

Y. Li, J. L. Fu, and N. S. Pollard. 2007. Data-driven grasp synthesis using shape matching and task-based pruning. IEEE Trans. Vis. Comput. Graph. 13, 4 (July 2007), 732--747.

Digital Library

[31]

Z. Liu, C. Xie, S. Bu, X. Wang, J. Han, H. Lin, and H. Zhang. 2015. Indirect shape analysis for 3D shape retrieval. Comput. Graphics 46 (2015), 110--116. Shape Modeling International 2014.

Digital Library

[32]

N. Mitra, M. Wand, H. (Richard) Zhang, D. Cohen-Or, V. Kim, and Q.-X. Huang. 2013b. Structure-aware shape processing. In SIGGRAPH Asia 2013 Courses (SA’13). ACM, New York, Article 1, 20 pages.

Digital Library

[33]

N. J. Mitra, M. Pauly, M. Wand, and D. Ceylan. 2013a. Symmetry in 3D geometry: Extraction and applications. Comput. Graph. Forum 32, 6 (2013), 1--23.

Digital Library

[34]

J. J. Monaghan. 1992. Smoothed particle hydrodynamics. Ann. Rev. Astron. Astrophys. 30 (1992), 543--574.

[35]

O. Oreifej and Z. Liu. 2013. HON4D: Histogram of oriented 4D normals for activity recognition from depth sequences. In 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’13). 716--723.

Digital Library

[36]

R. Osada, T. Funkhouser, B. Chazelle, and D. Dobkin. 2002. Shape distributions. ACM Trans. Graph. 21, 4 (Oct. 2002), 807--832.

Digital Library

[37]

M. Ovsjanikov, M. Ben-Chen, J. Solomon, A. Butscher, and L. Guibas. 2012. Functional maps: A flexible representation of maps between shapes. ACM Trans. Graph. 31, 4 (2012).

Digital Library

[38]

M. Pechuk, O. Soldea, and E. Rivlin. 2008. Learning function-based object classification from 3D imagery. Comput. Vis. Image Underst. 110, 2 (May 2008), 173--191.

Digital Library

[39]

E. Rivlin, S. J. Dickinson, and A, Rosenfeld. 1994. Recognition by functional parts {function-based object recognition}. In Proceedings of the 1994 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’94). 267--274.

[40]

Columbia University USA Robotics Lab, Computer Science Laboratory. 2012. GraspIt! (2012). http://www.cs.columbia.edu/ cmatei/graspit/.

[41]

Y. Rubner, C. Tomasi, and L. J. Guibas. 1998. A metric for distributions with applications to image databases. In Proceedings of the 6th International Conference on Computer Vision, 1998. 59--66.

Digital Library

[42]

M. Savva, A. X. Chang, P. Hanrahan, M. Fisher, and M. Nießner. 2014. SceneGrok: Inferring action maps in 3D environments. ACM Trans. Graph. 33, 6, Article 212 (Nov. 2014), 10 pages.

Digital Library

[43]

M. Savva, A. X. Chang, P. Hanrahan, M. Fisher, and M. Nießner. 2016. PiGraphs: Learning interaction snapshots from observations. ACM Trans. Graph. 35, 4, Article 139 (July 2016), 12 pages.

Digital Library

[44]

P. Shilane and T. Funkhouser. 2007. Distinctive regions of 3D surfaces. ACM Trans. Graph. 26, 2, Article 7 (June 2007).

Digital Library

[45]

O. Sidi, O. van Kaick, Y. Kleiman, H. Zhang, and D. Cohen-Or. 2011. Unsupervised co-segmentation of a set of shapes via descriptor-space spectral clustering. ACM Trans. Graph. 30, 6, Article 126 (2011), 10 pages.

Digital Library

[46]

H. O. Song, M. Fritz, C. Gu, and T. Darrell. 2011. Visual grasp affordances from appearance-based cues. In Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops). 998--1005.

[47]

M. Sutton, L. Stark, and K. Bowyer. 1994. GRUFF-3: Generalizing the domain of a function-based recognition system. Pattern Recog. 27, 12 (1994), 1743--1766.

[48]

A. Tevs, Q. Huang, M. Wand, H.-P. Seidel, and L. Guibas. 2014. Relating shapes via geometric symmetries and regularities. ACM Trans. Graph. 33, 4, Article 119 (July 2014), 12 pages.

Digital Library

[49]

A. Tversky. 1977. Features of similarity. Psych. Rev. 84 (1977), 327--352.

[50]

D. Tzionas and J. Gall. 2015. 3D object reconstruction from hand-object interactions. In Proceedings of the International Conference on Computer Vision (ICCV). http://files.is.tue.mpg.de/dtzionas/In-Hand-Scanning

Digital Library

[51]

Z. Wu, S. Song, A. Khosla, X. Tang, and J. Xiao. 2014. 3D shapenets for 2.5d object recognition and next-best-view prediction. CoRR abs/1406.5670 (2014). http://arxiv.org/abs/1406.5670

[52]

L. Xu, H. Quynh Dinh, P. Mordohai, and T. Ramsay. 2011. Detecting patterns in vector fields. 49th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition 6 (2011).

[53]

X. Zhao, H. Wang, and T. Komura. 2014. Indexing 3D scenes using the interaction bisector surface. ACM Trans. Graph. 33, 3, Article 22 (2014), 14 pages.

Digital Library

[54]

Y. Zheng, D. Cohen-Or, and N. J. Mitra. 2013. Smart variations: Functional substructures for part compatibility. Comp. Graph. Forum 32, 2 pt2 (2013), 195--204.

[55]

Y. Zhu, A. Fathi, and L. Fei-Fei. 2014. Reasoning about object affordances in a knowledge base representation. In Computer Vision ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars (Eds.). Lecture Notes in Computer Science, Vol. 8690. Springer International Publishing, 408--424.

Cited By

Taheri OZhou YTzionas DZhou YCeylan DPirk SBlack M(2024)GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistency2024 International Conference on 3D Vision (3DV)10.1109/3DV62453.2024.00064(933-943)Online publication date: 18-Mar-2024
https://doi.org/10.1109/3DV62453.2024.00064
Huang ZDai SXu KZhang HHuang HHu R(2024)DINA: Deformable INteraction AnalogyGraphical Models10.1016/j.gmod.2024.101217133(101217)Online publication date: Jun-2024
https://doi.org/10.1016/j.gmod.2024.101217
Qi ZDong RZhang SGeng HHan CGe ZYi LMa K(2024)ShapeLLM: Universal 3D Object Understanding for Embodied InteractionComputer Vision – ECCV 202410.1007/978-3-031-72775-7_13(214-238)Online publication date: 30-Sep-2024
https://doi.org/10.1007/978-3-031-72775-7_13
Show More Cited By

Index Terms

Understanding and Exploiting Object Interaction Landscapes
1. Computing methodologies

Recommendations

Learning how objects function via co-analysis of interactions

We introduce a co-analysis method which learns a functionality model for an object category, e.g., strollers or backpacks. Like previous works on functionality, we analyze object-to-object interactions and intra-object properties and relations. ...
Understanding and Exploiting Object Interaction Landscapes

Interactions play a key role in understanding objects and scenes for both virtual and real-world agents. We introduce a new general representation for proximal interactions among physical objects that is agnostic to the type of objects or interaction ...
Interaction context (ICON): towards a geometric functionality descriptor

We introduce a contextual descriptor which aims to provide a geometric description of the functionality of a 3D object in the context of a given scene. Differently from previous works, we do not regard functionality as an abstract label or represent it ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 36, Issue 3

June 2017

165 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3087678

Editor:
Kavita Bala
Cornell University

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 June 2017

Accepted: 01 March 2017

Revised: 01 January 2017

Received: 01 September 2016

Published in TOG Volume 36, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

NSF
JSPS Strategic Young Researchers Visits Program for Acceleration Brain Circulations
Stanford AI Lab-Toyota Center for Artificial Intelligence Research
Google Focused Research Award
Max Planck Center for Visual Computing and Communications
National Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

26
Total Citations
View Citations
809
Total Downloads

Downloads (Last 12 months)87
Downloads (Last 6 weeks)14

Reflects downloads up to 01 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Taheri OZhou YTzionas DZhou YCeylan DPirk SBlack M(2024)GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistency2024 International Conference on 3D Vision (3DV)10.1109/3DV62453.2024.00064(933-943)Online publication date: 18-Mar-2024
https://doi.org/10.1109/3DV62453.2024.00064
Huang ZDai SXu KZhang HHuang HHu R(2024)DINA: Deformable INteraction AnalogyGraphical Models10.1016/j.gmod.2024.101217133(101217)Online publication date: Jun-2024
https://doi.org/10.1016/j.gmod.2024.101217
Qi ZDong RZhang SGeng HHan CGe ZYi LMa K(2024)ShapeLLM: Universal 3D Object Understanding for Embodied InteractionComputer Vision – ECCV 202410.1007/978-3-031-72775-7_13(214-238)Online publication date: 30-Sep-2024
https://doi.org/10.1007/978-3-031-72775-7_13
Clever HGrady PTurk GKemp C(2023)BodyPressure - Inferring Body Pose and Contact Pressure From a Depth ImageIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.315890245:1(137-153)Online publication date: 1-Jan-2023
https://doi.org/10.1109/TPAMI.2022.3158902
Huang ZXu JDai SXu KZhang HHuang HHu R(2023)NIFT: Neural Interaction Field and Template for Object Manipulation2023 IEEE International Conference on Robotics and Automation (ICRA)10.1109/ICRA48891.2023.10160666(1875-1881)Online publication date: 29-May-2023
https://doi.org/10.1109/ICRA48891.2023.10160666
Cui JLi SHou FHao AQin H(2023)Analyzing part functionality via multi-modal latent space embedding and interweavingComputers & Graphics10.1016/j.cag.2023.06.031115(1-12)Online publication date: Oct-2023
https://doi.org/10.1016/j.cag.2023.06.031
She QHu RXu JLiu MXu KHuang H(2022)Learning high-DOF reaching-and-grasping via dynamic representation of gripper-object interactionACM Transactions on Graphics10.1145/3528223.353009141:4(1-14)Online publication date: 22-Jul-2022
https://dl.acm.org/doi/10.1145/3528223.3530091
Li CLi WHuang HYu L(2022)Interactive augmented reality storytelling guided by scene semanticsACM Transactions on Graphics10.1145/3528223.353006141:4(1-15)Online publication date: 22-Jul-2022
https://dl.acm.org/doi/10.1145/3528223.3530061
Guan YLiu HLiu KYin KHu Rvan Kaick OZhang YYumer ECarr NMech RZhang H(2022)FAME: 3D Shape Generation via Functionality-Aware Model EvolutionIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2020.302975928:4(1758-1772)Online publication date: 1-Apr-2022
https://doi.org/10.1109/TVCG.2020.3029759
Hong YMo KYi LGuibas LTorralba ATenenbaum JGan C(2022)Fixing Malfunctional Objects With Learned Physical Simulation and Functional Prediction2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52688.2022.00147(1403-1413)Online publication date: Jun-2022
https://doi.org/10.1109/CVPR52688.2022.00147
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents