Pattern Matching for Perception Streams
Authors: 
Jacob Anderson, 
Georgios Fainekos, Bardh Hoxha, Hideki Okamoto, Danil ProkhorovAuthors Info & Claims
Jacob Anderson, Georgios Fainekos, Bardh Hoxha, Hideki Okamoto, Danil ProkhorovAuthors Info & ClaimsPages 251 - 270
Abstract
We introduce Spatial Regular Expressions (SpREs) as a novel querying language for pattern matching over perception streams containing spatial and temporal data. To highlight the capabilities of SpREs, we developed the Strem tool as a matching framework that works in both the offline and online domain. We demonstrate the tool through an offline example with an AV dataset, an online example through an integration with the ROS and CARLA simulators, and an initial set of performance benchmarks on various SpRE queries. From our designed matching framework, we are able to find over 20,000 matches within 296 ms making it highly usable in runtime monitoring applications.
References
[1]
Aho, A.V.: Pattern matching in strings. In: Formal Language Theory, pp. 325–347. Elsevier (1980)
[2]
Aho AV, Lam MS, Sethi R, and Ullman JD Compilers: Principles, Techniques and Tools 2020 Pearson Addison-Wesley
[3]
Alfred, V.: Algorithms for finding patterns in strings. In: Algorithms and Complexity, vol. 1, p. 255 (2014)
[4]
Allen JF Maintaining knowledge about temporal intervals Commun. ACM 1983 26 11 832-843
[5]
Bai Z et al. Cyber mobility mirror: a deep learning-based real-world object perception platform using roadside LiDAR IEEE Trans. Intell. Transp. Syst. 2023 24 9476-9489
[6]
Balakrishnan A, Deshmukh J, Hoxha B, Yamaguchi T, and Fainekos G Feng L and Fisman D PerceMon: online monitoring for perception systems Runtime Verification 2021 Cham Springer 297-308
[7]
Beer I, Ben-David S, Eisner C, Fisman D, Gringauze A, and Rodeh Y Berry G, Comon H, and Finkel A The temporal logic sugar Computer Aided Verification 2001 Heidelberg Springer 363-367
[8]
Boyer RS and Moore JS A fast string searching algorithm Commun. ACM 1977 20 10 762-772
[9]
Caesar, H., et al.: nuScenes: a multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11621–11631 (2020)
[10]
Del Bimbo A, Vicario E, and Zingoni D Symbolic description and visual querying of image sequences using spatio-temporal logic IEEE Trans. Knowl. Data Eng. 1995 7 4 609-622
[11]
Dokhanchi A, Amor HB, Deshmukh JV, and Fainekos G Colombo C and Leucker M Evaluating perception systems for autonomous vehicles using quality temporal logic Runtime Verification 2018 Cham Springer 409-416
[12]
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: Conference on Robot Learning, pp. 1–16. PMLR (2017)
[13]
Everingham M, Van Gool L, Williams CK, Winn J, and Zisserman A The pascal visual object classes (VOC) challenge Int. J. Comput. Vis. 2010 88 303-338
[14]
Fang W et al. Computer vision applications in construction safety assurance Autom. Constr. 2020 110 103013
[15]
Fremont, D.J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Scenic: a language for scenario specification and scene generation. In: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 63–78 (2019)
[16]
Friedl JE Mastering Regular Expressions 2006 Sebastopol O’Reilly Media Inc.
[17]
Gabelaia D, Kontchakov R, Kurucz A, Wolter F, and Zakharyaschev M Combining spatial and temporal logics: expressiveness vs. complexity J. Artif. Intell. Res. 2005 23 167-243
[18]
Gallant, A.: regex-automata (2023). https://github.com/rust-lang/regex
[19]
Hekmatnejad, M., Hoxha, B., Deshmukh, J.V., Yang, Y., Fainekos, G.: Formalizing and evaluating requirements of perception systems for automated vehicles using spatio-temporal perception logic. arXiv preprint arXiv:2206.14372 (2022)
[20]
Janai, J., Güney, F., Behl, A., Geiger, A., et al.: Computer vision for autonomous vehicles: problems, datasets and state of the art. Found. Trends® Comput. Graph. Vis. 12(1–3), 1–308 (2020)
[21]
Kapach K, Barnea E, Mairon R, Edan Y, and Ben-Shahar O Computer vision for fruit harvesting robots-state of the art and challenges ahead Int. J. Comput. Vis. Robot. 2012 3 1–2 4-34
[22]
Kesten, R., et al.: Woven planet perception dataset 2020 (2019). https://woven.toyota/en/perception-dataset
[23]
Kim, E., et al.: Querying labelled data with scenario programs for sim-to-real validation. In: 2022 ACM/IEEE 13th International Conference on Cyber-Physical Systems (ICCPS), pp. 34–45. IEEE (2022)
[24]
Knuth DE, Morris JH Jr, and Pratt VR Fast pattern matching in strings SIAM J. Comput. 1977 6 2 323-350
[25]
Kontchakov, R., Kurucz, A., Wolter, F., Zakharyaschev, M.: Spatial logic+ temporal logic=?. In: Handbook of Spatial Logics, pp. 497–564 (2007)
[26]
Lin T-Y et al. Fleet D, Pajdla T, Schiele B, Tuytelaars T, et al. Microsoft COCO: common objects in context Computer Vision – ECCV 2014 2014 Cham Springer 740-755
[27]
Lu, D., et al.: CAROM air-vehicle localization and traffic scene reconstruction from aerial videos. arXiv preprint arXiv:2306.00075 (2023)
[28]
Matsakis ND and Klock FS The rust language ACM SIGAda Ada Lett. 2014 34 3 103-104
[29]
Meng T, Huang J, Chew CM, Yang D, and Zhong Z Configuration and design schemes of environmental sensing and vehicle computing systems for automated driving: a review IEEE Sens. J. 2023 23 15305-15320
[30]
Pitropov M et al. Canadian adverse driving conditions dataset Int. J. Robot. Res. 2021 40 4–5 681-690
[31]
Pnueli, A.: The temporal logic of programs. In: 18th Annual Symposium on Foundations of Computer Science (SFCS 1977), pp. 46–57. IEEE (1977)
[32]
Quigley, M., et al.: ROS: an open-source robot operating system. In: ICRA Workshop on Open Source Software, p. 5. No. 3.2 in 3, Kobe, Japan (2009)
[33]
Roşu G and Bensalem S Ball T and Jones RB Allen linear (interval) temporal logic – translation to LTL and monitor synthesis Computer Aided Verification 2006 Heidelberg Springer 263-277
[34]
Sun, P., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2446–2454 (2020)
[35]
Thomas G, Gade R, Moeslund TB, Carr P, and Hilton A Computer vision for sports: current applications and research topics Comput. Vis. Image Underst. 2017 159 3-18
[36]
Turtiainen, H., Costin, A., Lahtinen, T., Sintonen, L., Hamalainen, T.: Towards large-scale, automated, accurate detection of CCTV camera objects using computer vision. applications and implications for privacy, safety, and cybersecurity. arXiv preprint arXiv:2006.03870 (2020)
[37]
Ward TM et al. Computer vision in surgery Surgery 2021 169 5 1253-1256
[38]
Wolper P Temporal logic can be more expressive Inf. Control 1983 56 1–2 72-99
[39]
Xiao, P., et al.: PandaSet: advanced sensor suite dataset for autonomous driving. In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 3095–3101. IEEE (2021)
[40]
Xu Z and Julius AA Census signal temporal logic inference for multiagent group behavior analysis IEEE Trans. Autom. Sci. Eng. 2016 15 1 264-277
[41]
Yadav, P., Curry, E.: VidCEP: complex event processing framework to detect spatiotemporal patterns in video streams. In: 2019 IEEE International conference on big data (big data), pp. 2513–2522. IEEE (2019)
[42]
Yu, F., et al.: Bdd100k: a diverse driving dataset for heterogeneous multitask learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2636–2645 (2020)
[43]
Zhang Y, Carballo A, Yang H, and Takeda K Perception and sensing for autonomous vehicles under adverse weather conditions: a survey ISPRS J. Photogrammetry Remote Sens. 2023 196 146-177
Recommendations
Elements of Timed Pattern Matching
The rise of machine learning and cloud technologies has led to a remarkable influx of data within modern cyber-physical systems. However, extracting meaningful information from this data has become a significant challenge due to its volume and complexity. ...
On the average-case complexity of pattern matching with wildcards
AbstractPattern matching with wildcards is a string matching problem with the goal of finding all factors of a text t of length n that match a pattern x of length m, where wildcards (characters that match everything) may be present. In this paper we ...
Suffix array for multi-pattern matching with variable length wildcards
Approximate multi-pattern matching is an important issue that is widely and frequently utilized, when the pattern contains variable-length wildcards. In this paper, two suffix array-based algorithms have been proposed to solve this problem. Suffix ...
Comments
Information & Contributors
Information
Published In
Oct 2023
493 pages
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2023.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 03 October 2023
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 05 Jan 2025