research-article

Open access

Exquisitor at the Lifelog Search Challenge 2024: Blending Conversational Search with User Relevance Feedback

Authors:

Omar Shahbaz Khan,

Stevan Rudinac,

Björn Þór JónssonAuthors Info & Claims

LSC '24: Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge

Pages 117 - 121

https://doi.org/10.1145/3643489.3661132

Published: 18 June 2024 Publication History

Abstract

The past decade has seen a rapid expansion of personal and interpersonal multimedia collections. These collections offer a wealth of information about individuals, including their interests, health, and significant life events. While automated techniques can assist in structuring and organizing these collections, they often have limitations in helping users effectively navigate and find relevant items within such large datasets. The Lifelog Search Challenge (LSC) provides a valuable benchmark for evaluating interactive retrieval systems designed for personal multimedia collections. Exquisitor utilizes a large-scale user relevance feedback (URF) approach for searching through large collections. To address challenges in highly descriptive retrieval tasks where the relevance feedback model may fail to identify essential elements, we have enhanced Exquisitor with conversational search capabilities powered by a Vision Language Model (VLM) and refined the features underlying the URF model. Furthermore, Exquisitor has been updated with a streamlined user interface that enables seamless switching between conversational search and URF modes.

References

[1]

Hervé Abdi and Lynne J Williams. 2010. Principal component analysis. Wiley interdisciplinary reviews: computational statistics 2, 4 (2010), 433--459.

Digital Library

[2]

Corinna Cortes and Vladimir Vapnik. 1995. Support-Vector Networks. Machine Learning 20, 3 (1995), 273--297.

[3]

Shiv Ram Dubey. 2021. A decade survey of content based image retrieval using deep learning. IEEE Transactions on Circuits and Systems for Video Technology 32, 5 (2021), 2687--2704.

Digital Library

[4]

Cathal Gurrin, Klaus Schoeffmann, Hideo Joho, Duc-Tien Dang-Nguyen, Michael Riegler, and Luca Piras (Eds.). 2018. Proceedings of the ACM Workshop on Lifelog Search Challenge, LSC 2018. ACM, Yokohama, Japan.

[5]

Cathal Gurrin, Liting Zhou, Graham Healy, Werner Bailer, Duc-Tien Dang-Nguyen, Steve Hodges, Björn Þór Jónsson, Jakub Lokoč, Luca Rossetto, Minh-Triet Tran, and Klaus Schöffmann. 2024. Introduction to the Seventh Annual Lifelog Search Challenge, LSC'24. International Conference on Multimedia Retrieval (ICMR'24).

Digital Library

[6]

Omar Shahbaz Khan, Aaron Duane, Björn Þór Jónsson, Jan Zahálka, Stevan Rudinac, and Marcel Worring. 2021. Exquisitor at the Lifelog Search Challenge 2021: Relationships Between Semantic Classifiers. In Proceedings of the 4th Annual on Lifelog Search Challenge (LSC '21). Association for Computing Machinery, New York, NY, USA, 3--6.

Digital Library

[7]

Omar Shahbaz Khan, Björn Þór Jónsson, Stevan Rudinac, Jan Zahálka, Hanna Ragnarsdóttir, Þórhildur Þorleiksdóttir, Gylfi Þór Guðmundsson, Laurent Amsaleg, and Marcel Worring. 2020. Interactive Learning for Multimedia at Large. In Proceedings of the European Conference on Information Retrieval (ECIR). Springer, Lisboa, Portugal, 16.

Digital Library

[8]

Omar Shahbaz Khan, Björn Þór Jónsson, Jan Zahálka, Stevan Rudinac, and Marcel Worring. 2019. Exquisitor at the Lifelog Search Challenge 2019. In Proceedings of the ACM Workshop on Lifelog Search Challenge, LSC 2019. ACM, Ottawa, ON, Canada, 7--11.

Digital Library

[9]

Omar Shahbaz Khan and Björn Þór Jónsson. 2023. User Relevance Feedback and Novices: Anecdotes from Exquisitor's Participation in Interactive Retrieval Competitions. In Proceedings of the 20th International Conference on Content-Based Multimedia Indexing (CBMI '23). Association for Computing Machinery, New York, NY, USA, 173--177.

Digital Library

[10]

Omar Shahbaz Khan, Mathias Dybkjær Larsen, Liam Alex Sonto Poulsen, Björn Þór Jónsson, Jan Zahálka, Stevan Rudinac, Dennis Koelma, and Marcel Worring. 2020. Exquisitor at the Lifelog Search Challenge 2020. In Proceedings of the Third Annual Workshop on Lifelog Search Challenge. ACM.

Digital Library

[11]

Omar Shahbaz Khan, Hongyi Zhu, Ujjwal Sharma, Evangelos Kanoulas, Stevan Rudinac, and Björn Þór Jónsson. 2024. Exquisitor at Video Browser Showdown 2024: Relevance Feedback Meets Conversational Search. In MultiMedia Modeling: 30th International Conference, MMM 2024, Amsterdam, The Netherlands, January 29 -- February 2, 2024, Proceedings, Part IV. Springer-Verlag, Berlin, Heidelberg, 347--355.

Digital Library

[12]

Pascal Mettes, Dennis C Koelma, and Cees GM Snoek. 2016. The imagenet shuffle: Reorganized pre-training for video event detection. In Proceedings of the 2016 ACM on International Conference on Multimedia Retrieval. 175--182.

Digital Library

[13]

Arnold WM Smeulders, Marcel Worring, Simone Santini, Amarnath Gupta, and Ramesh Jain. 2000. Content-based image retrieval at the end of the early years. IEEE Transactions on pattern analysis and machine intelligence 22, 12 (2000), 1349--1380.

Digital Library

[14]

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 11975--11986.

[15]

Hongyi Zhu, Jia-Hong Huang, Stevan Rudinac, and Evangelos Kanoulas. 2024. Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Models. arXiv e-prints, Article arXiv:2404.18746 (April 2024), arXiv:2404.18746 pages. arXiv:cs.MM/2404.18746

Cited By

Sharma UKhan ORudinac SJónsson B(2025)Exquisitor at the Video Browser Showdown 2025: Unifying Conversational Search and User Relevance FeedbackMultiMedia Modeling10.1007/978-981-96-2074-6_31(264-271)Online publication date: 1-Jan-2025
https://doi.org/10.1007/978-981-96-2074-6_31
Huang JZhu HShen YRudinac SKanoulas E(2025)Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion ModelsMultiMedia Modeling10.1007/978-981-96-2071-5_30(413-427)Online publication date: 2-Jan-2025
https://doi.org/10.1007/978-981-96-2071-5_30
Gurrin CZhou LHealy GBailer WDang Nguyen DHodges SJónsson BLokoč JRossetto LTran MSchöffmann KGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Introduction to the Seventh Annual Lifelog Search Challenge, LSC'24Proceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658891(1334-1335)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658891

Index Terms

Exquisitor at the Lifelog Search Challenge 2024: Blending Conversational Search with User Relevance Feedback
1. Information systems
  1. Information retrieval
    1. Users and interactive retrieval
      1. Collaborative search

Recommendations

lifeXplore at the Lifelog Search Challenge 2024
LSC '24: Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge

Building on the success of lifeXplore 2023, which is the winner system of last year's Lifelog Search Challenge (LSC2023), we present a lifelog retrieval system that is both easy-to-use and effective in retrieving lifelog data. lifeXplore 2024 employs ...
Exquisitor at the Lifelog Search Challenge 2020
LSC '20: Proceedings of the Third Annual Workshop on Lifelog Search Challenge

We present an enhanced version of Exquisitor, our interactive and scalable media exploration system. At its core, Exquisitor is an interactive learning system using relevance feedback on media items to build a model of the users' information need. ...
Exquisitor at the Lifelog Search Challenge 2019
LSC '19: Proceedings of the ACM Workshop on Lifelog Search Challenge

Interactive learning is an umbrella term for methods that attempt to understand the information need of the user and formulate queries that satisfy that information need. We propose to apply the state of the art in interactive multimodal learning to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

LSC '24: Proceedings of the 7th Annual ACM Workshop on the Lifelog Search Challenge

June 2024

128 pages

ISBN:9798400705502

DOI:10.1145/3643489

This work is licensed under a Creative Commons Attribution-ShareAlike International 4.0 License.

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 June 2024

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Icelandic Research Fund

Conference

LSC '24

Sponsor:

SIGMM

LSC '24: 7th Annual ACM Workshop on the Lifelog Search Challenge

June 10, 2024

Phuket, Thailand

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
141
Total Downloads

Downloads (Last 12 months)141
Downloads (Last 6 weeks)23

Reflects downloads up to 06 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sharma UKhan ORudinac SJónsson B(2025)Exquisitor at the Video Browser Showdown 2025: Unifying Conversational Search and User Relevance FeedbackMultiMedia Modeling10.1007/978-981-96-2074-6_31(264-271)Online publication date: 1-Jan-2025
https://doi.org/10.1007/978-981-96-2074-6_31
Huang JZhu HShen YRudinac SKanoulas E(2025)Image2Text2Image: A Novel Framework for Label-Free Evaluation of Image-to-Text Generation with Text-to-Image Diffusion ModelsMultiMedia Modeling10.1007/978-981-96-2071-5_30(413-427)Online publication date: 2-Jan-2025
https://doi.org/10.1007/978-981-96-2071-5_30
Gurrin CZhou LHealy GBailer WDang Nguyen DHodges SJónsson BLokoč JRossetto LTran MSchöffmann KGurrin CKongkachandra RSchoeffmann KDang-Nguyen DRossetto LSatoh SZhou L(2024)Introduction to the Seventh Annual Lifelog Search Challenge, LSC'24Proceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658891(1334-1335)Online publication date: 30-May-2024
https://dl.acm.org/doi/10.1145/3652583.3658891

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents