research-article

Semantic Completion and Filtration for Image–Text Retrieval

Authors:

Anan LiuAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications and Applications, Volume 19, Issue 4

Article No.: 140, Pages 1 - 20

https://doi.org/10.1145/3572844

Published: 27 February 2023 Publication History

Abstract

Image–text retrieval is a vital task in computer vision and has received growing attention, since it connects cross-modality data. It comes with the critical challenges of learning unified representations and eliminating the large gap between visual and textual domains. Over the past few decades, although many works have made significant progress in image–text retrieval, they are still confronted with the challenge of incomplete text descriptions of images, i.e., how to fully learn the correlations between relevant region–word pairs with semantic diversity. In this article, we propose a novel semantic completion and filtration (SCAF) method to alleviate the above issue. Specifically, the text semantic completion module is presented to generate a complete semantic description of an image using multi-view text descriptions, guiding the model to explore the correlations of relevant region–word pairs fully. Meanwhile, the adaptive structural semantic matching module is presented to filter irrelevant region–word pairs by considering the relevance score of each region–word pair, which facilitates the model to focus on learning the relevance of matching pairs. Extensive experiments show that our SCAF outperforms the existing methods on Flickr30K and MSCOCO datasets, which demonstrates the superiority of our proposed method.

References

[1]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision. 2425–2433.

Digital Library

[2]

Regina Barzilay and Lillian Lee. 2003. Learning to paraphrase: An unsupervised approach using multiple-sequence alignment. CoRR cs.CL/0304006 (2003).

[3]

Ali Furkan Biten, Lluís Gómez, Marçal Rusiñol, and Dimosthenis Karatzas. 2019. Good news, everyone! Context driven entity-aware captioning for news images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12466–12475.

[4]

Hui Chen, Guiguang Ding, Xudong Liu, Zijia Lin, Ji Liu, and Jungong Han. 2020. IMRAM: Iterative matching with recurrent attention memory for cross-modal image-text retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12652–12660.

[5]

Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR abs/1412.3555 (2014).

[6]

Gaoxiang Cong, Liang Li, Zhenhuan Liu, Yunbin Tu, Weijun Qin, Shenyuan Zhang, Chengang Yan, Wenyu Wang, and Bin Jiang. 2022. LS-GAN: Iterative language-based image manipulation via long and short term consistency reasoning. In Proceedings of the 30th ACM International Conference on Multimedia (MM’22), João Magalhães, Alberto Del Bimbo, Shin’ichi Satoh, Nicu Sebe, Xavier Alameda-Pineda, Qin Jin, Vincent Oria, and Laura Toni (Eds.). ACM, 4496–4504.

Digital Library

[7]

Marie-Catherine de Marneffe and Christopher D. Manning. 2008. The stanford typed dependencies representation. In Proceedings of the Workshop on Cross-Framework and Cross-Domain Parser Evaluation. 1–8.

Digital Library

[8]

Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2018. VSE++: Improving visual-semantic embeddings with hard negatives. In Proceedings of the British Machine Vision Conference. 12.

[9]

Zhihao Fan, Zhongyu Wei, Siyuan Wang, and Xuanjing Huang. 2019. Bridging by word: Image grounded vocabulary construction for visual captioning. In Proceedings of the Conference of the Association for Computational Linguistics. 6514–6524.

[10]

Katja Filippova. 2010. Multi-Sentence compression: Finding shortest paths in word graphs. In Proceedings of the International Conference on Computational Linguistics, Proceedings of the Conference. 322–330.

[11]

Andrea Frome, Gregory S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, Marc’Aurelio Ranzato, and Tomás Mikolov. 2013. DeViSE: A deep visual-semantic embedding model. In Proceedings of the Annual Conference on Neural Information Processing Systems. 2121–2129.

[12]

Jiuxiang Gu, Jianfei Cai, Shafiq R. Joty, Li Niu, and Gang Wang. 2018. Look, Imagine and match: Improving textual-visual cross-modal retrieval with generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7181–7189.

[13]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770–778.

[14]

Feiran Huang, Xiaoming Zhang, Zhonghua Zhao, and Zhoujun Li. 2019. Bi-Directional spatial-semantic attention networks for image-text matching. IEEE Trans. Image Process. 28, 4 (2019), 2008–2020.

Digital Library

[15]

Yan Huang, Wei Wang, and Liang Wang. 2017. Instance-Aware image and sentence matching with selective multimodal LSTM. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7254–7262.

[16]

Yan Huang, Qi Wu, Chunfeng Song, and Liang Wang. 2018. Learning semantic concepts and order for image and sentence matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6163–6171.

[17]

Andrej Karpathy and Li Fei-Fei. 2017. Deep visual-semantic alignments for generating image descriptions. IEEE Trans. Pattern Anal. Mach. Intell. 39, 4 (2017), 664–676.

Digital Library

[18]

Andrej Karpathy, Armand Joulin, and Fei-Fei Li. 2014. Deep fragment embeddings for bidirectional image sentence mapping. In Proceedings of the Annual Conference on Neural Information Processing Systems. 1889–1897.

[19]

Ryan Kiros, Ruslan Salakhutdinov, and Richard S. Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. CoRR abs/1411.2539 (2014).

[20]

Logan Lebanoff, John Muchovej, Franck Dernoncourt, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. 2019. Analyzing sentence fusion in abstractive summarization. CoRR abs/1910.00203 (2019).

[21]

Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. 2018. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision, Vol. 11208. 212–228.

Digital Library

[22]

Liang Li, Xingyu Gao, Jincan Deng, Yunbin Tu, Zheng-Jun Zha, and Qingming Huang. 2022. Long short-term relation transformer with global gating for video captioning. IEEE Trans. Image Process. 31 (2022), 2726–2738.

Digital Library

[23]

Yongzhi Li, Duo Zhang, and Yadong Mu. 2020. Visual-Semantic matching by exploring high-order attention and distraction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 12783–12792.

[24]

Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In Proceedings of the European Conference on Computer Vision, Vol. 8693. 740–755.

[25]

Chunxiao Liu, Zhendong Mao, An-An Liu, Tianzhu Zhang, Bin Wang, and Yongdong Zhang. 2019. Focus your attention: A bidirectional focal attention network for image-text matching. In Proceedings of the ACM International Conference on Multimedia. 3–11.

Digital Library

[26]

Chunxiao Liu, Zhendong Mao, Tianzhu Zhang, Hongtao Xie, Bin Wang, and Yongdong Zhang. 2020. Graph structured network for image-text matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 10918–10927.

[27]

Junhao Liu, Min Yang, Chengming Li, and Ruifeng Xu. 2021. Improving cross-modal image-text retrieval with teacher-student learning. IEEE Trans. Circuits Syst. Video Technol. 31, 8 (2021), 3242–3253.

[28]

Xuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha, Zechao Li, Qi Tian, and Qingming Huang. 2022. Entity-enhanced adaptive reconstruction network for weakly supervised referring expression grounding. arXiv:2207.08386. Retrieved from https://arxiv.org/abs/2207.08386.

[29]

Minnan Luo, Xiaojun Chang, Zhihui Li, Liqiang Nie, Alexander G. Hauptmann, and Qinghua Zheng. 2017. Simple to complex cross-modal learning to rank. Comput. Vis. Image Underst. 163 (2017), 67–77.

Digital Library

[30]

Lin Ma, Zhengdong Lu, Lifeng Shang, and Hang Li. 2015. Multimodal convolutional neural networks for matching image and sentence. In Proceedings of the IEEE International Conference on Computer Vision. 2623–2631.

Digital Library

[31]

Yanjun Ma, Dianhai Yu, Tian Wu, and Haifeng Wang. 2019. PaddlePaddle: An open-source deep learning platform from industrial practice. Front. Data Comput. 1, 1 (2019), 105–115.

[32]

Hyeonseob Nam, Jung-Woo Ha, and Jeonghee Kim. 2017. Dual attention networks for multimodal reasoning and matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2156–2164.

[33]

Xiushan Nie, Bowei Wang, Jiajia Li, Fanchang Hao, Muwei Jian, and Yilong Yin. 2021. Deep multiscale fusion hashing for cross-modal retrieval. IEEE Trans. Circ. Syst. Video Technol. 31, 1 (2021), 401–410.

Digital Library

[34]

Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. 2017. Hierarchical multimodal LSTM for dense visual-semantic embedding. In Proceedings of the IEEE International Conference on Computer Vision. 1899–1907.

[35]

Paddlepaddle. 2019. PaddlePaddle: An Easy-to-use, Easy-to-learn Deep Learning Platform. Retrieved from http://www.paddlepaddle.org/.

[36]

Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2017. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. Int. J. Comput. Vis. 123, 1 (2017), 74–93.

Digital Library

[37]

Tingting Qiao, Jing Zhang, Duanqing Xu, and Dacheng Tao. 2019. MirrorGAN: Learning text-to-image generation by redescription. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1505–1514.

[38]

Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2017. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39, 6 (2017), 1137–1149.

Digital Library

[39]

Nikolaos Sarafianos, Xiang Xu, and Ioannis A. Kakadiaris. 2019. Adversarial representation learning for text-to-image matching. In Proceedings of the IEEE International Conference on Computer Vision. 5813–5823.

[40]

Kevin J. Shih, Saurabh Singh, and Derek Hoiem. 2016. Where to look: Focus regions for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4613–4621.

[41]

Hao Wang, Doyen Sahoo, Chenghao Liu, Ee-Peng Lim, and Steven C. H. Hoi. 2019. Learning cross-modal embeddings with adversarial networks for cooking recipes and food images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11572–11581.

[42]

Hao Wang, Zheng-Jun Zha, Liang Li, Dong Liu, and Jiebo Luo. 2021. Structured multi-level interaction network for video moment localization via language query. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’21). Computer Vision Foundation/IEEE, 7026–7035.

[43]

Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik. 2019. Learning two-branch neural networks for image-text matching tasks. IEEE Trans. Pattern Anal. Mach. Intell. 41, 2 (2019), 394–407.

Digital Library

[44]

Sijin Wang, Ruiping Wang, Ziwei Yao, Shiguang Shan, and Xilin Chen. 2020. Cross-modal scene graph matching for relationship-aware image-text retrieval. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision. 1497–1506.

[45]

Yaxiong Wang, Hao Yang, Xueming Qian, Lin Ma, Jing Lu, Biao Li, and Xin Fan. 2019. Position focused attention network for image-text matching. In Proceedings of the ACM International Joint Conference on Artificial Intelligence. 3792–3798.

[46]

Zihao Wang, Xihui Liu, Hongsheng Li, Lu Sheng, Junjie Yan, Xiaogang Wang, and Jing Shao. 2019. CAMP: Cross-Modal adaptive message passing for text-image retrieval. In Proceedings of the IEEE International Conference on Computer Vision. 5763–5772.

[47]

Xin Wen, Zhizhong Han, and Yu-Shen Liu. 2021. CMPD: Using cross memory network with pair discrimination for image-text retrieval. IEEE Trans. Circuits Syst. Video Technol. 31, 6 (2021), 2427–2437.

[48]

Lingxiang Wu, Min Xu, Lei Sang, Ting Yao, and Tao Mei. 2021. Noise augmented double-stream graph convolutional networks for image captioning. IEEE Trans. Circuits Syst. Video Technol. 31, 8 (2021), 3118–3127.

[49]

Yiling Wu, Shuhui Wang, Guoli Song, and Qingming Huang. 2021. Augmented adversarial training for cross-modal retrieval. IEEE Trans. Multimedia 23 (2021), 559–571.

[50]

Xing Xu, Li He, Huimin Lu, Lianli Gao, and Yanli Ji. 2019. Deep adversarial metric learning for cross-modal retrieval. World Wide Web 22, 2 (2019), 657–672.

Digital Library

[51]

Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. 2018. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5831–5840.

[52]

Chenyang Zhang, Yongqiang Tang, Zhizhong Zhang, Ding Li, Xuebing Yang, and Wensheng Zhang. 2021. Improving domain-adaptive person re-identification by dual-alignment learning with camera-aware image generation. IEEE Trans. Circ. Syst. Video Technol. 31, 11 (2021), 4334–4346.

Digital Library

[53]

Qi Zhang, Zhen Lei, Zhaoxiang Zhang, and Stan Z. Li. 2020. Context-Aware attention network for image-text retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3533–3542.

[54]

Bin Zhu, Chong-Wah Ngo, Jingjing Chen, and Yanbin Hao. 2019. R2GAN: Cross-Modal recipe retrieval with generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 11477–11486.

Cited By

Jin RHou TJin TYuan JDu C(2024)A method for image–text matching based on semantic filtering and adaptive adjustmentJournal on Image and Video Processing10.1186/s13640-024-00639-y2024:1Online publication date: 29-Aug-2024
https://dl.acm.org/doi/10.1186/s13640-024-00639-y
Zhu SFang JFang PXue H(2024)Improving Scene Text Retrieval via Stylized Middle ModalityACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3696209Online publication date: 16-Sep-2024
https://doi.org/10.1145/3696209
Yang KDu W(2024)A Low-Density Parity-Check Coding Scheme for LoRa NetworkingACM Transactions on Sensor Networks10.1145/366592820:4(1-29)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3665928
Show More Cited By

Index Terms

Semantic Completion and Filtration for Image–Text Retrieval
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

MKVSE: Multimodal Knowledge Enhanced Visual-semantic Embedding for Image-text Retrieval
Image-text retrieval aims to take the text (image) query to retrieve the semantically relevant images (texts), which is fundamental and critical in the search system, online shopping, and social network. Existing works have shown the effectiveness of ...
Semantic Completion: Enhancing Image-Text Retrieval with Information Extraction and Compression
Advances in Knowledge Discovery and Data Mining
Abstract
Image-text retrieval is an essential branch in the field of information retrieval, facing the serious challenge of the cross-modal semantic gap. Although significant progress has been made in recent years, most research has ignored an essential ...
Multi-view and region reasoning semantic enhancement for image-text retrieval
Abstract
Image and text retrieval is a crucial topic in the fields of language and vision. The key to successful Image-Text retrieval is achieving accurate cross-modal representation and capturing essential correlations between image-sentence or words-...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 19, Issue 4

July 2023

263 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3582888

Editor:
Abdulmotaleb El Saddik
Mohamed Bin Zayed University of Artificial Intelligence, UAE and University of Ottawa, Canada

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 February 2023

Online AM: 23 November 2022

Accepted: 20 November 2022

Revised: 16 September 2022

Received: 23 April 2022

Published in TOMM Volume 19, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China
China Postdoctoral Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
584
Total Downloads

Downloads (Last 12 months)321
Downloads (Last 6 weeks)19

Reflects downloads up to 14 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jin RHou TJin TYuan JDu C(2024)A method for image–text matching based on semantic filtering and adaptive adjustmentJournal on Image and Video Processing10.1186/s13640-024-00639-y2024:1Online publication date: 29-Aug-2024
https://dl.acm.org/doi/10.1186/s13640-024-00639-y
Zhu SFang JFang PXue H(2024)Improving Scene Text Retrieval via Stylized Middle ModalityACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3696209Online publication date: 16-Sep-2024
https://doi.org/10.1145/3696209
Yang KDu W(2024)A Low-Density Parity-Check Coding Scheme for LoRa NetworkingACM Transactions on Sensor Networks10.1145/366592820:4(1-29)Online publication date: 8-Jul-2024
https://dl.acm.org/doi/10.1145/3665928
Yin SZhao SWang HXu TChen E(2024)Exploiting Instance-level Relationships in Weakly Supervised Text-to-Video RetrievalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366357120:10(1-21)Online publication date: 12-Sep-2024
https://dl.acm.org/doi/10.1145/3663571
Zhang PLiu MSong XCao DGao ZNie L(2024)Universal Relocalizer for Weakly Supervised Referring Expression GroundingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/365604520:7(1-23)Online publication date: 16-May-2024
https://dl.acm.org/doi/10.1145/3656045
Yang JWang ZHuang BAi JYang YXiong Z(2024)Joint Distortion Restoration and Quality Feature Learning for No-reference Image Quality AssessmentACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364989920:7(1-20)Online publication date: 27-Mar-2024
https://dl.acm.org/doi/10.1145/3649899
Yang KChen YDu WOkoshi TKo JLiKamWa R(2024)OrchLoc: In-Orchard Localization via a Single LoRa Gateway and Generative Diffusion Model-based FingerprintingProceedings of the 22nd Annual International Conference on Mobile Systems, Applications and Services10.1145/3643832.3661876(304-317)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3643832.3661876
Li SXu XJiang XShen FSun ZCichocki A(2024)Cross-Modal Attention Preservation with Self-Contrastive Learning for Composed Query-Based Image RetrievalACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363946920:6(1-22)Online publication date: 8-Mar-2024
https://dl.acm.org/doi/10.1145/3639469
Chen JZhang H(2024)Semantic enhancement and multi-level alignment network for cross-modal retrievalMultimedia Tools and Applications10.1007/s11042-023-17956-5Online publication date: 12-Jan-2024
https://doi.org/10.1007/s11042-023-17956-5
Cheng WHan ZHe DWu L(2024)Multi-view and region reasoning semantic enhancement for image-text retrievalMultimedia Systems10.1007/s00530-024-01383-z30:4Online publication date: 15-Jun-2024
https://dl.acm.org/doi/10.1007/s00530-024-01383-z
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents