Relevance Feedback in Deep Convolutional Neural Networks for Content Based Image Retrieval

In this paper a novel Relevance Feedback approach that uses deep Convolutional Neural Networks (CNNs) for image retrieval is proposed. We utilize a deep CNN model to refine the feature representations of the deeper layer used for the retrieval, based on the feedback of the user. To this end, we adapt the pretrained model and we re-train the corresponding neural layers on relevant and irrelevant images, as qualified by the user. If the feedback of the users is large enough a generic model refinement method is also proposed, for improving the entire performance of the system in the retrieval task. In this case, we refine the deep CNN weights based on the feedback gathered from multiple users, forming a new training set. Experimental results denote the effectiveness of the proposed method in accomplishing better retrieval results with respect to a certain user's information need. The validation of the proposed approach on the queries used in the relevance feedback as well as on a generic unseen query dataset, demonstrates the improvement on the retrieval performance, as shown in the experimental results.