Article

Ridge regression, hubness, and zero-shot learning

Authors:

Yutaro Shigeto,

Masashi Shimbo,

Yuji MatsumotoAuthors Info & Claims

ECMLPKDD'15: Proceedings of the 2015th European Conference on Machine Learning and Knowledge Discovery in Databases - Volume Part I

Pages 135 - 151

https://doi.org/10.1007/978-3-319-23528-8_9

Published: 07 September 2015 Publication History

Abstract

This paper discusses the effect of hubness in zero-shot learning, when ridge regression is used to find a mapping between the example space to the label space. Contrary to the existing approach, which attempts to find a mapping from the example space to the label space, we show that mapping labels into the example space is desirable to suppress the emergence of hubs in the subsequent nearest neighbor search step. Assuming a simple data model, we prove that the proposed approach indeed reduces hubness. This was verified empirically on the tasks of bilingual lexicon extraction and image labeling: hubness was reduced with both of these tasks and the accuracy was improved accordingly.

References

[1]

Ács, J., Pajkossy, K., Kornai, A.: Building basic vocabulary across 40 languages. In: Proceedings of the 6th Workshop on Building and Using Comparable Corpora, pp. 52-58 (2013)

[2]

Akata, Z., Lee, H., Schiele, B.: Zero-shot learning with structured embeddings (2014). arXiv preprint arXiv:1409.8403v1

[3]

Al-Rfou, R., Perozzi, B., Skiena, S.: Polyglot: Distributed word representations for multilingual NLP. In: CoNLL 2013, pp. 183-192 (2013)

[4]

Bakir, G., Hofmann, T., Schölkopf, B., Smola, A.J., Taskar, B., Vishwanathan, S.V.N. (eds.): Predicting Structured Data. MIT press (2007)

Digital Library

[5]

Bingham, E., Mannila, H.: Random projection in dimensionality reduction: applications to image and text data. In: KDD 2001, pp. 245-250 (2001)

Digital Library

[6]

Dasgupta, S.: Experiments with random projection. In: UAI 2000, pp. 143-151 (2000)

Digital Library

[7]

Dinu, G., Baroni, M.: How to make words with vectors: phrase generation in distributional semantics. In: ACL 2014, pp. 624-633 (2014)

[8]

Dinu, G., Baroni, M.: Improving zero-shot learning by mitigating the hubness problem. In: Workshop at ICLR 2015 (2015)

[9]

Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.: DeCAF: a deep convolutional activation feature for generic visual recognition (2013). arXiv preprint arXiv:1310.1531

[10]

Farhadi, A., Endres, I., Hoiem, D., Forsyth, D.: Describing objects by their attributes. In: CVPR 2009, pp. 1778-1785 (2009)

[11]

Frome, A., Corrado, G.S., Shlens, J., Bengio, S., Dean, J., Ronzato, M., Mikolov, T.: Devise: a deep visual-semantic embedding model. In: NIPS 2013, pp. 2121-2129 (2013)

Digital Library

[12]

Hardoon, D.R., Szedmak, S., Shawe-Taylor, J.: Canonical correlation analysis: An overview with application to learning methods. Neural Computation 16, 2639-2664 (2004)

Digital Library

[13]

Jegou, H., Harzallah, H., Schmid, C.: A contextual dissimilarity measure for accurate and efficient image search. In: CVPR 2007, pp. 1-8 (2007)

[14]

Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. In: CVPR 2009. pp. 951-958 (2009)

[15]

Larochelle, H., Erhan, D., Bengio, Y.: Zero-data learning of new tasks. In: AAAI 2008, pp. 646-651 (2008)

Digital Library

[16]

Lazaridou, A., Bruni, E., Baroni, M.: Is this a wampimuk? Cross-modal mapping between distributional semantics and the visual world. In: ACL 2014, pp. 1403-1414 (2014)

[17]

Manning, C.D., Raghavan, P., Schütze, H.: Introduction to Information Retrieval. Cambridge University Press (2008)

Digital Library

[18]

Mika, S., Schölkopf, B., Smola, A., Müller, K.R., Scholz, M., Rätsch, G.: Kernel PCA and de-noising in feature space. In: NIPS 1998, pp. 536-542 (1998)

Digital Library

[19]

Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: Workshop at ICLR 2013 (2013)

[20]

Mikolov, T., Le, Q.V., Sutskever, I.: Exploiting similarities among languages for machine translation (2013). arXiv preprint arXiv:1309.4168

[21]

Norouzi, M., Mikolov, T., Bengio, S., Singer, Y., Shlens, J., Frome, A., Corrado, G.S., Dean, J.: Zero-shot learning by convex combination of semantic embeddings. In: ICLR 2014 (2014)

[22]

Palatucci, M., Pomerleau, D., Hinton, G., Mitchell, T.M.: Zero-shot learning with semantic output codes. In: NIPS 2009, pp. 1410-1418 (2009)

Digital Library

[23]

Radovanović, M., Nanopoulos, A., Ivanović, M.: Hubs in space: Popular nearest neighbors in high-dimensional data. Journal of Machine Learning Research 11, 2487-2531 (2010)

Digital Library

[24]

Schnitzer, D., Flexer, A., Schedl, M., Widmer, G.: Local and global scaling reduce hubs in space. Journal of Machine Learning Research 13, 2871-2902 (2012)

Digital Library

[25]

Socher, R., Ganjoo, M., Manning, C.D., Ng, A.Y.: Zero-shot learning through cross-modal transfer. In: NIPS 2013, pp. 935-943 (2013)

Digital Library

[26]

Suzuki, I., Hara, K., Shimbo, M., Saerens, M., Fukumizu, K.: Centering similarity measures to reduce hubs. In: EMNLP 2013, pp. 613-623 (2013)

[27]

Tomašev, N., Rupnik, J., Mladenić, D.: The role of hubs in cross-lingual supervised document retrieval. In: Pei, J., Tseng, V.S., Cao, L., Motoda, H., Xu, G. (eds.) PAKDD 2013, Part II. LNCS, vol. 7819, pp. 185-196. Springer, Heidelberg (2013)

[28]

Vinokourov, A., Shawe-Taylor, J., Cristianini, N.: Inferring a semantic representation of text via cross-language correlation analysis. In: NIPS 2002, pp. 1473-1480 (2002)

Digital Library

[29]

Weston, J., Chapelle, O., Vapnik, V., Elisseeff, A., Schölkopf, B.: Kernel dependency estimation. In: NIPS 2002, pp. 873-880 (2002)

Digital Library

Cited By

Zhou YLiu DMok PLarson K(2024)Zero-shot sketch based image retrieval via modality capacity guidanceProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/197(1780-1787)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/197
Chen RZhu XChen NLi WMa YYang RWang WEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Bridging Language and Geometric Primitives for Zero-shot Point Cloud SegmentationProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612409(5380-5388)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612409
Zhang YFeng SEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Enhancing Domain-Invariant Parts for Generalized Zero-Shot LearningProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611764(6283-6291)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3611764
Show More Cited By

Recommendations

Ordinal zero-shot learning
IJCAI'17: Proceedings of the 26th International Joint Conference on Artificial Intelligence

Zero-shot learning predicts new class even if no training data is available for that class. The solution to conventional zero-shot learning usually depends on side information such as attribute or text corpora. But these side information is not easy to ...
Transductive Visual-Semantic Embedding for Zero-shot Learning
ICMR '17: Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval

Zero-shot learning (ZSL) aims to bridge the knowledge transfer via available semantic representations (e.g., attributes) between labeled source instances of seen classes and unlabelled target instances of unseen classes. Most existing ZSL approaches ...
Zero-Shot Learning with Noisy Labels
Abstract
Zero-shot learning (ZSL) is an attractive technique that can recognize novel object classes without any visual examples, but most existing methods assume that the class labels of the training instances from seen classes are accurate and reliable. ...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

ECMLPKDD'15: Proceedings of the 2015th European Conference on Machine Learning and Knowledge Discovery in Databases - Volume Part I

September 2015

707 pages

ISBN:9783319235271

Editors:
Annalisa Appice
University of Bari Aldo Moro, Bari, Italy
,
Pedro Pereira Rodrigues
University of Porto, Porto, Portugal
,
Vítor Santos Costa
University of Porto - CRACS/INESC TEC, Porto, Portugal
,
Carlos Soares
University of Porto - INESC TEC, Porto, Portugal
,
João Gama
University of Porto - INESC TEC, Porto, Portugal

Sponsors

Huawei Technologies Co. Ltd.: Huawei Technologies Co. Ltd.
Zalando: Zalando
ONRGlobal: U.S. Office of Naval Research Global
BNPPARIBAS: BNP PARIBAS
Amazon: Amazon.com

Publisher

Springer

Gewerbestrasse 11 CH-6330, Cham (ZG), Switzerland

Publication History

Published: 07 September 2015

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhou YLiu DMok PLarson K(2024)Zero-shot sketch based image retrieval via modality capacity guidanceProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence10.24963/ijcai.2024/197(1780-1787)Online publication date: 3-Aug-2024
https://dl.acm.org/doi/10.24963/ijcai.2024/197
Chen RZhu XChen NLi WMa YYang RWang WEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Bridging Language and Geometric Primitives for Zero-shot Point Cloud SegmentationProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3612409(5380-5388)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3612409
Zhang YFeng SEl Saddik AMei TCucchiara RBertini MTobon Vallejo DAtrey PHossain M(2023)Enhancing Domain-Invariant Parts for Generalized Zero-Shot LearningProceedings of the 31st ACM International Conference on Multimedia10.1145/3581783.3611764(6283-6291)Online publication date: 26-Oct-2023
https://dl.acm.org/doi/10.1145/3581783.3611764
Hu YChapman AWen GHall D(2022)What Can Knowledge Bring to Machine Learning?—A Survey of Low-shot Learning for Structured DataACM Transactions on Intelligent Systems and Technology10.1145/351003013:3(1-45)Online publication date: 3-Mar-2022
https://dl.acm.org/doi/10.1145/3510030
Tian JXu XWang KCao ZCai XShen HAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Structure-Aware Semantic-Aligned Network for Universal Cross-Domain RetrievalProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3532061(278-289)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3532061
Gowda SSevilla-Lara LKeller FRohrbach M(2022)CLASTER: Clustering with Reinforcement Learning for Zero-Shot Action RecognitionComputer Vision – ECCV 202210.1007/978-3-031-20044-1_11(187-203)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1007/978-3-031-20044-1_11
Li BXie CYang JDuan H(2022)Contrast and Aggregation Network for Generalized Zero-shot LearningArtificial Neural Networks and Machine Learning – ICANN 202210.1007/978-3-031-15931-2_32(383-394)Online publication date: 6-Sep-2022
https://dl.acm.org/doi/10.1007/978-3-031-15931-2_32
Li KLin MHu SLi R(2022)CLZT: A Contrastive Learning Based Framework for Zero-Shot Text ClassificationDatabase Systems for Advanced Applications10.1007/978-3-031-00126-0_45(623-630)Online publication date: 11-Apr-2022
https://dl.acm.org/doi/10.1007/978-3-031-00126-0_45
Badirli SAkata ZMohler GPicard CDundar MRanzato MBeygelzimer ADauphin YLiang PVaughan J(2021)Fine-grained zero-shot learning with DNA as side informationProceedings of the 35th International Conference on Neural Information Processing Systems10.5555/3540261.3541741(19352-19362)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.5555/3540261.3541741
Wang CMin SChen XSun XLi HRanzato MBeygelzimer ADauphin YLiang PVaughan J(2021)Dual progressive prototype network for generalized zero-shot learningProceedings of the 35th International Conference on Neural Information Processing Systems10.5555/3540261.3540486(2936-2948)Online publication date: 6-Dec-2021
https://dl.acm.org/doi/10.5555/3540261.3540486
Show More Cited By

View Options

View options

Figures

Tables

Media

View Table of Conten