research-article

Detecting outlying properties of exceptional objects

Authors:

Fabrizio Angiulli,

Fabio Fassetti,

Luigi PalopoliAuthors Info & Claims

ACM Transactions on Database Systems (TODS), Volume 34, Issue 1

Article No.: 7, Pages 1 - 62

https://doi.org/10.1145/1508857.1508864

Published: 23 April 2009 Publication History

Abstract

Assume you are given a data population characterized by a certain number of attributes. Assume, moreover, you are provided with the information that one of the individuals in this data population is abnormal, but no reason whatsoever is given to you as to why this particular individual is to be considered abnormal. In several cases, you will be indeed interested in discovering such reasons. This article is precisely concerned with this problem of discovering sets of attributes that account for the (a priori stated) abnormality of an individual within a given dataset. A criterion is presented to measure the abnormality of combinations of attribute values featured by the given abnormal individual with respect to the reference population. In this respect, each subset of attributes is intended to somehow represent a “property” of individuals. We distinguish between global and local properties. Global properties are subsets of attributes explaining the given abnormality with respect to the entire data population. With local ones, instead, two subsets of attributes are singled out, where the former one justifies the abnormality within the data subpopulation selected using the values taken by the exceptional individual on those attributes included in the latter one. The problem of individuating abnormal properties with associated explanations is formally stated and analyzed. Such a formal characterization is then exploited in order to devise efficient algorithms for detecting both global and local forms of most abnormal properties. The experimental evidence, which is accounted for in the article, shows that the algorithms are both able to mine meaningful information and to accomplish the computational task by examining a negligible fraction of the search space.

References

[1]

Aggarwal, C. C. and Yu, P. 2001. Outlier detection for high dimensional data. In Proceedings of the International Conference on Managment of Data (SIGMOD), 37--46.

Digital Library

[2]

Agrawal, R. and Srikant, R. 1994. Fast algorithms for mining association rules in large databases. In Proceedings of the International Conference on Very Large Databases (VLDB), 487--499.

Digital Library

[3]

Angiulli, F. and Pizzuti, C. 2002. Fast outlier detection in large high-dimensional datasets. In Proceedings of the International Conference on Principles of Data Mining and Knowledge Discovery (PKDD), 15--26.

Digital Library

[4]

Angiulli, F. and Pizzuti, C. 2005. Outlier mining in large high-dimensional datasets. IEEE Trans. Knowl. Data Eng. 17, 2, 203--215.

Digital Library

[5]

Arning, A., Aggarwal, C., and Raghavan, P. 1996. A linear method for deviation detection in large databases. In Proceedings of the International Conference on Knowledge Discovery and Data Mining (KDD), 164--169.

[6]

Barnett, V. and Lewis, T. 1994. Outliers in Statistical Data. John Wiley & Sons.

[7]

Breiman, L., Friedman, J., Olshen, R., and Stone, C. 1984. Classification and Regression Trees. Wadsworth, Belmont.

[8]

Breunig, M. M., Kriegel, H., Ng, R., and Sander, J. 2000. Lof: Identifying density-based local outliers. In Proceedings of the International Conference on Managment of Data (SIGMOD), 93--104.

Digital Library

[9]

Chaudhuri, S. and Dayal, U. 1997. An overview of data warehousing and olap technology. SIGMOD Rec. 26, 1, 65--74.

Digital Library

[10]

Chen, Z. and Toda, S. 1995. The complexity of selecting maximal solutions. Inf. Comput. 1192, 2, 231--239.

Digital Library

[11]

Codd, E., Codd, S., and Salley, C. 1993. Providing OLAP (on-line analytical processing) to user-analysts: An it mandate. Tech. rep., Codd & Date, Inc.

[12]

De Benedictis, G., Rose, G., Carrieri, G., Luca, M. D., Falcone, E., Passarino, G., Bonafè, M., Monti, D., Baggio, G., Bertolini, S., Mari, D., Mattace, R., and Franceschi, C. 1999. Mitochondrial DNA inherited variants are associates with successful aging and longevity in humans. FASEB J. 13, 12, 1532--1536.

[13]

Garasto, S., Berardelli, M., Rango, F. D., Mari, V., Feraco, E., and Benedictis, G. D. 2004. A study of the average effect of the 3'apob-vntr polymorphism on lipidemic parameters could explain why the short alleles (< 35 repeats) are rare in centenarians. BMC Medical Genetics 5, 3.

[14]

Garey, M. R. and Johnson, D. S. 1979. Computer and Intractability. W. H. Freeman and Company, New York.

Digital Library

[15]

Gini, C. 1921. Measurement of inequality of incomes. The Economic J. 31, 124--126.

[16]

Griffiths, A. J. F., Miller, J. H., Suzuki, D. T., Lewontin, R. C., and Gelbart, W. M. 1996. An Introduction to Genetic Analysis. W. H. Freeman.

[17]

Hodge, V. and Austin, J. 2004. A survey of outlier detection methodologies. Artif. Intell. Rev. 22, 2, 85--126.

Digital Library

[18]

Karp, R. 1972. Reducibility among combinatorial problems. In Complexity of Computer Computations. Plenum, New York, 85--103.

[19]

Klösgen, W. 1996. Explora: A multipattern and multistrategy discovery assistant. In Advances in Knowledge Discovery and Data Mining (KDD), 249--271.

Digital Library

[20]

Knorr, E. and Ng, R. 1998. Algorithms for mining distance-based outliers in large datasets. In Proceedings of the International Conference on Very Large Databases (VLDB), 392--403.

Digital Library

[21]

Knorr, E. and Ng, R. 1999. Finding intensional knowledge of distance-based outliers. In Proceedings of the International Conference on Very Large Databases(VLDB), 211--222.

Digital Library

[22]

Knorr, E., Ng, R., and Tucakov, V. 2000. Distance-based outlier: Algorithms and applications. VLDB J. 8, 3-4, 237--253.

Digital Library

[23]

Knuth, D. 1998. The Art of Computer Programming, Volume 3: Sorting and Searching, 2nd ed. Addison-Wesley.

Digital Library

[24]

Krentel, M. 1988. The complexity of optimization problems. J. Comput. Syst. Sci. 36, 3, 490--509.

Digital Library

[25]

Mehta, M., Agrawal, R., and Rissanen, J. 1996. Sliq: A fast scalable classifier for data mining. In International Conference on Extending Database Technology (EDBT), 18--32.

Digital Library

[26]

Newman, D. J., Hettich, S., Blake, C. L., and Merz, C. 1998. UCI repository of machine learning databases.

[27]

Papadimitriou, C. 1994. Computational Complexity. Addison-Wesley, USA.

[28]

Passarino, G., Montesanto, A., Dato, S., Giordano, S., Domma, F., Mari, V., Feraco, E., and Benedictis, G. D. 2006. Sex and age specificity of susceptibility genes modulating survival at old age. Human Heredity (Int. J. Hum. Medical Genetics) 62, 4, 213--220.

[29]

Ramaswamy, S., Rastogi, R., and Shim, K. 2000. Efficient algorithms for mining outliers from large datasets. In Proceedings of the International Conference on Managment of Data (SIGMOD), 427--438.

Digital Library

[30]

Rymon, R. 1992. Search through systematic set enumeration. In Proceedings of the International Conference on Principles of Knowledge and Reasoning (KR), 539--550.

[31]

Sarawagi, S., Agrawal, R., and Megiddo, N. 1998. Discovery-Driven exploration of OLAP data cubes. In Proceedings of the International Conference on Extending Database Technology (EDBT), 168--182.

Digital Library

[32]

Selman, A. 1994. A taxonomy of complexity classes of functions. J. Comput. Syst. Sci. 48, 2, 357--381.

Digital Library

[33]

Shafer, J., Agrawal, R., and Mehta, M. 1996. Sprint: A scalable parallel classifier for data mining. In Proceedings of the International Conference on Very Large Data Bases (VLDB), 544--555.

Digital Library

[34]

Suzuki, E. 2006. Data mining methods for discovering interesting exceptions from an unsupervised table. J. Universal Comput. Sci. 12, 6, 627--653.

[35]

Tan, P. N., Steinbach, M., and Kumar, V. 2005. Introduction to Data Mining. Addison-Wesley.

Digital Library

[36]

Wei, L., Qian, W., Zhou, A., Jin, W., and Yu, J. 2003. Hot: Hypergraph-Based outlier test for categorical data. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD), 399--410.

Digital Library

[37]

Zhang, J., Gao, Q., and Wang, H. 2006. A novel method for detecting outlying subspaces in high-dimensional databases using genetic algorithm. In Proceedings of the International Conference on Data Mining (ICDM), 731--740.

Digital Library

[38]

Zhang, J. and Wang, H. 2006. Detecting outlying subspaces for high-dimensional data: The new task, algorithms, and performance. Knowl. Inf. Syst. 10, 3, 333--355.

Digital Library

[39]

Zhu, C., Kitagawa, H., and Faloutsos, C. 2005. Example-Based robust outlier detection in high dimensional datasets. In Proceedings of the International Conference on Data Mining (ICDM), 829--832.

Digital Library

Cited By

Angiulli FFassetti FFerragina L(2024)Data Mining: Outleir DetectionReference Module in Life Sciences10.1016/B978-0-323-95502-7.00011-7Online publication date: 2024
https://doi.org/10.1016/B978-0-323-95502-7.00011-7
Angiulli FFassetti FNisticò SPalopoli L(2024)Explaining outliers and anomalous groups via subspace density contrastive lossMachine Language10.1007/s10994-024-06618-8113:10(7565-7589)Online publication date: 23-Sep-2024
https://dl.acm.org/doi/10.1007/s10994-024-06618-8
Liu TBarnard AKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Shapley based residual decomposition for instance analysisProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619290(21375-21387)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619290
Show More Cited By

Index Terms

Detecting outlying properties of exceptional objects
1. Information systems
  1. Data management systems

Recommendations

Mining Optimized Association Rules with Categorical and Numeric Attributes

Mining association rules on large data sets has received considerable attention in recent years. Association rules are useful for determining correlations between attributes of a relation and have applications in marketing, financial, and retail ...
Dataless Transitions Between Concise Representations of Frequent Patterns

For many data mining problems in order to solve them it is required to discover frequent patterns. Frequent itemsets are useful e.g. in the discovery of association and episode rules, sequential patterns and clusters. Nevertheless, the number of ...
Mining N-most interesting itemsets without support threshold by the COFI-tree

Data mining is the discovery of interesting and hidden patterns from a large amount of collected data. Applications can be found in many organisations with large databases, for many different purposes such as customer relationships, marketing, planning, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Database Systems

ACM Transactions on Database Systems Volume 34, Issue 1

April 2009

349 pages

ISSN:0362-5915

EISSN:1557-4644

DOI:10.1145/1508857

Issue’s Table of Contents

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 April 2009

Accepted: 01 September 2008

Revised: 01 May 2008

Received: 01 September 2007

Published in TODS Volume 34, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

54
Total Citations
View Citations
783
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)1

Reflects downloads up to 04 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Angiulli FFassetti FFerragina L(2024)Data Mining: Outleir DetectionReference Module in Life Sciences10.1016/B978-0-323-95502-7.00011-7Online publication date: 2024
https://doi.org/10.1016/B978-0-323-95502-7.00011-7
Angiulli FFassetti FNisticò SPalopoli L(2024)Explaining outliers and anomalous groups via subspace density contrastive lossMachine Language10.1007/s10994-024-06618-8113:10(7565-7589)Online publication date: 23-Sep-2024
https://dl.acm.org/doi/10.1007/s10994-024-06618-8
Liu TBarnard AKrause ABrunskill ECho KEngelhardt BSabato SScarlett J(2023)Shapley based residual decomposition for instance analysisProceedings of the 40th International Conference on Machine Learning10.5555/3618408.3619290(21375-21387)Online publication date: 23-Jul-2023
https://dl.acm.org/doi/10.5555/3618408.3619290
Zhang ZWang BChen WWu YQin JChen PSun HHe A(2023)Recognition of Abnormal Individuals Based on Lightweight Deep Learning Using Aerial Images in Complex Forest Landscapes: A Case Study of Pine Wood NematodeRemote Sensing10.3390/rs1505118115:5(1181)Online publication date: 21-Feb-2023
https://doi.org/10.3390/rs15051181
Li ZZhu YVan Leeuwen M(2023)A Survey on Explainable Anomaly DetectionACM Transactions on Knowledge Discovery from Data10.1145/360933318:1(1-54)Online publication date: 6-Sep-2023
https://dl.acm.org/doi/10.1145/3609333
Trummer IDas SPandis ISelçuk Candan KAmer-Yahia S(2023)Demonstrating NaturalMiner: Searching Large Data Sets for Abstract Patterns Described in Natural LanguageCompanion of the 2023 International Conference on Management of Data10.1145/3555041.3589694(139-142)Online publication date: 4-Jun-2023
https://dl.acm.org/doi/10.1145/3555041.3589694
Angiulli FFassetti FNisticó SPalopoli L(2023)Counterfactuals Explanations for Outliers via Subspaces Density Contrastive LossDiscovery Science10.1007/978-3-031-45275-8_11(159-173)Online publication date: 9-Oct-2023
https://dl.acm.org/doi/10.1007/978-3-031-45275-8_11
Boukela LZhang GYacoub MBouzefrane SBaba Ahmadi S(2022)An approach for unsupervised contextual anomaly detection and characterizationIntelligent Data Analysis10.3233/IDA-21590626:5(1185-1209)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.3233/IDA-215906
Trummer I(2022)BABOONSProceedings of the VLDB Endowment10.14778/3551793.355184615:11(2980-2993)Online publication date: 29-Sep-2022
https://dl.acm.org/doi/10.14778/3551793.3551846
Yang C(2022)RETRACTED ARTICLE: Incremental Outlier Feature Clustering Algorithm in Blockchain Networks Based on Big Data AnalysisIETE Journal of Research10.1080/03772063.2022.2060876(1-9)Online publication date: 14-Apr-2022
https://doi.org/10.1080/03772063.2022.2060876
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents