article

Outlier detection for high dimensional data

Authors:

Charu C. Aggarwal,

Philip S. YuAuthors Info & Claims

ACM SIGMOD Record, Volume 30, Issue 2

Pages 37 - 46

https://doi.org/10.1145/376284.375668

Published: 01 May 2001 Publication History

Abstract

The outlier detection problem has important applications in the field of fraud detection, network robustness analysis, and intrusion detection. Most such applications are high dimensional domains in which the data can contain hundreds of dimensions. Many recent algorithms use concepts of proximity in order to find outliers based on their relationship to the rest of the data. However, in high dimensional space, the data is sparse and the notion of proximity fails to retain its meaningfulness. In fact, the sparsity of high dimensional data implies that every point is an almost equally good outlier from the perspective of proximity-based definitions. Consequently, for high dimensional data, the notion of finding meaningful outliers becomes substantially more complex and non-obvious. In this paper, we discuss new techniques for outlier detection which find the outliers by studying the behavior of projections from the data set.

References

[1]

C. C. Aggarwal. Re-designing Distance Functions and Distance Based Applications for High Dimensional Data. ACM SIGMOD Record, March 2001.]]

Digital Library

[2]

C. C. Aggarwal et al. Fast Algorithms for Projected Clustering. ACM SIGMOD Conference Proceedings, 1999.]]

Digital Library

[3]

C. C. Aggarwal, P. Yu. Finding Generalized Projected Clusters in High Dimensional Spaces. ACM SIGMOD Conference Proceedings, 2000.]]

Digital Library

[4]

C. C. Aggarwal, J. B. Orlin, R. P. Tai. Optimized Crossover for the Independent Set Problem. Operations Research 45(2), March 1997.]]

[5]

R. Agrawal, J. Gehrke, D. Gunopulos, P. Raghavan. Automatic Subspace Clustering of High Dimensional Data for Data Mining Applications. ACM SIGMOD Conference Proceedings, 1998.]]

Digital Library

[6]

R. Agrawal, T. Imielinski, A. Swami. Mining Association Rules Between Sets of Items in Large Databases. ACM SIGMOD Conference Proceedings, 1993.]]

Digital Library

[7]

A. Arning, R. Agrawal, P. Raghavan. A Linear Method for Deviation Detection in Large Databases. KDD Conference Proceedings, 1995.]]

[8]

V. Barnett, T. Lewis. Outliers in Statistical Data. John Wiley and Sons, NY 1994.]]

[9]

K. Beyer, J. Goldstein, R. Ramakrishnan, U. Shaft. When is Nearest Neighbors Meaningful? ICDT Conference Proceedings, 1999.]]

Digital Library

[10]

M. M. Breunig, H.-P. Kriegel, R. T. Ng, J. Sander. LOF: Identifying Density-Based Local Outliers. ACM SIGMOD Conference Proceedings, 2000.]]

Digital Library

[11]

K. Chakrabarti, S. Mehrotra. Local Dimensionality Reduction: A New Approach to Indexing High Dimensional Spaces. VLDB Conference Proceedings, 2000.]]

Digital Library

[12]

C. Darwin. The Origin of the Species by Natural Selection. Published, 1859.]]

[13]

D. Hawkins. Identification of Outliers, Chapman and Hall, London, 1980.]]

[14]

K. A. De Jong. Analysis of the Behavior of a Class of Genetic Adaptive Systems. Ph. D. Dissertation, University of Michigan, Ann Arbor, MI, 1975.]]

Digital Library

[15]

M. Ester, H.-P. Kriegel, J. Sander, X. Xu. A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise. KDD Conference Proceedings, 1996.]]

[16]

J. J. Grefenstette. Genesis Software Version 5.0. Available at http://www.santafe.edu.]]

[17]

D. E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning. Addison Wesley, Reading, MA, 1989.]]

Digital Library

[18]

S. Guha, R. Rastogi, K. Shim. CURE: An Efficient Clustering Algorithm for Large Databases. ACM SIGMOD Conference Proceedings, 1998.]]

Digital Library

[19]

A. Hinneburg, C. C. Aggarwal, D. A. Keim. What is the nearest neighbor in high dimensional spaces? VLDB Conference Proceedings, 2000.]]

Digital Library

[20]

J. H. Holland. Adaptation in Natural and Artificial Systems. University of Michigan Press, Ann Arbor MI 1975.]]

Digital Library

[21]

S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi. Optimization by Simulated Annealing. Science (220) (4589): pages 671-680, 1983.]]

[22]

E. Knorr, R. Ng. Algorithms for Mining Distance-based Outliers in Large Data Sets. VLDB Conference Proceedings, September 1998.]]

Digital Library

[23]

E. Knorr, R. Ng. Finding Intensional Knowledge of Distance-based Outliers. VLDB Conference Proceedings, 1999.]]

Digital Library

[24]

R. Ng, J. Han. Efficient and Effective Clustering Methods for Spatial Data Mining. VLDB Conference Proceedings, pages 144-155, 1994.]]

Digital Library

[25]

S. Ramaswamy, R. Rastogi, K. Shim. Efficient Algorithms for Mining Outliers from Large Data Sets. ACM SIGMOD Conference Proceedings, 2000.]]

Digital Library

[26]

S. Sarawagi, R. Agrawal, N. Meggido. Discovery Driven Exploration of OLAP Data Cubes. EDBT Conference Proceedings, 1998.]]

Digital Library

[27]

T. Zhang, R. Ramakrishnan, M. Livny. BIRCH: An Efficient Data Clustering Method for Very Large Databases. ACM SIGMOD Conference Proceedings, 1996.]]

Digital Library

Cited By

Mr. Abhale B. A Ansari Aiman Jamdar Omkar Dange Satyam (2024)Electricity Theft Detection in Smart Grids Using Sarimax and OCRInternational Journal of Advanced Research in Science, Communication and Technology10.48175/IJARSCT-18528(222-225)Online publication date: 26-May-2024
https://doi.org/10.48175/IJARSCT-18528
Gerloff CYücel MMehlem LKonrad KReindl V(2024)NiReject: toward automated bad channel detection in functional near-infrared spectroscopyNeurophotonics10.1117/1.NPh.11.4.04500811:04Online publication date: 1-Oct-2024
https://doi.org/10.1117/1.NPh.11.4.045008
Abd Majeed WHamad M(2024)Exploring High-Dimensional Outlier Detection: A Comprehensive Study on Methods and Applications Using PCA and k-NN Algorithm2024 21st International Multi-Conference on Systems, Signals & Devices (SSD)10.1109/SSD61670.2024.10548429(699-704)Online publication date: 22-Apr-2024
https://doi.org/10.1109/SSD61670.2024.10548429
Show More Cited By

Index Terms

Outlier detection for high dimensional data
1. Information systems
  1. Data management systems
    1. Database design and models
  2. Information retrieval
    1. Information retrieval query processing

Recommendations

Angle-based outlier detection in high-dimensional data
KDD '08: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining

Detecting outliers in a large set of data objects is a major data mining task aiming at finding different mechanisms responsible for different groups of objects in a data set. All existing approaches, however, are based on an assessment of distances (...
Outlier detection for high dimensional data
SIGMOD '01: Proceedings of the 2001 ACM SIGMOD international conference on Management of data

The outlier detection problem has important applications in the field of fraud detection, network robustness analysis, and intrusion detection. Most such applications are high dimensional domains in which the data can contain hundreds of dimensions. ...
Towards outlier detection for high-dimensional data streams using projected outlier analysis strategy

Comments

Information & Contributors

Information

Published In

cover image ACM SIGMOD Record

ACM SIGMOD Record Volume 30, Issue 2

June 2001

625 pages

ISSN:0163-5808

DOI:10.1145/376284

Editors:
Timos Sellis
National Technical Univ. of Athens
,
Sharad Mehrotra
Univ. of California at Irvine

Issue’s Table of Contents

SIGMOD '01: Proceedings of the 2001 ACM SIGMOD international conference on Management of data
May 2001
630 pages
ISBN:1581133324
DOI:10.1145/375663
Editors:
Timos Sellis,
Sharad Mehrotra

Copyright © 2001 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 May 2001

Published in SIGMOD Volume 30, Issue 2

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

824
Total Citations
View Citations
8,507
Total Downloads

Downloads (Last 12 months)280
Downloads (Last 6 weeks)34

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mr. Abhale B. A Ansari Aiman Jamdar Omkar Dange Satyam (2024)Electricity Theft Detection in Smart Grids Using Sarimax and OCRInternational Journal of Advanced Research in Science, Communication and Technology10.48175/IJARSCT-18528(222-225)Online publication date: 26-May-2024
https://doi.org/10.48175/IJARSCT-18528
Gerloff CYücel MMehlem LKonrad KReindl V(2024)NiReject: toward automated bad channel detection in functional near-infrared spectroscopyNeurophotonics10.1117/1.NPh.11.4.04500811:04Online publication date: 1-Oct-2024
https://doi.org/10.1117/1.NPh.11.4.045008
Abd Majeed WHamad M(2024)Exploring High-Dimensional Outlier Detection: A Comprehensive Study on Methods and Applications Using PCA and k-NN Algorithm2024 21st International Multi-Conference on Systems, Signals & Devices (SSD)10.1109/SSD61670.2024.10548429(699-704)Online publication date: 22-Apr-2024
https://doi.org/10.1109/SSD61670.2024.10548429
Musa HAhmad NNor A(2024)Expanding financial inclusion participation in Muslim-dominated communities through Islamic finance products (IFP) adoptionJournal of Islamic Accounting and Business Research10.1108/JIABR-09-2023-0308Online publication date: 14-Feb-2024
https://doi.org/10.1108/JIABR-09-2023-0308
Schrijver GSarmah DEl-hajj M(2024)Automobile insurance fraud detection using data mining: A systematic literature reviewIntelligent Systems with Applications10.1016/j.iswa.2024.20034021(200340)Online publication date: Mar-2024
https://doi.org/10.1016/j.iswa.2024.200340
Li ZFeng DLi J(2024) Outlier detection for heterogeneous data via fuzzy covering Expert Systems with Applications10.1016/j.eswa.2024.124227252(124227)Online publication date: Oct-2024
https://doi.org/10.1016/j.eswa.2024.124227
Calikus ENowaczyk SDikmen O(2024)Context discovery for anomaly detectionInternational Journal of Data Science and Analytics10.1007/s41060-024-00586-xOnline publication date: 18-Jun-2024
https://doi.org/10.1007/s41060-024-00586-x
Ghamry FEl-Banby GEl-Fishawy AEl-Samie FDessouky M(2024)A survey of anomaly detection techniquesJournal of Optics10.1007/s12596-023-01147-4Online publication date: 16-Feb-2024
https://doi.org/10.1007/s12596-023-01147-4
Trongnukul NMasomtob MFuengwarodsakul N(2024)Robust noise-correction recursive least square method for parameter identification of equivalent circuit model in battery management system using Bayes’ theorem-based preprocessing techniqueElectrical Engineering10.1007/s00202-024-02371-2Online publication date: 26-Apr-2024
https://doi.org/10.1007/s00202-024-02371-2
Dutschmann TSchlenker VBaumann K(2024)Chemoinformatic regression methods and their applicability domainMolecular Informatics10.1002/minf.20240001843:7Online publication date: 28-May-2024
https://doi.org/10.1002/minf.202400018
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents