research-article

BasisDetect: a model-based network event detection framework

Authors:

Brian Eriksson,

Matthew RoughanAuthors Info & Claims

IMC '10: Proceedings of the 10th ACM SIGCOMM conference on Internet measurement

Pages 451 - 464

https://doi.org/10.1145/1879141.1879200

Published: 01 November 2010 Publication History

Abstract

The ability to detect unexpected events in large networks can be a significant benefit to daily network operations. A great deal of work has been done over the past decade to develop effective anomaly detection tools, but they remain virtually unused in live network operations due to an unacceptably high false alarm rate. In this paper, we seek to improve the ability to accurately detect unexpected network events through the use of BasisDetect, a flexible but precise modeling framework. Using a small dataset with labeled anomalies, the BasisDetect framework allows us to define large classes of anomalies and detect them in different types of network data, both from single sources and from multiple, potentially diverse sources. Network anomaly signal characteristics are learned via a novel basis pursuit based methodology. We demonstrate the feasibility of our BasisDetect framework method and compare it to previous detection methods using a combination of synthetic and real-world data. In comparison with previous anomaly detection methods, our BasisDetect methodology results show a 50% reduction in the number of false alarms in a single node dataset, and over 65% reduction in false alarms for synthetic network-wide data.

References

[1]

A. Lakhina, M. Crovella, and C. Diot, "Diagnoising Network-Wide Traffic Anomalies," in Proceedings of ACM SIGCOMM Conference, Portland, OR, August 2004.

Digital Library

[2]

P. Chhabra, C. Scott, E. Kolaczyk, and M. Crovella, "Distributed Spatial Anomaly Detection," in Proceedings of IEEE INFOCOM Conference, Phoenix, AZ, March 2008.

[3]

A. Lakhina, M. Crovella, and C. Diot, "Characterization of Network-Wide Anomalies in Traffic Flows," in Proceedings of ACM SIGCOMM Internet Measurement Conference, Taormina, Sicily, Italy, October 2004.

Digital Library

[4]

A. Lakhina, K. Papagiannaki, M. Crovella, C. Diot, E. D. Kolaczyk, and N. Taft, "Structural Analysis of Network Traffic Flows," in ACM SIGMETRICS / Performance, 2004.

Digital Library

[5]

P. Barford, J. Kline, D. Plonka, and A. Ron, "A Signal Analysis of Network Traffic Anomalies," in Proceedings of ACM SIGCOMM Internet Measurement Workshop, Marseilles, France, November 2002.

Digital Library

[6]

Y. Zhang, Z. Ge, M. Roughan, and A. Greenberg, "Network Anomography," in Proceedings of the Internet Measurement Conference, Berkeley, CA, USA, October 2005.

Digital Library

[7]

M. Roughan, T. Griffin, M. Mao, A. Greenberg, and B. Freeman, "IP forwarding anomalies and improving their detection using multiple data sources," in ACM SIGCOMM Workshop on Network Troubleshooting (NetTS), Portland, OR, September 2004, pp. 307--312.

Digital Library

[8]

S. H. Steiner, "Grouped Data Exponentially Weighted Moving Average Control Charts," Applied Statistics, vol. 47, no. 2, 1998.

[9]

H. Ringberg, A. Soule, J. Rexford, and C. Diot, "Sensitivity of PCA for Traffic Anomaly Detection," in Proceedings of ACM SIGMETRICS Conference, San Diego, CA, June 2007.

Digital Library

[10]

B. Rubinstein, B. Nelson, L. Huang, A. Joseph, S. Lau, S. Rao, N. Taft, and J. Tygar, "ANTIDOTE: Understanding and Defending Against Poisoning of Anomaly Detectors," in Proceedings of ACM SIGCOMM Internet Measurements Conference, Chicago, Illinois, November 2009.

Digital Library

[11]

Y. Benjamini and Y. Hochberg, "Controlling the False Discovery Rate," in Journal of the Royal Statistical Society B, vol. 57, no. 1, 1995, pp. 289--300.

[12]

R. Miller, in Simultaneous Statistical Inference. Springer-Verlag, 1991.

[13]

L. Huang, X. Nguyen, M. Garofalakis, J. Hellerstein, M. Jordan, M.Joseph, and N.Taft., "Communication-Efficient Online Detection of Network-Wide Anomalies," in Proceedings of IEEE INFOCOM Conference, Anchorage, Alaska, May 2007.

[14]

Y. Liu, L. Zhang, and Y. Guan, "A Distributed Data Streaming Algorithm for Network-Wide Traffic Anomaly Detection," SIGMETRICS Performance Evaluation Review, vol. 37, no. 2, pp. 81--82, 2009.

Digital Library

[15]

C. Scott and E. Kolaczyk, "Nonparametric Assessment of Contamination in Multivariate Data using Generalized Quantile Sets and FDR," in Accepted for Publication in J. Computational and Graphical Statistics, 2007.

[16]

S. Chen, D. Donoho, and M. Saunders, "Atomic Decomposition by Basis Pursuit," in SIAM Journal of Scientific Computing, vol. 20, 1998, pp. 33--61.

Digital Library

[17]

G. Davis, S. Mallat, and M. Avellaneda, "Greedy Adaptive Approximation," in Journal of Constructive Approximation, vol. 13, 1997, pp. 57--98.

[18]

P. Huggins and S. Zucker, "Greedy Basis Pursuit," in IEEE Transactions on Signal Processing, vol. 55, no. 7, July 2007, pp. 3760--3771.

[19]

H. Ringberg, M. Roughan, and J. Rexford, "The Need For Simulation In Evaluating Anomaly Detectors," ACM SIGCOMM CCR, vol. 38, no. 1, pp. 55--59, January 2008.

Digital Library

[20]

Y. Zhang, M. Roughan, W. Willinger, and L. Qui, "Spatio-Temporal Compressive Sensing and Internet Traffic Matrices," in Proceedings of ACM SIGCOMM Conference, Barcellona, Spain, August 2009, pp. 267--278.

Digital Library

[21]

"Geant Project Website," http://www.geant.net/.

[22]

A. Markopoulou, G. Iannaccone, S. Bhattacharrya, C.-N. Chuah, and C. Diot, "Characterization of failures in an IP backbone," in Proceedings of IEEE INFOCOM Conference, Hong Kong, China, March 2004.

[23]

D. Oppenheimer, A. Ganapathi, and D. A. Patterson, "Why do Internet services fail, and what can be done about it?" in 4th Usenix Symposium on Internet Technologies and Systems (USITS'03), 2003.

Digital Library

[24]

L. Wasserman, "All of Nonparametric Statistics," in Springer Texts, 2006.

Digital Library

Cited By

Lin WLiu WChen GWu SFu JLiang XLing SChen Z(2021)Network Telemetry by Observing and Recording on Programmable Data Plane2021 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking52078.2021.9472807(1-6)Online publication date: 21-Jun-2021
https://doi.org/10.23919/IFIPNetworking52078.2021.9472807
Ling SLiu WZhu YTan MHuang JGuo ZLin W(2021)FullSight: Towards Scalable, High-Coverage, and Fine-grained Network Telemetry2021 17th International Conference on Mobility, Sensing and Networking (MSN)10.1109/MSN53354.2021.00050(263-270)Online publication date: Dec-2021
https://doi.org/10.1109/MSN53354.2021.00050
Zhou YSun CLiu HMiao RBai SLi BZheng ZZhu LShen ZXi YZhang PCai DZhang MXu M(2020)Flow Event Telemetry on Programmable Data PlaneProceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication10.1145/3387514.3406214(76-89)Online publication date: 30-Jul-2020
https://doi.org/10.1145/3387514.3406214
Show More Cited By

Index Terms

BasisDetect: a model-based network event detection framework
1. Networks
  1. Network services
    1. Network monitoring

Recommendations

Deep learning for anomaly detection in multivariate time series: Approaches, applications, and challenges
Abstract
Anomaly detection has recently been applied to various areas, and several techniques based on deep learning have been proposed for the analysis of multivariate time series. In this study, we classify the anomalies into three types, ...
Highlights
- The methods for anomaly detection on multivariate time series are reviewed.
- The ...
Multi-stage change-point detection scheme for large-scale simultaneous events

Change-point detection schemes, which represent one type of anomaly detection schemes, are a promising approach for detecting network anomalies, such as attacks and epidemics by unknown viruses and worms. These events are detected as change-points. ...
Two-stage anomaly detection algorithm via dynamic community evolution in temporal graph
Abstract
Detecting anomalies from a massive amount of user behavioral data is often liken to finding a needle in a haystack. While tremendous efforts have been devoted to anomaly detection from temporal graphs, existing studies rarely consider community ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IMC '10: Proceedings of the 10th ACM SIGCOMM conference on Internet measurement

November 2010

496 pages

ISBN:9781450304832

DOI:10.1145/1879141

Program Chair:
Mark Allman
International Computer Science Institute

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

USENIX Assoc: USENIX Assoc

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tag

anomaly detection

Qualifiers

Research-article

Conference

IMC '10

Sponsor:

IMC '10: Internet Measurement Conference

November 1 - 30, 2010

Melbourne, Australia

Acceptance Rates

Overall Acceptance Rate 277 of 1,083 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

20
Total Citations
View Citations
476
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)1

Reflects downloads up to 27 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lin WLiu WChen GWu SFu JLiang XLing SChen Z(2021)Network Telemetry by Observing and Recording on Programmable Data Plane2021 IFIP Networking Conference (IFIP Networking)10.23919/IFIPNetworking52078.2021.9472807(1-6)Online publication date: 21-Jun-2021
https://doi.org/10.23919/IFIPNetworking52078.2021.9472807
Ling SLiu WZhu YTan MHuang JGuo ZLin W(2021)FullSight: Towards Scalable, High-Coverage, and Fine-grained Network Telemetry2021 17th International Conference on Mobility, Sensing and Networking (MSN)10.1109/MSN53354.2021.00050(263-270)Online publication date: Dec-2021
https://doi.org/10.1109/MSN53354.2021.00050
Zhou YSun CLiu HMiao RBai SLi BZheng ZZhu LShen ZXi YZhang PCai DZhang MXu M(2020)Flow Event Telemetry on Programmable Data PlaneProceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication10.1145/3387514.3406214(76-89)Online publication date: 30-Jul-2020
https://doi.org/10.1145/3387514.3406214
Erhan DAnarim E(2020)Hybrid DDoS Detection Framework Using Matching Pursuit AlgorithmIEEE Access10.1109/ACCESS.2020.30057818(118912-118923)Online publication date: 2020
https://doi.org/10.1109/ACCESS.2020.3005781
Fan XXu X(2020)Sparse representation for network traffic recoveryComputer Communications10.1016/j.comcom.2020.07.003Online publication date: Jul-2020
https://doi.org/10.1016/j.comcom.2020.07.003
Meng FLi HLiu YKong JLu BYang ZRen S(2019)A STFT-Based Traffic Acquirement Algorithm to Remote Smart Managements of Optical Fiber Cores2019 IEEE International Conference on Industrial Internet (ICII)10.1109/ICII.2019.00037(143-147)Online publication date: Nov-2019
https://doi.org/10.1109/ICII.2019.00037
Li HMeng FKong JWang GLu BLiu YLi T(2019)An ARMA-Based Traffic Abnormal Identification Approach for Remote Smart Scheduling of Optical Fiber Cores2019 IEEE International Conference on Industrial Internet (ICII)10.1109/ICII.2019.00034(126-131)Online publication date: Nov-2019
https://doi.org/10.1109/ICII.2019.00034
Li HMeng FLiu YWang GWang D(2019)A BPNN-Based Traffic Anomaly Detection Method for Remote Managements of Optical Fiber Cores2019 IEEE International Conference on Industrial Internet (ICII)10.1109/ICII.2019.00033(120-125)Online publication date: Nov-2019
https://doi.org/10.1109/ICII.2019.00033
Lu YLi HLu BZhao YWang DGong XWei X(2019)Network Traffic Model with Multi-fractal Discrete Wavelet Transform in Power Telecommunication Access NetworksSimulation Tools and Techniques10.1007/978-3-030-32216-8_5(53-62)Online publication date: 24-Oct-2019
https://doi.org/10.1007/978-3-030-32216-8_5
Din MQazi S(2018)A compressed framework for monitoring and anomaly detection in cloud networks2018 International Conference on Computing, Mathematics and Engineering Technologies (iCoMET)10.1109/ICOMET.2018.8346394(1-7)Online publication date: Mar-2018
https://doi.org/10.1109/ICOMET.2018.8346394
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents