research-article

TriD-MAE: A Generic Pre-trained Model for Multivariate Time Series with Missing Values

Authors:

Qinmin YangAuthors Info & Claims

CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

Pages 3164 - 3173

https://doi.org/10.1145/3583780.3615097

Published: 21 October 2023 Publication History

Abstract

Multivariate time series(MTS) is a universal data type related to various real-world applications. Data imputation methods are widely used in MTS applications to deal with the frequent data missing problem. However, these methods inevitably introduce biased imputation and training-redundancy problems in downstream training. To address these challenges, we propose TriD-MAE, a generic pre-trained model for MTS data with missing values. Firstly, we introduce TriD-TCN, an end-to-end module based on TCN that effectively extracts temporal features by integrating dynamic kernel mechanisms and a time-flipping trick. Building upon that, we designed an MAE-based pre-trained model as the precursor of specialized downstream models. Our model cooperates with a dynamic positional embedding mechanism to represent the missing information and generate transferable representation through our proposed encoder units. The overall mixed data feed-in strategy and weighted loss function are established to ensure adequate training of the whole model. Comparative experiment results in time series prediction and classification manifest that our TriD-MAE model outperforms the other state-of-the-art methods within six real-world datasets. Moreover, ablation and interpretability experiments are delivered to verify the validity of TriD-MAE's

References

[1]

Arthur Asuncion and David Newman. 2007. UCI machine learning repository.

[2]

Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. 2018. The UEA multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 (2018).

[3]

Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 2018. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018).

[4]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. 2020. Language models are few-shot learners. Advances in neural information processing systems, Vol. 33 (2020), 1877--1901.

[5]

Luis Candanedo. 2017. Appliances energy prediction. UCI Machine Learning Repository.

[6]

Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. 2018. Brits: Bidirectional recurrent imputation for time series. Advances in neural information processing systems, Vol. 31 (2018).

[7]

Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. 2018. Recurrent neural networks for multivariate time series with missing values. Scientific reports, Vol. 8, 1 (2018), 1--12.

[8]

Song Chen. 2019. Beijing Multi-Site Air-Quality Data. UCI Machine Learning Repository.

[9]

Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. 2020. Dynamic convolution: Attention over convolution kernels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11030--11039.

[10]

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 (2014).

Digital Library

[11]

Andrea Cini, Ivan Marisca, and Cesare Alippi. 2021. Filling the G_ap_s: Multivariate Time Series Imputation by Graph Neural Networks. In International Conference on Learning Representations.

[12]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).

[13]

A Rogier T Donders, Geert JMG Van Der Heijden, Theo Stijnen, and Karel GM Moons. 2006. A gentle introduction to imputation of missing values. Journal of clinical epidemiology, Vol. 59, 10 (2006), 1087--1091.

[14]

Wei Fan, Shun Zheng, Xiaohan Yi, Wei Cao, Yanjie Fu, Jiang Bian, and Tie-Yan Liu. 2022. DEPTS: deep expansion learning for periodic time series forecasting. arXiv preprint arXiv:2203.07681 (2022).

[15]

Jean-Yves Franceschi, Aymeric Dieuleveut, and Martin Jaggi. 2019. Unsupervised Scalable Representation Learning for Multivariate Time Series. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2019/file/53c6de78244e9f528eb3e1cda69699bb-Paper.pdf

[16]

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2021. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 (2021).

[17]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, Vol. 9, 8 (1997), 1735--1780.

[18]

Kathan Kashiparekh, Jyoti Narwariya, Pankaj Malhotra, Lovekesh Vig, and Gautam Shroff. 2019. ConvTimeNet: A Pre-trained Deep Convolutional Neural Network for Time Series Classification. In 2019 International Joint Conference on Neural Networks (IJCNN). 1--8. https://doi.org/10.1109/IJCNN.2019.8852105

[19]

Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451 (2020).

[20]

.Ibrahim Kök and Suat Özdemir. 2020. Deepmdp: A novel deep-learning-based missing data prediction protocol for iot. IEEE Internet of Things Journal, Vol. 8, 1 (2020), 232--243.

[21]

Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. 95--104.

Digital Library

[22]

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. https://doi.org/10.48550/arXiv.1907.11692 arXiv:1907.11692 [cs].

[23]

Yonghong Luo, Ying Zhang, Xiangrui Cai, and Xiaojie Yuan. 2019. E2gan: End-to-end generative adversarial network for multivariate time series imputation. In Proceedings of the 28th international joint conference on artificial intelligence. AAAI Press, 3094--3100.

[24]

Pankaj Malhotra, Vishnu TV, Lovekesh Vig, Puneet Agarwal, and Gautam Shroff. 2017. TimeNet: Pre-trained deep recurrent neural network for time series classification. arXiv preprint arXiv:1706.08838 (2017).

[25]

Jingci Ming, Le Zhang, Wei Fan, Weijia Zhang, Yu Mei, Weicen Ling, and Hui Xiong. 2022. Multi-Graph Convolutional Recurrent Network for Fine-Grained Lane-Level Traffic Flow Imputation. In 2022 IEEE International Conference on Data Mining (ICDM). IEEE, 348--357.

[26]

Vladimiro Miranda, Jakov Krstulovic, Hrvoje Keko, Cristiano Moreira, and Jorge Pereira. 2011. Reconstructing missing data in state estimation with autoencoders. IEEE Transactions on power systems, Vol. 27, 2 (2011), 604--611.

[27]

Vladimiro Miranda, Jakov Krstulovic, Hrvoje Keko, Cristiano Moreira, and Jorge Pereira. 2012. Reconstructing Missing Data in State Estimation With Autoencoders. IEEE Transactions on Power Systems, Vol. 27, 2 (2012), 604--611. https://doi.org/10.1109/TPWRS.2011.2174810

[28]

Zhuofu Pan, Yalin Wang, Kai Wang, Hongtian Chen, Chunhua Yang, and Weihua Gui. 2022. Imputation of Missing Values in Time Series Using an Adaptive-Learned Median-Filled Deep Autoencoder. IEEE Transactions on Cybernetics (2022), 1--12. https://doi.org/10.1109/TCYB.2022.3167995

[29]

Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. 2017. A dual-stage attention-based recurrent neural network for time series prediction. arXiv preprint arXiv:1704.02971 (2017).

Digital Library

[30]

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. (2018).

[31]

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, Vol. 1, 8 (2019), 9.

[32]

Francisco Romeu-Guallart, Pablo Zamora-Martinez. 2014. SML2010. UCI Machine Learning Repository.

[33]

Donald B Rubin. 1976. Inference and missing data. Biometrika, Vol. 63, 3 (1976), 581--592.

[34]

Joseph L Schafer. 1999. Multiple imputation: a primer. Statistical methods in medical research, Vol. 8, 1 (1999), 3--15.

[35]

Shun-Yao Shih, Fan-Keng Sun, and Hung-yi Lee. 2019. Temporal pattern attention for multivariate time series forecasting. Machine Learning, Vol. 108, 8 (2019), 1421--1441.

Digital Library

[36]

Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Charu Aggarwal, Prasenjit Mitra, and Suhang Wang. 2020. Joint modeling of local and global temporal dynamics for multivariate time series forecasting with missing values. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 5956--5963.

[37]

Roman Tkachenko, Ivan Izonin, Natalia Kryvinska, Ivanna Dronyuk, and Khrystyna Zub. 2020. An approach towards increasing prediction accuracy for the recovery of missing IoT data based on the GRNN-SGTM ensemble. Sensors, Vol. 20, 9 (2020), 2625.

[38]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, Vol. 30 (2017).

[39]

Wang Weihan. 2020. MAGAN: A masked autoencoder generative adversarial network for processing missing IoT sequence data. Pattern Recognition Letters, Vol. 138 (2020), 211--216.

[40]

Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, Vol. 34 (2021), 22419--22430.

[41]

Jinsung Yoon, James Jordon, and Mihaela Schaar. 2018. Gain: Missing data imputation using generative adversarial nets. In International conference on machine learning. PMLR, 5689--5698.

[42]

George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021. A Transformer-Based Framework for Multivariate Time Series Representation Learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Virtual Event, Singapore) (KDD '21). Association for Computing Machinery, New York, NY, USA, 2114--2124. https://doi.org/10.1145/3447548.3467401

Digital Library

[43]

Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 11106--11115.

Cited By

Yu CWang FShao ZQian TZhang ZWei WXu YBaeza-Yates RBonchi F(2024)GinAR: An End-To-End Multivariate Time Series Forecasting Model Suitable for Variable MissingProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3672055(3989-4000)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3672055
Ma QLiu ZZheng ZHuang ZZhu SYu ZKwok J(2024)A Survey on Time-Series Pre-Trained ModelsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.347580936:12(7536-7555)Online publication date: Dec-2024
https://doi.org/10.1109/TKDE.2024.3475809
Zhou QHe SLiu HChen JMeng W(2024)Label-Free Multivariate Time Series Anomaly DetectionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.334961336:7(3166-3179)Online publication date: Jul-2024
https://doi.org/10.1109/TKDE.2024.3349613
Show More Cited By

Index Terms

TriD-MAE: A Generic Pre-trained Model for Multivariate Time Series with Missing Values

Recommendations

Deep imputation of missing values in time series health data: A review with benchmarking
Abstract
The imputation of missing values in multivariate time series (MTS) data is critical in ensuring data quality and producing reliable data-driven predictive models. Apart from many statistical approaches, a few recent studies have proposed state-of-...
Graphical abstract

Display Omitted
Handling missing data in multivariate time series using a vector autoregressive model-imputation (VAR-IM) algorithm

Imputing missing data from a multivariate time series dataset remains a challenging problem. There is an abundance of research on using various techniques to impute missing, biased, or corrupted values to a dataset. While a great amount of work has been ...
A time series continuous missing values imputation method based on generative adversarial networks
Abstract
Generative adversarial networks (GANs) have been widely utilized in time series analysis and modeling, wherein generators and discriminators interact to generate realistic data. However, when addressing the challenge of imputing continuous ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '23: Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

October 2023

5508 pages

ISBN:9798400701245

DOI:10.1145/3583780

General Chairs:
Ingo Frommholz
University of Wolverhampton, UK
,
Frank Hopfgartner
University of Koblenz, Germany
,
Mark Lee
University of Birmingham, UK
,
Michael Oakes
University of Birmingham, UK
,
Program Chairs:
Mounia Lalmas
Spotify, UK
,
Min Zhang
Tsinghua University, China
,
Rodrygo Santos
Federal University of Minas Gerais, Brazil

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 October 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Fundamental Research Funds for the Central Universities
Key Research and Development Program of Zhejiang Province
National Natural Science Foundation of China

Conference

CIKM '23

Sponsor:

CIKM '23: The 32nd ACM International Conference on Information and Knowledge Management

October 21 - 25, 2023

Birmingham, United Kingdom

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
327
Total Downloads

Downloads (Last 12 months)252
Downloads (Last 6 weeks)23

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yu CWang FShao ZQian TZhang ZWei WXu YBaeza-Yates RBonchi F(2024)GinAR: An End-To-End Multivariate Time Series Forecasting Model Suitable for Variable MissingProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3672055(3989-4000)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3672055
Ma QLiu ZZheng ZHuang ZZhu SYu ZKwok J(2024)A Survey on Time-Series Pre-Trained ModelsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.347580936:12(7536-7555)Online publication date: Dec-2024
https://doi.org/10.1109/TKDE.2024.3475809
Zhou QHe SLiu HChen JMeng W(2024)Label-Free Multivariate Time Series Anomaly DetectionIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2024.334961336:7(3166-3179)Online publication date: Jul-2024
https://doi.org/10.1109/TKDE.2024.3349613
Xiang YLi CHe SChen J(2024)AGC-ODE: Adaptive Graph Controlled Neural ODE for Human Mobility PredictionIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2024.344716125:11(18449-18460)Online publication date: Nov-2024
https://doi.org/10.1109/TITS.2024.3447161
Liu CHe SLiu HChen JDong H(2024)WindTrans: Transformer-Based Wind Speed Forecasting Method for High-Speed RailwayIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.333715025:6(4947-4963)Online publication date: Jun-2024
https://doi.org/10.1109/TITS.2023.3337150
Li YChen XZhu R(2024)Deep learning-based clustering method for single-cell RNA data2024 7th International Symposium on Autonomous Systems (ISAS)10.1109/ISAS61044.2024.10552501(1-6)Online publication date: 7-May-2024
https://doi.org/10.1109/ISAS61044.2024.10552501
Zhong XXiang YYi FLi CYang Q(2024)HMP-LLM: Human Mobility Prediction Based on Pre-trained Large Language Models2024 IEEE 4th International Conference on Digital Twins and Parallel Intelligence (DTPI)10.1109/DTPI61353.2024.10778764(687-692)Online publication date: 18-Oct-2024
https://doi.org/10.1109/DTPI61353.2024.10778764
Zhu CPu YLyu ZYang KYang Q(2024)Multidimensional information fusion and broad learning system-based condition recognition for energy pipeline safetyKnowledge-Based Systems10.1016/j.knosys.2024.112259300(112259)Online publication date: Sep-2024
https://doi.org/10.1016/j.knosys.2024.112259
Zhu CPu YYang YLyu ZLi CYang Q(2024)Localizing and tracking of in-pipe inspection robots based on distributed optical fiber sensingAdvanced Engineering Informatics10.1016/j.aei.2024.10242460(102424)Online publication date: Apr-2024
https://doi.org/10.1016/j.aei.2024.102424
Wang SHu QChen XLi CYang Q(2023)NTDformer: A Multi-Scale Forecasting Model for Non-Stationary Multilevel Time Series2023 5th International Conference on Smart Power & Internet Energy Systems (SPIES)10.1109/SPIES60658.2023.10474846(105-110)Online publication date: 1-Dec-2023
https://doi.org/10.1109/SPIES60658.2023.10474846

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents