research-article

Public Access

Surrogate optimization of deep neural networks for groundwater predictions

Authors:

Juliane Müller,

Charuleka Varadharajan,

Boris Faybishenko,

Deborah AgarwalAuthors Info & Claims

Journal of Global Optimization, Volume 81, Issue 1

Pages 203 - 231

https://doi.org/10.1007/s10898-020-00912-0

Published: 01 September 2021 Publication History

Abstract

Sustainable management of groundwater resources under changing climatic conditions require an application of reliable and accurate predictions of groundwater levels. Mechanistic multi-scale, multi-physics simulation models are often too hard to use for this purpose, especially for groundwater managers who do not have access to the complex compute resources and data. Therefore, we analyzed the applicability and performance of four modern deep learning computational models for predictions of groundwater levels. We compare three methods for optimizing the models’ hyperparameters, including two surrogate model-based algorithms and a random sampling method. The models were tested using predictions of the groundwater level in Butte County, California, USA, taking into account the temporal variability of streamflow, precipitation, and ambient temperature. Our numerical study shows that the optimization of the hyperparameters can lead to reasonably accurate performance of all models (root mean squared errors of groundwater predictions of 2 meters or less), but the “simplest” network, namely a multilayer perceptron (MLP) performs overall better for learning and predicting groundwater data than the more advanced long short-term memory or convolutional neural networks in terms of prediction accuracy and time-to-solution, making the MLP a suitable candidate for groundwater prediction.

References

[1]

Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M. et al.: Tensorflow: large-scale machine learning on heterogeneous distributed systems (2016). arXiv:1603.04467

[2]

Abramson MA, Audet C, Chrissis J, and Walston J Mesh adaptive direct search algorithms for mixed variable optimization Optim. Lett. 2009 3 35-47

[3]

Ali, Z., Hussain, I., Faisal, M., Nazir, H.M., Hussain, T., Shad, M.Y., Shoukry, A.M., Gani, S.H.: Forecasting drought using multilayer perceptron artificial neural network model. Adv. Meteorol., 5681308, 9 pages (2017)

[4]

Araujo P, Astray G, Ferrerio-Lage JA, Mejuto JC, Rodriguez-Suarez JA, and Soto B Multilayer perceptron neural network for flow prediction J. Environ. Monit. 2011 13 1 35-41

[5]

Audet C and Dennis JE Jr Mesh adaptive direct search algorithms for constrained optimization SIAM J. Optim. 2006 17 188-217

[6]

Audet C and Hare W Derivative-Free and Blackbox Optimization. Springer Series in Operations Research and Financial Engineering 2017 Berlin Springer

[7]

Audet C and Kokkolaras M Blackbox and derivative-free optimization: theory, algorithms and applications Optim. Eng. 2016 17 1 1-2

[8]

Audet C, Savard G, and Zghal W A mesh adaptive direct search algorithm for multiobjective optimization Eur. J. Oper. Res. 2010 204 3 545-556

[9]

Balaprakash, P., Salim, M., Uram, T.D., Vishwanath, V., Wild, S.M.: Deephyper: asynchronous hyperparameter search for deep neural networks. In: 2018 IEEE 25th International Conference on High Performance Computing (HiPC), pp. 42–51 (2018)

[10]

Bergstra J and Bengio Y Random search for hyper-parameter optimization J. Mach. Learn. Res. 2012 13 1 281-305

[11]

Bergstra, J., Yamins, D., Cox, D.D.: Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: Proceedings of the 30th International Conference on Machine Learning (2013)

[12]

Bishop CM et al. Neural networks for pattern recognition 1995 Oxford Oxford University Press

[13]

Booker AJ, Dennis JE Jr, Frank PD, Serafini DB, Torczon V, and Trosset MW A rigorous framework for optimization of expensive functions by surrogates Struct. Multidiscip. Optim. 1999 17 1-13

[14]

Borovykh, A., Bohte, S., Oosterlee, C.W.: Conditional Time Series Forecasting with Convolutional Neural Networks (2017). arXiv:1703.04691

[15]

Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: 19th International Conference on Computational Statistics, pp. 177–186 (2010)

[16]

California Department of Water Resources. SGMA groundwater management. https://www.waterboards.ca.gov/water_issues/programs/gmp/docs/sgma/sgma_20190101.pdf. Accessed 18 May 2020

[17]

Chiang Y-M, Chang L-C, and Chang F-J Comparison of static-feedforward and dynamic-feedback neural networks for rainfall-runoff modeling J. Hydrol. 2004 290 3–4 297-311

[18]

Chollet, F.: keras. GitHub Repository (2015). https://github.com/fchollet/keras. Accessed 18 May 2020

[19]

Cook BI, Mankin JS, and Anchukaitis KJ Climate change and drought: from past to future Curr. Clim. Change Rep. 2018 4 2 164-179

[20]

Coulibaly P, Anctil F, Aravena R, and Bobée B Artificial neural network modeling of water table depth fluctuations Water Resour. Res. 2001 37 4 885-896

[21]

Cui, Z., Chen, W., Chen, Y.: Multi-scale Convolutional Neural Networks for Time Series Classification (2016). arXiv:1603.06995

[22]

Daliakopoulos IN, Coulibaly P, and Tsanis IK Groundwater level forecasting using artificial neural networks J. Hydrol. 2005 309 1–4 229-240

[23]

Datta R and Regis RG A surrogate-assisted evolution strategy for constrained multi-objective optimization Expert Syst. Appl. 2016 57 270-284

[24]

Davis E and Ierapetritou M Kriging based method for the solution of mixed-integer nonlinear programs containing black-box functions J. Global Optim. 2009 43 191-205

[25]

Faunt, C.C.: Groundwater Availability of the Central Valley Aquifer, California. Professional paper 1766, 225 p., U.S. Geological Survey (2009). https://pubs.usgs.gov/pp/1766/PP_1766.pdf. Accessed 18 May 2020

[26]

Forrester AIJ, Sóbester A, and Keane AJ Multi-fidelity optimization via surrogate modelling Proc. R. Soc. 2007 463 3251-3269

[27]

Fortin F-A, De Rainville F-M, Gardner M-A, Gagné C, and Parizeau M DEAP: evolutionary algorithms made easy J. Mach. Learn. Res. 2012 13 2171-2175

[28]

Gardner MW and Dorling SR Artificial neural networks (the multilayer perceptron): a review of applications in the atmospheric sciences Atmos. Environ. 1998 32 14–15 2627-2636

[29]

Gers FA, Schmidhuber J, and Cummins F Learning to forget: continual prediction with LSTM Neural Comput. 2000 12 2451-2471

[30]

Gramacy R and Le Digabel S The mesh adaptive direct search algorithm with treed Gaussian process surrogates Pac. J. Optim. 2015 11 419-447

[31]

Graves, A., Mohamed, A., Hinton, G.: Speech recognition with deep recurrent neural networks. In: Proceedings of the 2013 International Conference on Acoustics, Speech, and Signal Processing (2013)

[32]

Gutmann H-M A radial basis function method for global optimization J. Global Optim. 2001 19 201-227

[33]

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

[34]

Hinton, G., Srivastava, N., Swersky, K.: Neural Networks for Machine Learning. lecture 6a, Overview of Mini-batch Gradient Descent. Lecture Notes (2012). https://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf. Accessed 18 May 2020

[35]

Hochreiter S and Schmidhuber J Long short-term memory Neural Comput. 1997 9 1735-1780

[36]

Holmström K An adaptive radial basis algorithm (ARBF) for expensive black-box mixed-integer global optimization J. Global Optim. 2008 9 311-339

[37]

Holmström K An adaptive radial basis algorithm (ARBF) for expensive black-box global optimization J. Global Optim. 2008 41 447-464

[38]

Hsu, D.: Multi-period Time Series Modeling with Sparsity Via Bayesian Variational Inference (2018). arXiv:1707.00666v3

[39]

Ilievski, I., Akhtar, T., Feng, J., Shoemaker, C.A.: Efficient hyperparameter optimization of deep learning algorithms using deterministic RBF surrogates. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (2017)

[40]

Jin, H., Song, Q., Hu, X.: Auto-Keras: An Efficient Neural Architecture Search System (2019). arXiv:1806.10282 [cs.LG]

[41]

Jones DR, Schonlau M, and Welch WJ Efficient global optimization of expensive black-box functions J. Global Optim. 1998 13 455-492

[42]

Karandish F and Šimunek J A comparison of numerical and machine-learning modeling of soil water content with limited input data J. Hydrol. 2016 543 892-909

[43]

Karslıoğlu O, Gehlmann M, Müller J, Nemšàk S, Sethian J, Kaduwela A, Bluhm H, and Fadley C An efficient algorithm for automatic structure optimization in x-ray standing-wave experiments J. Electron Spectrosc. Relat. Phenom. 2019 230 10-20

[44]

Kingma, D.P., Ba, J.L.: ADAM: a method for stochastic optimization. In: ICLR 2015 (2015)

[45]

Klein, A., Falkner, S., Bartels, S., Hennig, P., Hutter, F.: Fast Bayesian optimization of machine learning hyperparameters on large datasets. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS) 2017, Fort Lauderdale, Florida, USA, vol. 54 (2017)

[46]

Kratzert F, Klotz D, Brenner C, Schulz K, and Herrnegger M Rainfall-runoff modelling using long short-term memory (LSTM) networks Hydrol. Earth Syst. Sci. 2018 22 11 6005-6022

[47]

Kuderer, M., Gulati, S., Burgard, W.: Learning driving styles for autonomous vehicles from demonstration. In: 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 2641–2646 (2015)

[48]

Lakhmiri, D., Digabel, S. Le, Tribes, C.: HyperNOMAD: Hyperparameter Optimization of Deep Neural Networks Using Mesh Adaptive Direct Search (2019). arXiv:1907.01698 [cs.LG]

[49]

Langevin, C.D., Hughes, J.D., Banta, E.R., Niswonger, R.G., Panday, S., Provost, A.M.: Documentation for the MODFLOW 6 Groundwater Flow Model. Technical Report, US Geological Survey (2017)

[50]

Langhans W, Müller J, and Collins W Optimization of the Eddy-diffusivity/mass-flux shallow cumulus and boundary-layer parameterization using surrogate models J. Adv. Model. Earth Syst. 2019 11 402-416

[51]

Le Digabel S Algorithm 909: NOMAD–nonlinear optimization with the MADS algorithm ACM Trans. Math. Softw. 2011 37 1-15

[52]

LeCun Y, Bottou L, Bengio Y, Haffner P, et al. Gradient-based learning applied to document recognition Proc. IEEE 1998 86 11 2278-2324

[53]

LeCun Y, Bengio Y, and Hinton G Deep learning Nature 2015 521 436-444

[54]

Lee HKH, Gramacy RB, Linkletter C, and Gray GA Optimization subject to hidden constraints via statistical emulation Pac. J. Optim. 2011 7 467-478

[55]

Ma X, Tao Z, Wang Y, Yu H, and Wang Y Long short-term memory neural network for traffic speed prediction using remote microwave sensor data Transp. Res. C Emerg. Technol. 2015 54 187-197

[56]

Matheron G Principles of geostatistics Econ. Geol. 1963 58 1246-1266

[57]

Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., Khudanpur, S.: Recurrent neural network based language model. In: Eleventh Annual Conference of the International Speech Communication Association (2010)

[58]

Mitchell M An Introduction to Genetic Algorithms 1996 Cambridge MIT Press

[59]

Moritz S and Bartz-Beielstein T imputeTS: Time Series Missing Value Imputation in R R J. 2017 9 207-218

[60]

Müller J MISO: mixed integer surrogate optimization framework Optim. Eng. 2015 17 1 177-203

[61]

Müller J SOCEMO: surrogate optimization of computationally expensive multiobjective problems INFORMS J. Comput. 2017 29 4 581-596

[62]

Müller J An algorithmic framework for the optimization of computationally expensive bi-fidelity black-box problems INFOR Inf. Syst. Oper. Res. 2019

[63]

Müller J and Day M Surrogate optimization of computationally expensive black-box problems with hidden constraints INFORMS J. Comput. 2019

[64]

Müller J and Woodbury J GOSAC: global optimization with surrogate approximation of constraints J. Glob. Optim. 2017

[65]

Müller J, Shoemaker CA, and Piché R SO-MI: a surrogate model algorithm for computationally expensive nonlinear mixed-integer black-box global optimization problems Comput. Oper. Res. 2013 40 1383-1400

[66]

Müller J, Shoemaker CA, and Piché R SO-I: a surrogate model algorithm for expensive nonlinear integer programming problems including global optimization applications J. Glob. Optim. 2013 59 865-889

[67]

Müller J, Paudel R, Shoemaker CA, Woodbury J, Wang Y, and Mahowald N CH4 parameter estimation in CLM4.5bgc using surrogate global optimization Geosci. Model Dev. Discus. 2015 8 141-207

[68]

Myers RH, Montgomery DC, and Anderson-Cook CM Response Surface Methodology: Process and Product Optimization Using Designed Experiments 2016 4 Hoboken, NJ John Wiley & Sons, Inc.

[69]

Najah A, El-Shafie A, Karim OA, and El-Shafie AH Application of artificial neural networks for water quality prediction Neural Comput. Appl. 2013 22 1 187-201

[70]

Nuñez L, Regis RG, and Varela K Accelerated random search for constrained global optimization assisted by radial basis function surrogates J. Comput. Appl. Math. 2018 340 276-295

[71]

Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, and Duchesnay E Scikit-learn: machine learning in Python J. Mach. Learn. Res. 2011 12 2825-2830

[72]

Powell MJD Advances in Numerical Analysis, Vol. 2: Wavelets, Subdivision Algorithms and Radial Basis Functions. Oxford University Press, Oxford, pp. 105–210, Chapter The Theory of Radial Basis Function Approximation in 1990 1992 London Oxford University Press

[73]

Powell MJD Recent Research at Cambridge on Radial Basis Functions New Developments in Approximation Theory 1999 Basel Birkhäuser 215-232

[74]

Regis RG Stochastic radial basis function algorithms for large-scale optimization involving expensive black-box objective and constraint functions Comput. Oper. Res. 2011 38 837-853

[75]

Regis RG and Shoemaker CA A stochastic radial basis function method for the global optimization of expensive functions INFORMS J. Comput. 2007 19 497-509

[76]

Robbins H and Monro S A stochastic approximation method Ann. Math. Stat. 1951 22 3 400-407

[77]

Rudy S, Alla A, Brunton SL, and Kutz JN Data-driven identification of parametric partial differential equations SIAM J. Appl. Dyn. Syst. 2019 18 2 643-660

[78]

Rumelhart DE, Hinton GE, Williams RJ, et al. Learning representations by back-propagating errors Cognit. Model. 1988 5 3 1

[79]

Sahoo S, Russo TA, Elliott J, and Foster I Machine learning algorithms for modeling groundwater level changes in agricultural regions of the US Water Resour. Res. 2017 53 5 3878-3895

[80]

Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems (2012)

[81]

Steefel CI, Appelo CAJ, Arora B, Jacques D, Kalbacher T, Kolditz O, Lagneau V, Lichtner PC, Mayer KU, Meeussen JCL, et al. Reactive transport codes for subsurface environmental simulation Comput. Geosci. 2015 19 3 445-478

[82]

Sundermeyer, M., Schluter, R., Ney, H.: LSTM neural networks for language modeling. In: Proceedings of the 12th Annual Conference of the International Speech Communication Association, Portland, Oregon, USA, pp. 601–608 (2012)

[83]

Sutskever, I., Martens, J., Hinton, G.E.: Generating text with recurrent neural networks. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 1017–1024 (2011)

[84]

Tabari H and Talaee PH Multilayer perceptron for reference evapotranspiration estimation in a semiarid region Neural Comput. Appl. 2013 23 2 341-348

[85]

Taylor, M.: Liquid Assets: Improving Management of the State’s Groundwater Resources. Legislative Analyst’s Office, Technical Report (2010)

[86]

Toal D and Keane A Efficient multi-point aerodynamic design optimization via co-kriging J. Aircr. 2011 48 5 1685-1695

[87]

Trenn S Multilayer perceptrons: approximation order and necessary number of hidden units IEEE Trans. Neural Netw. 2008 19 5 836-844

[88]

Wild SM and Shoemaker CA Global convergence of radial basis function trust-region algorithms for derivative-free optimization SIAM Rev. 2013 55 349-371

[89]

Xu T, Spycher N, Sonnenthal E, Zhang G, Zheng L, and Pruess K TOUGHREACT version 2.0: a simulator for subsurface reactive transport under non-isothermal multiphase flow conditions Comput. Geosci. 2011 37 6 763-774

[90]

Young, S.R., Rose, D.C., Karnowski, T.P., Lim, S.-H., Patton, R.M.: Optimizing deep learning hyper-parameters through an evolutionary algorithm. In: MLHPC’15 Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments, Volume Article No. 4 (2015)

[91]

Zhang J, Zhu Y, Zhang X, Ye M, and Yang J Developing a long short-term memory (LSTM) based model for predicting water table depth in agricultural areas J. Hydrol. 2018 561 918-929

Cited By

Sanu SMath M(2024)Machine Learning-Based Water Management Strategies for Sustainable Groundwater ResourcesSN Computer Science10.1007/s42979-024-02686-85:4Online publication date: 27-Mar-2024
https://dl.acm.org/doi/10.1007/s42979-024-02686-8
Nezami NAnahideh H(2022)An Empirical Review of Model-Based Adaptive Sampling for Global Optimization of Expensive Black-Box FunctionsProceedings of the Winter Simulation Conference10.5555/3586210.3586478(3182-3193)Online publication date: 11-Dec-2022
https://dl.acm.org/doi/10.5555/3586210.3586478
Pacheco AFlesch RFlesch CIervolino LBarros V(2022)Tool based on artificial neural networks to obtain cooling capacity of hermetic compressors through tests performed in production linesExpert Systems with Applications: An International Journal10.1016/j.eswa.2021.116494194:COnline publication date: 15-May-2022
https://dl.acm.org/doi/10.1016/j.eswa.2021.116494
Show More Cited By

Index Terms

Surrogate optimization of deep neural networks for groundwater predictions

Index terms have been assigned to the content through auto-classification.

Recommendations

A machine learning framework for multi-step-ahead prediction of groundwater levels in agricultural regions with high reliance on groundwater irrigation
Abstract
This study presents a machine learning framework for multi-step-ahead prediction of groundwater levels in agricultural regions heavily reliant on groundwater irrigation. The framework utilizes a comprehensive set of predictive factors, including ...
Graphical abstract

Display Omitted
Highlights
- Achieving accurate predictions with optimal combination of input variables and delays.
- Overfitting reduced by optimizing hyperparameters, NSE increase: 11.6%–38.5%.
- Enhanced efficiency with multi-core parallel computing: 15.35-fold ...
Hybrid neural modeling for groundwater level prediction

The accurate prediction of groundwater level is important for the efficient use and management of groundwater resources, particularly in sub-humid regions where water surplus in monsoon season and water scarcity in non-monsoon season is a common ...
Long-term missing value imputation for time series data using deep neural networks
Abstract
We present an approach that uses a deep learning model, in particular, a MultiLayer Perceptron, for estimating the missing values of a variable in multivariate time series data. We focus on filling a long continuous gap (e.g., multiple months of ...

Comments

Information & Contributors

Information

Published In

cover image Journal of Global Optimization

Journal of Global Optimization Volume 81, Issue 1

Sep 2021

266 pages

ISSN:0925-5001

Issue’s Table of Contents

© This is a U.S. government work and its text is not subject to copyright protection in the United States; however, its text may be subject to foreign copyright protection 2020.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 September 2021

Accepted: 25 April 2020

Received: 27 August 2019

Author Tags

Qualifiers

Research-article

Funding Sources

Office of Science

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sanu SMath M(2024)Machine Learning-Based Water Management Strategies for Sustainable Groundwater ResourcesSN Computer Science10.1007/s42979-024-02686-85:4Online publication date: 27-Mar-2024
https://dl.acm.org/doi/10.1007/s42979-024-02686-8
Nezami NAnahideh H(2022)An Empirical Review of Model-Based Adaptive Sampling for Global Optimization of Expensive Black-Box FunctionsProceedings of the Winter Simulation Conference10.5555/3586210.3586478(3182-3193)Online publication date: 11-Dec-2022
https://dl.acm.org/doi/10.5555/3586210.3586478
Pacheco AFlesch RFlesch CIervolino LBarros V(2022)Tool based on artificial neural networks to obtain cooling capacity of hermetic compressors through tests performed in production linesExpert Systems with Applications: An International Journal10.1016/j.eswa.2021.116494194:COnline publication date: 15-May-2022
https://dl.acm.org/doi/10.1016/j.eswa.2021.116494
Zhai JBoukouvala F(2022)Data-driven spatial branch-and-bound algorithms for box-constrained simulation-based optimizationJournal of Global Optimization10.1007/s10898-021-01045-882:1(21-50)Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1007/s10898-021-01045-8
Ramu PThananjayan PAcar EBayrak GPark JLee I(2022)A survey of machine learning techniques in structural and multidisciplinary optimizationStructural and Multidisciplinary Optimization10.1007/s00158-022-03369-965:9Online publication date: 1-Sep-2022
https://dl.acm.org/doi/10.1007/s00158-022-03369-9

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents