research-article

Open access

Fast and Scalable Low-Order Implicit Unstructured Finite-Element Solver for Earth's Crust Deformation Problem

Authors:

Tsuyoshi Ichimura,

Kentaro Koyama,

Lalith MaddegedaraAuthors Info & Claims

PASC '17: Proceedings of the Platform for Advanced Scientific Computing Conference

Article No.: 11, Pages 1 - 10

https://doi.org/10.1145/3093172.3093236

Published: 26 June 2017 Publication History

Abstract

A low-order implicit unstructured finite-element solver for Earth's crust deformation problem is proposed. The solver scales up to 294,912 CPU cores of K computer with 94.8% size-up efficiency, enabling solving a 504 billion degrees-of-freedom crust deformation problem with 126 billion second-order tetrahedral elements in 146.5 s at 0.970 PFLOPS (20.6% of peak) on 294,912 CPU cores. This is 11.1 times faster than a highly-tuned preconditioned conjugate gradient solver and 2.18 times faster than the state-of-the-art SC14 Gordon Bell Prize Finalist solver. With the proposed solver, we solved a practical 381 billion degrees-of-freedom Eastern Japan crust deformation problem (576 km x 800 km x 400 km area; resolution of 62.5 m) in 252.2 s using 294,912 CPU cores. Such high-resolution analysis assures the convergence of stress and strain required to evaluate nonlinear material constitutive models at plate boundaries and is expected to become a key tool for physics-based earthquake forecasting.

References

[1]

Dimitri Komatitsch, Seiji Tsuboi, Chen Ji and Jeroen Tromp, A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator, in: Proceedings of the ACM / IEEE Supercomputing SC'2003 conference, p. 4--11, (2003).

Digital Library

[2]

T. Tiankai, Y. Hongfeng, L. Ramirez-Guzman, J. Bielak, O. Ghattas, M. Kwan-Liu, and D. R. O'Hallaron, "From mesh generation to scientific visualization: an end-to-end approach to parallel supercomputing," Proceedings of the 2006 ACM/IEEE conference on Supercomputing (SC '06). ACM, New York, NY, USA, 2006, Article 91.

Digital Library

[3]

Laura Carrington, Dimitri Komatitsch, Michael Laurenzano, Mustafa Tikir, David Michea, Nicolas Le Goff, Allan Snavely and Jeroen Tromp, High-frequency simulations of global seismic wave propagation using SPECFEM3D_GLOBE, in: Proceedings of the ACM / IEEE Supercomputing SC'2008 conference, article #60, p. 60:1--60:11, (2008).

Digital Library

[4]

Y. Cui, K. B. Olsen, T. H. Jordan, K. Lee, J. Zhou, P. Small, D. Roten, G. Ely, D. K. Panda, A. Chourasia, J. Levesque, S. M. Day and P. Maechling, "Scalable Earthquake Simulation on Petascale Supercomputersm," Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis (SC '10). IEEE Computer Society, Washington, DC, USA, 2010, pp.1--20.

Digital Library

[5]

M. Rietmann, P. Messmer, T. Nissen-Meyer, D. Peter, P. Basini, D. Komatitsch, O. Schenk, J. Tromp, L. Boschi, and D. Giardini. "Forward and adjoint simulations of seismic wave propagation on emerging large-scale GPU architectures," Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC '12). IEEE Computer Society Press, Los Alamitos, CA, USA, 2012, Article 38, 11 pages.

Digital Library

[6]

Y. Cui, E. Poyraz, K. B. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin, C. Guest, D. Choi, A. Chourasia, Z. Shi, S. M. Day, P. J. Maechling and T.H. Jordan. "Physics-based seismic hazard analysis on petascale heterogeneous supercomputers," Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, (SC'13). IEEE Computer Society Press, New York, NY, USA, 2013, Article 70, 12 pages.

Digital Library

[7]

A. Heinecke, A. Breuer, S. Rettenberger, M. Bader, A-A. Gabriel, C. Pelties, A. Bode, W. Barth, X-K. Liao, K. Vaidyanathan, M. Smelyanskiy, P. Dubey, "Petascale High Order Dynamic Rupture Earthquake Simulations on Heterogeneous Supercomputers," Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, (SC'14), pp 3--14, 2014.

Digital Library

[8]

T. Ichimura, K. Fujita, S. Tanaka, M. Hori, M. Lalith, Y. Shizawa, and H. Kobayashi. "Physics-based urban earthquake simulation enhanced by 10.7 BlnDOF x 30 K time-step unstructured FE non-linear seismic wave simulation," Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, (SC'14), pp 15--26, 2014.

Digital Library

[9]

T. Ichimura, K. Fujita, P.E.B. Quinay, L. Maddegedara, M. Hori, S. Tanaka, Y. Shizawa, H. Kobayashi, and K. Minami, "Implicit nonlinear wave simulation with 1.08T DOF and 0.270T unstructured finite elements to enhance comprehensive earthquake simulation," Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, (SC'15), 2015.

Digital Library

[10]

Daniel Roten, Yifeng Cui, Kim B. Olsen, Steven M. Day, Kyle Withers, William H. Savran, Peng Wang, and Dawei Mu. 2016. High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC '16). IEEE Press, Piscataway, NJ, USA, Article 82, 12 pages.

Digital Library

[11]

Heinecke A., Breuer A., Bader M., Dubey P. (2016) High Order Seismic Simulations on the Intel Xeon Phi Processor (Knights Landing). In: Kunkel J., Balaji P., Dongarra J. (eds) High Performance Computing. ISC High Performance 2016. Lecture Notes in Computer Science, vol 9697. Springer, Cham.

[12]

T. Hori, M. Hyodo, R. Nakata, S. Miyazaki, and Y. Kaneda, "A forecasting procedure for plate boundary earthquakes based on sequential data assimilation," Oceanography 27(2), pp.94--102, 2014.

[13]

Takane Hori, Mamoru Hyodo, Shin'ichi Miyazaki, Yoshiyuki Kaneda, Numerical forecasting of the time interval between successive M8 earthquakes along the Nankai Trough, southwest Japan, using ocean bottom cable network data, Marine Geophysical Research, 2014, 35 pp 285--294.

[14]

H. Miyazaki, Y. Kusano, N. Shinjou, F. Shoji, M. Yokokawa, and T. Watanabe. "Overview of the K computer system," FUJITSU Sci. Tech. J., 48, 3, 302--309, 2012.

[15]

Top 500, {Online}. https://www.top500.org/

[16]

Sequoia, {Online}. http://computation.llnl.gov/computers/sequoia

[17]

J. Rudi, A. Cristiano I. Malossi, T. Isaac, G. Stadler, M. Gurnis, P. W. J. Staar, Y. Ineichen, C. Bekas, A. Curioni, O. Ghattas, "An Extreme-Scale Implicit Solver for Complex PDEs: Highly Heterogeneous Flow in Earth's Mantle," Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis, (SC'15), 2015.

Digital Library

[18]

T. Ichimura, R. Agata, T. Hori, K. Hirahara, C. Hashimoto, M. Hori, and Y. Fukahata, "An elastic/viscoelastic finite element analysis method for crustal deformation using a 3D island-scale high-fidelity model," Geophysical Journal International. (in advance access, first published online April 4, 2016).

[19]

G. H. Golub and Q. Ye Inexact conjugate gradient method with inner-outer iteration. SIAM., Journal on Scientific Computing 21(4), 1305--1320, 1997.

Digital Library

[20]

J. M. Winget and T. J. R. Hughes, "Solution algorithms for nonlinear transient heat conduction analysis employing element-by-element iterative strategies," Computer Methods in Applied Mechanics and Engineering, 52, 711--815, 1985.

[21]

Y. Ajima, T. Inoue, S. Hiramoto and T. Shimizu. "Tofu: interconnect for the K computer," FUJITSU Sci. Tech. J., Vol.48, No.3, pp.280--285, 2012.

[22]

OpenMPI, {Online}. http://www.open-mpi.org/

[23]

METIS 5.1.0, {Online}. http://glaros.dtc.umn.edu/gkhome/metis/metis/overview

[24]

Japan Seismic Hazard Information Station (J-SHIS) {Online}. http://www.j-shis.bosai.go.jp/.

Cited By

Murakami SHashima AIinuma TFujita KIchimura THori T(2024)Detectability of low-viscosity zone along lithosphere–asthenosphere boundary beneath the Nankai Trough, Japan, based on high-fidelity viscoelastic simulationEarth, Planets and Space10.1186/s40623-024-02008-576:1Online publication date: 26-Jun-2024
https://doi.org/10.1186/s40623-024-02008-5
Hashima AHori TIinuma TMurakami SFujita KIchimura T(2024)Stress change in southwest Japan due to the 1944–1946 Nankai megathrust rupture sequence based on a 3-D heterogeneous rheological modelEarth, Planets and Space10.1186/s40623-023-01943-z76:1Online publication date: 8-May-2024
https://doi.org/10.1186/s40623-023-01943-z
Plata‐Martinez RIinuma TTomita FNakamura YNishimura THori T(2024)Revisiting Slip Deficit Rates and Its Insights Into Large and Slow Earthquakes at the Nankai Subduction ZoneJournal of Geophysical Research: Solid Earth10.1029/2023JB027942129:12Online publication date: 20-Dec-2024
https://doi.org/10.1029/2023JB027942
Show More Cited By

Index Terms

Fast and Scalable Low-Order Implicit Unstructured Finite-Element Solver for Earth's Crust Deformation Problem
1. Applied computing
  1. Physical sciences and engineering
    1. Earth and atmospheric sciences
2. Computing methodologies
  1. Parallel computing methodologies
    1. Parallel algorithms
      1. Massively parallel algorithms

Recommendations

Development of Element-by-Element Kernel Algorithms in Unstructured Implicit Low-Order Finite-Element Earthquake Simulation for Many-Core Wide-SIMD CPUs
Computational Science – ICCS 2019
Abstract
Acceleration of the Element-by-Element (EBE) kernel in matrix-vector products is essential for high-performance in unstructured implicit finite-element applications. However, the EBE kernel is not straightforward to attain high performance due to ...
Complex-shaped beam finite element

Two approaches to derive two-dimensional beam finite-element matrices to map the structural behavior of complex-shaped beams are studied. The beams have varying thickness and the centerline may be strongly curved. The beam-theory approach combined with ...
Nonlinear, finite deformation, finite element analysis
Abstract
The roles of the consistent Jacobian matrix and the material tangent moduli, which are used in nonlinear incremental finite deformation mechanics problems solved using the finite element method, are emphasized in this paper, and demonstrated ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

PASC '17: Proceedings of the Platform for Advanced Scientific Computing Conference

June 2017

136 pages

ISBN:9781450350624

DOI:10.1145/3093172

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing
CSCS: Swiss National Supercomputing Centre
ETH Zurich: Federal Institute of Technology - University of Zurich

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 June 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

PASC '17

Sponsor:

SIGHPC
CSCS
ETH Zurich

PASC '17: Platform for Advanced Scientific Computing Conference

June 26 - 28, 2017

Lugano, Switzerland

Acceptance Rates

PASC '17 Paper Acceptance Rate 13 of 33 submissions, 39%;

Overall Acceptance Rate 109 of 221 submissions, 49%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
395
Total Downloads

Downloads (Last 12 months)82
Downloads (Last 6 weeks)20

Reflects downloads up to 01 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Murakami SHashima AIinuma TFujita KIchimura THori T(2024)Detectability of low-viscosity zone along lithosphere–asthenosphere boundary beneath the Nankai Trough, Japan, based on high-fidelity viscoelastic simulationEarth, Planets and Space10.1186/s40623-024-02008-576:1Online publication date: 26-Jun-2024
https://doi.org/10.1186/s40623-024-02008-5
Hashima AHori TIinuma TMurakami SFujita KIchimura T(2024)Stress change in southwest Japan due to the 1944–1946 Nankai megathrust rupture sequence based on a 3-D heterogeneous rheological modelEarth, Planets and Space10.1186/s40623-023-01943-z76:1Online publication date: 8-May-2024
https://doi.org/10.1186/s40623-023-01943-z
Plata‐Martinez RIinuma TTomita FNakamura YNishimura THori T(2024)Revisiting Slip Deficit Rates and Its Insights Into Large and Slow Earthquakes at the Nankai Subduction ZoneJournal of Geophysical Research: Solid Earth10.1029/2023JB027942129:12Online publication date: 20-Dec-2024
https://doi.org/10.1029/2023JB027942
Murakami SFujita KIchimura THori THori MLalith MUeda N(2023)Development of 3D Viscoelastic Crustal Deformation Analysis Solver with Data-Driven Method on GPUComputational Science – ICCS 202310.1007/978-3-031-36021-3_45(423-437)Online publication date: 26-Jun-2023
https://doi.org/10.1007/978-3-031-36021-3_45
Murakami SIchimura TFujita KHori TOhta Y(2022)Impact of Ambiguity of Physical Properties of Three-Dimensional Crustal Structure Model on Coseismic Slip and Interseismic Slip Deficit in the Nankai Trough RegionGeoHazards10.3390/geohazards30200093:2(162-177)Online publication date: 6-Apr-2022
https://doi.org/10.3390/geohazards3020009
Fujita KMurakami SIchimura THori THori MLalith MUeda N(2022)Scalable Finite-Element Viscoelastic Crustal Deformation Analysis Accelerated with Data-Driven Method2022 IEEE/ACM Workshop on Latest Advances in Scalable Algorithms for Large-Scale Heterogeneous Systems (ScalAH)10.1109/ScalAH56622.2022.00008(18-25)Online publication date: Nov-2022
https://doi.org/10.1109/ScalAH56622.2022.00008
Hori TAgata RIchimura TFujita KYamaguchi TIinuma T(2021)High-fidelity elastic Green’s functions for subduction zone models consistent with the global standard geodetic reference systemEarth, Planets and Space10.1186/s40623-021-01370-y73:1Online publication date: 9-Feb-2021
https://doi.org/10.1186/s40623-021-01370-y
Yamaguchi TFujita KIchimura TNaruse AWells JZimmer CStraatsma THori MMaddegedara LUeda N(2020)Low-Order Finite Element Solver with Small Matrix-Matrix Multiplication Accelerated by AI-Specific Hardware for Crustal Deformation ComputationProceedings of the Platform for Advanced Scientific Computing Conference10.1145/3394277.3401860(1-11)Online publication date: 29-Jun-2020
https://dl.acm.org/doi/10.1145/3394277.3401860
Agata R(2020)Introduction of covariance components in slip inversion of geodetic data following a non-uniform spatial distribution and application to slip deficit rate estimation in the Nankai Trough subduction zoneGeophysical Journal International10.1093/gji/ggaa116221:3(1832-1844)Online publication date: 17-Mar-2020
https://doi.org/10.1093/gji/ggaa116
Agata RHori TBarbot SHyodo MIchimura T(2020)Quasi-static Simulation Method of Earthquake Cycles Based on Viscoelastic Finite Element ModelingMathematical Analysis of Continuum Mechanics and Industrial Applications III10.1007/978-981-15-6062-0_11(159-169)Online publication date: 30-Aug-2020
https://doi.org/10.1007/978-981-15-6062-0_11
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten