skip to main content
other

Scalability studies and large grid computations for surface combatant using CFDShip-Iowa

Published: 01 November 2011 Publication History

Abstract

Scalability studies and computations using the largest grids to date for free-surface flows are performed using message-passing interface (MPI)-based CFDShip-Iowa toolbox curvilinear (V4) and Cartesian (V6) grid solvers on Navy high-performance computing systems. Both solvers show good strong scalability up to 2048 processors, with V6 showing somewhat better performance than V4. V6 also outperforms V4 in terms of the memory requirements and central processing unit (CPU) time per time-step per grid point. The explicit solvers show better scalability than the implicit solvers, but the latter allows larger time-step sizes, resulting in a lower total CPU time. The multi-grid HYPRE solver shows better scalability than the portable, extensible toolkit for scientific computation solver. The main scalability bottleneck is identified to be the pressure Poisson solver. The memory bandwidth test suggests that further scalability improvements could be obtained by using hybrid MPI/open multi-processing (OpenMP) parallelization. V4-detached eddy simulation (DES) on a 300 M grid for the surface combatant model DTMB 5415 in the straight-ahead condition provides a plausible description of the vortical structures and mean flow patterns observed in the experiments. However, the vortex strengths are over predicted and the turbulence is not resolved. V4-DESs on up to 250 M grids for DTMB 5415 at 20 ° static drift angle significantly improve the forces and moment predictions compared to the coarse grid unsteady Reynolds averaged Navier-Stokes, due to the improved resolved turbulence predictions. The simulations provide detailed resolution of the free-surface and breaking pattern and vortical and turbulent structures, which will guide planned experiments. V6 simulations on up to 276 M grids for DTMB 5415 in the straight-ahead condition predict diffused vortical structures due to poor wall-layer predictions. This could be due to the limitations of the wall-function implementation for the immersed boundary method.

References

[1]
Balay S,Buschelman K,Gropp W,Kaushik D,Knepley M,Curfman L, et alPETSc User Manual, ANL-95/11, Revision 2.1.5. USA: Argonne National Laboratory; 2002:
[2]
Bhushan S,Xing T,Carrica PM,Stern F.Model- and full-Scale URANS simulations of Athena resistance, powering and seakeeping, and 5415 maneuvering.Journal of Ship Research. 2009;53 (4): 179-198
[3]
Carrica PM,Huang J,Noack R,Kaushik D,Smith B,Stern F.Large-scale DES computations of the forward speed diffraction and pitch and heave problems for a surface combatant.Computers and Fluids. 2010;39 (7): 1095-1111
[4]
Carrica PM,Paik KJ,Hosseini HS,Stern F.URANS analysis of a broaching event in irregular quartering seas.Journal of Marine Science and Technology. 2008;13 (4): 395-407
[5]
Carrica PM,Wilson RV,Noack R,Stern F.Ship motions using single-level set with dynamic overset grids.Computers and Fluids. 2007a;36 (9): 1415-1433
[6]
Carrica PM,Wilson RV,Stern F.An unsteady single-phase level set method for viscous free surface flows.International Journal of Numerical Methods in Fluids. 2007b;53 (2): 229-256
[7]
Chorley MJ,Walker DW,Guest MF.Hybrid message-passing and shared-memory programming in a molecular dynamics application on multicore clusters.The International Journal of High Performance Computing Applications. 2009;23 (3): 196-211
[8]
Cohen J,Molemaker J.A fast double precision CFD code using CUDA.Proceedings of the 21st Parallel CFD Conference; 2009Moffett Field, CA; 2009. 252.
[9]
Cuicchi C.Navy DSRC retrospective: 1990-2009.HPC Insights. 2009;:
[10]
Dongarra J,Gannon D,Fox G,Kenned K.The impact of multicore on computational science software.CTWatch Quarterly. 2007;3:3-10
[11]
Falgout RD,Jones JE,Yang UMNumerical Solution of Partial Differential Equations on Parallel Computers. Bruaset AMTveito A, ed. Netherlands: Springer-Verlag; 2006:267-294.
[12]
Fureby C.Large eddy simulation of ship hydrodynamics.Proceedings of the 27th Symposium on Naval Hydrodynamics; 2008Seoul, Korea; 2008. .
[13]
Gicquel LYM,Staffelbach G,Cuenot B,Poinsot T.Large eddy simulations of turbulent reacting flows in real burners: the status and challenges.Journal of Physics: Conference Series. 2008;125:012029
[14]
Hino T.<atl/>.<conftl/>; 2005; 2005. .
[15]
Huang J,Carrica PM,Stern F.Coupled ghost fluid/two-phase level set method for curvilinear body-fitted grids.International Journal Numerical Methods Fluids. 2007;55:867-897
[16]
Huang J,Carrica PM,Stern F.Semi-coupled air/water immersed boundary approach for curvilinear dynamic overset grids with application to ship hydrodynamics.International Journal Numerical Methods Fluids. 2008;58:591-624
[17]
Ismail F,Carrica PM,Xing T,Stern F.Evaluation of linear and non-linear convection schemes on multidimensional non-orthogonal grids with applications to KVLCC2 tanker.International Journal for Numerical Methods in Fluids. 2010;64 (8): 850-886
[18]
Issa RI.Solution of the implicit discretised fluid flow equations by operator splitting.Journal of Computational Physics. 1985;62:40-65
[19]
Kalitzin G,Medic G,Iaccarino G,Durbin P.Near-wall behavior of RANS turbulence models and implications for wall functions.Journal of Computational Physics. 2005;204:265-291
[20]
Kaushik D,Balay S,Keyes D,Smith B.Understanding the performance of hybrid MPI/OpenMP programming model for implicit CFD codes.Proceedings of the 21st Parallel CFD Conference; 2009Moffett Field, CA; 2009. 174.
[21]
Keating A,Prisco GD,Piomelli U.Interface conditions for hybrid RANS/LES calculations.International Journal of Heat and Fluid Flow. 2006;27:777-788
[22]
Kremenetsky M,Kodiyalam S.Parallel performance of CFD algorithm and the ubiquitous need for HPC with high fidelity, multidisciplinary analysis and optimization (MDO).Proceedings of the 21st Parallel CFD Conference; 2009Moffett Field, CA; 2009. 338.
[23]
Larsson L,Stern F,Bertram VProceedings of Gothenburg 2000- A workshop on numerical ship hydrodynamics. Gothenburg, Sweden: Chalmers University of Technology; 2000:
[24]
Lignell D,Yoo C,Chen J,Sankaran R,Fahey MProceedings of the Cray Scaling Workshop. TN: Oak Ridge National Laboratory; 2007:
[25]
Lim J,Strawn RC.Prediction of HART II rotor BVI loading and wake system using CFD/CSD loose coupling, AIAA paper 2007-1281.Proceedings of the 45th AIAA Aerospace Sci. Meeting and Exhibit; 2007Reno, NV; 2007. .
[26]
Longo J,Shao J,Irvine M,Stern F.Phase-averaged PIV for the nominal wake of a surface ship in regular head waves.Journal of Fluids Engineering. 2007;129 (5): 524-540
[27]
Menter FR.Two equation eddy viscosity turbulence models for engineering applications.AIAA Journal. 1994;32 (8): 1598-1605
[28]
Mohd-Yusof J,Livescu D,Kelly T.Adapting the CFDNS Compressible Navier-Stokes Solver to the Roadrunner Hybrid Supercomputer.Proceedings of the 21st Parallel CFD Conference; 2009; 2009. 64.
[29]
Noack R.SUGGAR: a general capability for moving body overset grid assembly.Proceedings of the 17th AIAA Computational Fluid Dynamics Conference; 2005Toronto; 2005. .
[30]
Oed W.Scaling applications on Cray supercomputers.Proceedings of the T-Systems Solutions for Research GmbH HPCN-Workshop; 2009Germany; 2009. .
[31]
Olivieri A,Pistani F,Avanzini A,Stern F,Penna R.Towing tank experiments of resistance, sinkage and trim, boundary layer, wake, and free surface flow around a naval combatant INSEAN 2340 model.IIHR Technical Report: 421. 2001;:
[32]
Olivieri A,Pistani F,Wilson R,Campana EF,Stern F.Scars and vortices induced by ship bow and shoulder wave breaking.Journal of Fluids Engineering. 2007;129 (11): 1445-1459
[33]
<atl/>.Proceedings of 21st International Conference on Parallel Computational Fluid Dynamics; 2009Moffett Field, CA; 2009. .
[34]
Pulliam TH,Jespersen DC.Large scale aerodynamic calculations on pleiades.Proceedings of the 21st Parallel CFD Conference; 2009Moffett Field, CA; 2009. 73.
[35]
Rabenseifner R,Wellein G.Communication and optimization aspects of parallel programming models on hybrid architectures.The International Journal of High Performance Computing Applications. 2003;17 (1): 49-62
[36]
Sahni O,Zhou M,Shephard MS,Jansen KE.Scalable implicit finite element solver for massively parallel processing with demonstration to 160K cores.Proceedings of the International Conference of High Performance Computing, SC09; 2009Portland, OR; 2009. .
[37]
Saini S,Jespersen DC,Talcott D,Djomehri J,Sandstrom T.Performance comparison of SGI Altix 4700 and SGI Altix 300 Bx2.NAS Technical Report NAS-01-2001. 2008;:
[38]
SakamotoN (2009) URANS and DES simulations of static and dynamic maneuvering for surface combatant. PhD thesis, The University of Iowa.
[39]
Sanmiguel R-E,Ortega C-J,Fernandez F-R.On the efficiency of a numerical method with periodic time strides for solving incompressible flows.Journal of Computational Physics. 2003;186:212-229
[40]
Shalf J.Hardware trends. Panel discussion presentation.Proceedings of the 21st Parallel CFD Conference; 2009Moffett Field, CA; 2009. .
[41]
Spalart PR.Detached-eddy simulation.Annual Review of Fluid Mechanics. 2009;41:181-202
[42]
Stern F,Agdrup C.<atl/>.Proceedings of SIMMAN 2008 Workshop on Verification and Validation of Ship Maneuvering Simulation Methods; 2008; 2008. .
[43]
Stern F,Bhushan S,Carrica P,Yang J.Large scale parallel computing and scalability study for surface combatant static maneuver and straight ahead conditions using CFDShip-Iowa.Proceedings of the 21st Parallel CFD Conference; 2009Moffett Field, CA; 2009. 52.
[44]
Stern F,Wilson R,Shao J.Quantitative approach to V&amp;V of CFD simulations and certification of CFD codes.International Journal of Computational Fluid Dynamics. 2006;50:1335-1355
[45]
Thibault JC,Senocak I.CUDA Implementation of a Navier-Stokes solver on multi-GPU desktop platforms for incompressible flows.Proceedings of the 47th AIAA Aerospace Sciences Meeting; 2009Orlando, FL; 2009. .
[46]
Top 500 (November2009). Top 500 List - November 2009. Available at: http://www.top500.org/list/2009/11/100.
[47]
Wallin S,Johansson A.An explicit algebraic Reynolds stress model for incompressible and compressible turbulent flows.Journal of Fluid Mechanics. 2000;403:89-132
[48]
Wang Z,Yang J,Stern F.An improved particle correction procedure for the particle level set method.Journal of Computational Physics. 2009a;228 (16): 5819-5837
[49]
Wang Z,Yang J,Koo B,Stern F.A coupled level set and volume-of-fluid method for sharp interface simulation of plunging breaking waves.International Journal of Multiphase Flow. 2009b;35 (3): 227-246
[50]
Xing T,Carrica PM,Stern F.Computational towing tank procedures for single run curves of resistance and propulsion.Journal of Fluids Engineering. 2008;130 (101102): 1-14
[51]
Xing T,Kandasamy M,Stern F.Unsteady free-surface wave-induced separation: analysis of turbulent structures using DES and single-phase level-set.Journal of Turbulence. 2007a;8 (44): 1-35
[52]
Xing T,Shao J,Stern F.BKW-RS-DES of unsteady vortical flow for KVLCC2 at large drift angles.Proceedings of the 9th International Conference on Numerical Ship Hydrodynamics; 2007bAnn Arbor, MI; 2007b. 187.
[53]
Xing T,Stern F.Factors of safety for Richardson extrapolation for industrial applications.Journal of Fluids Engineering. 2010;132 (061403): 1-13
[54]
Yang J,Bhushan S,Suh JS,Wang Z,Koo B,Sakamoto N, et al.Large-eddy simulation of ship flows with wall-layer models on Cartesian grids.Proceedings of the 27th Symposium on Naval Hydrodynamics; 2008Seoul, Korea; 2008. .
[55]
Yang J,Michael T,Bhushan S,Hanaoka A,Wang Z,Stern F.Motion prediction using wall-resolved and wall-modeled approaches on a Cartesian grid.Proceedings of the 28th Symposium on Naval Hydrodynamics; 2010Pasadena, CA; 2010. .
[56]
Yang J,Stern F.Sharp interface immersed-boundary/level-set methods for wave-body interactions.Journal of Computational Physics. 2009;228 (17): 6590-6616
[57]
YoonHS (2009) Force/moment and phase-averaged stereo PIV flow measurements for surface combatant in PMM maneuvers. PhD thesis, The University of Iowa.

Cited By

View all
  1. Scalability studies and large grid computations for surface combatant using CFDShip-Iowa

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image International Journal of High Performance Computing Applications
        International Journal of High Performance Computing Applications  Volume 25, Issue 4
        November 2011
        157 pages

        Publisher

        Sage Publications, Inc.

        United States

        Publication History

        Published: 01 November 2011

        Author Tags

        1. large grid computation
        2. scalability study
        3. ship hydrodynamics
        4. surface combatant
        5. turbulent flows

        Qualifiers

        • Other

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • 0
          Total Citations
        • 0
          Total Downloads
        • Downloads (Last 12 months)0
        • Downloads (Last 6 weeks)0
        Reflects downloads up to 06 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all

        View Options

        View options

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media