Massively Parallel MPI Implementation of the SPH Code DualSPHysics

Key Personnel

PI/Co-I: Benedict Rogers (University of Manchester)

Technical: Athanasios Mokos (University of Manchester)

Relevant documents

eCSE Technical Report: Massively Parallel MPI Implementation of the SPH Code DualSPHysics

Project summary

Smoothed Particle Hydrodynamics (SPH) is a novel computational technique that is fast becoming a popular methodology to tackle industrial problems with violent flows that often occur in nature and industry. The main aim of this project is the implementation of MPI functionality to the open-source DualSPHysics software package, developed by the universities of Manchester, Vigo and Parma, enabling the simulation of violent, highly transient flows using thousands of cores such as slam forces due to waves on offshore and coastal structures, impact loads on high speed ships, ditching of aircraft, sloshing of fuel tanks and many more. A highly topical application concerns breaking wave loads on offshore wind turbine columns which are subject to heavy load due to non-breaking waves and is highly transient. These simulations require a large number of particles and large computing times. In addition, the code will be re-engineered to increase its modularity easing the implementation of new models and features such as a multi-phase model. The following list highlights the major developments:

1. Implementation of MPI functionality to existing DualSPHysics code, retaining the existing structure, neighbour list and optimisations of the existing OpenMP implementation. Developed a system exchanging information between nodes.
2. Developed a geometric domain decomposition scheme. Use of asynchronous communications to overlap node communication with the particle computation minimising idle time.
3. Implemented the Zoltan domain decomposition package with a Hilbert space-filling curve to handle domain decomposition. Developed a load balancing system, ensuring nodes have a nearly identical computational workload.
4. Two executables have been created, one using the geometric domain decomposition scheme for smaller scale applications and one with the integration of the Zoltan library for High Performance Computing architectures.

Achievement of objectives

Task 1: Implementation of MPI functionality to existing DualSPHysics code, retaining the existing structure, neighbour list and optimisations of the existing OpenMP implementation. (Total 8 months' effort, in conjunction with Task 2)

To complete this task, the following steps were taken

1. A new buffer system was created to handle asynchronous communications between nodes.
2. A geometric domain decomposition scheme is implemented. Cells are organized in slices along a geometric axis (selected depending on domain dimensions). Each node is then assigned a specific group of slices (a subdomain) and receives the appropriate particle data to manage. Within each node, the existing structure, neighbour list and optimisations of the existing OpenMP implementation are used.
3. A halo exchange system is developed. Slices on the edge of the subdomain are identified and their cell and particle data transferred to the nodes requesting them.
4. A particle exchange system is developed. Particles exiting the subdomain are identified and sent to a new subdomain. Their original data is deleted to minimize the memory footprint.

Task 2: Utilise asynchronous communications to overlap node communication with the particle computation minimising idle time. (Total 8 months' effort, in conjunction with Task 1)

1. Slices and cells on the edge of the subdomain were pre-identified and the data is sent to neighbouring nodes while processing the neighbour list for interior particles.
2. The same procedure is applied for halo particles, minimising idle computational resources.
3. A lightweight version of the code suitable for smaller test cases in conjunction with the existing GPU code, sharing a similar modular architecture to enable easier user transition is created.

Task 3: Optimisation of the code and the MPI implementation for efficient scaling to 1,000+ cores while maintaining spatial locality (Total 4 months' effort, spent in 8 months period)

1. Implemented the Zoltan domain decomposition package with a Hilbert space-filling curve to create a cell map for each node, maintaining spatial locality.
2. Used the load balancing capabilities of the Zoltan library to transfer cells between nodes based on a cell weight metric.
3. Moved the neighbour list creation, the halo and the particle exchange to a cell based algorithm instead of a geometry based algorithm and incorporate the Zoltan library. [Incomplete]
4. Transition to an HDF5 based output to handle large files and output from hundreds of nodes [Incomplete].

Task 4: Application and demonstration of the new DualSPHysics capabilities:

1. Demonstrate the overall performance of the lightweight code, showing an efficiency of 90% for 32 cores and 70% for 128 cores.
2. Use the new code on real application test cases with very large numbers of particles using thousands of cores. Emphasis will be given on 3D flows with different spatial scales. [Incomplete]

Summary of the Software

The DualSPHysics software package already uses GIT to manage software revision control. The DualSPHysics with integrated MPI support (MPISPHysics) software development repositories are currently hosted at bitbucket (https://bitbucket.org/mpisphysics). They include branches for the development and maintenance as well as a master branch, which, once granted access permission, the archer users can use. The MPISPHysics code will be integrated with the main DualSPHysics project (http://dual.sphysics.org/) DualSPHysics has an existing wiki page (https://github.com/DualSPHysics/DualSPHysics/wiki) .

Scientific Benefits

SPH is an efficient modelling tool to study and gain better understanding of violent waves and their interaction with various offshore and coastal structures with repeated regular or irregular wave impacts. So, this project benefits several UK and international academic communities: marine/offshore renewable energy and structures, marine transport, marine hydrodynamics. The main industrial beneficiaries from this project will be the emerging offshore and marine energy industries, the existing offshore oil and gas industries and engineering consultancies involved in the design of coastal and flood defences.

The modular architecture of the project will allow an easy expansion of the project to new applications such as multi-phase flow, enabling the modelling of complex air-water-sediment flows. Schemes for the latter have been developed in the Manchester SPH group and have been published to international journals.