An adjoint solver for variable-density flows in the low Mach number limit

Key Personnel

PI/Co-I: Sylvain Laizet

Technical: Paul Bartholomew

Relevant Documents

eCSE Technical Report: An adjoint solver for variable-density flows in the low Mach number limit

Project summary

Incompact3D (https://www.incompact3d.com/, github repository https://github.com/xcompact3d/Incompact3d) is a high-order, highly scalable code for simulating constant density flows. The high scalability is vital to be able to effectively utilise modern computing environments such as ARCHER to study complex flows, with applications ranging from fundamental turbulence research to investigating drag-reduction devices and wind farms.

Incompact3D is based on an assumption of constant density, which limits its applicability to various flows of interest that may feature significant density variations, for example environmental flows such as dispersal of pollutants or avalanches.

The primary goal of this work was to extend Incompact3D to simulate flows in which density variation is significant using the Low Mach Number approximation. This was achieved by first augmenting Incompact3D with additional required variables (density, temperature) and transport equations (continuity/heat). Due to the variable density, the pressure equation required modification by using an extrapolation step, which can then be solved using the fast Poisson solver. This approach is known to be limited to low density ratios (order 3), therefore an iterative solver was implemented to enable the simulation of larger density ratios. This iterative solver utilises the same fast Poisson solver, however as the problem can no longer be approached directly, a correction equation is solved until a user-specified tolerance is achieved.

Both solvers were implemented and tested on ARCHER for scalability, demonstrating that QuasIncompact3D maintains near ideal scaling across a range of architectures on up to 16k cores on ARCHER and 65k cores on Hazel Hen (which has a similar architecture to ARCHER). Marconi is a KNL-based machine and is representative of the accelerator-based architectures that appear to be the future of HPC - this indicates that the code should be adaptable to the next generation of supercomputers.

The solver was also tested by simulating a non-isothermal mixing layer and comparing the results with reference data [1]. This showed that the constant- and variable-coefficient Poisson solvers gave the same answers (the case was chosen within the range of applicability for the constant-coefficient Poisson solver).

This work is already having scientific impact: a paper with a preliminary study of non-Boussinesq gravity currents has been submitted for peer-review with a follow-up paper planned.

Comparison of density contours for non-isothermal mixing layer at times t=24,82,182 from top to bottom. Results are for (from left to right): QuasIncompact3D constant-coefficient pressure-Poisson equation, variable-coefficient pressure-Poisson equation using minimum and harmonically interpolated density and the reference data of Golanski et al. [1].

References

1. François Golanski, Véronique Fortuné, and Eric Lamballais. Noise radiated by a non-isothermal, temporal mixing layer. Part II: Prediction using DNS in the framework of low Mach number approximation. Theoretical and Computational Fluid Dynamics, 19(6):391-416, 2005.

Achievement of objectives

• Work Package 1
  • Objectives: Implement a low Mach number solver based on constant transport properties solving the constant-coefficient pressure-Poisson equation.
  • Success metrics: Demonstrate scalability of solver up to 16k cores on ARCHER and compare to reference data for non-isothermal mixing layer and coaxial round jet.
  • Outcome: Solver shown to be scalable on ARCHER up to 16k cores (65k on Hazel Hen). Good agreement found with reference data for non-isothermal mixing layer. Coaxial round jet test case was replaced with non-Boussinesq gravity currents, which showed good agreement with literature data.
• Work Package 2
  • Objectives: Implement variable transport properties in the solver and solve a variable-coefficient pressure-Poisson equation.
  • Success metrics: Solver to remain scalable on ARCHER up to 16k cores, and compare with the coaxial round jet and round jet impinging on a flat plate.
  • Outcome: The variable-coefficient pressure-Poisson solver was demonstrated to be scalable on ARCHER up to 16k cores (65k on Hazel Hen). This solver was also tested against the non-isothermal mixing layer and the non-Boussinesq gravity current, again showing good agreement with literature data. The non-Boussinesq gravity current reference data allowed the variable transport properties to be tested as the case uses a variable dynamic viscosity.
• Work Package 3
  • Objectives: Implement and test an adjoint solver for the LMN solver.
  • Success metrics: Assess the accuracy of the adjoint solver using the non-isothermal mixing layer case.
  • Outcome: An adjoint solver has been implemented in collaboration with Dr Ubaid Ali Qadri in Cambridge and is currently being tested.

Summary of the software

QuasIncompact3D is freely available through the GitHub repository: https://github.com/ptb0890/Quasincompact3d. The installation and general use of the software follows/is similar to that of Incompact3D so should be immediately useful to current users and similarly easy to use for new users.

QuasIncompact3 is available on ARCHER by cloning the repository and builds without modification by issuing the command 'make' in the source directory.

The general availability of QuasIncompact3D is good - it has been successfully built and run on all systems attempted (this includes KNL in addition to more typical x86-64 systems). The only requirements are a Fortran compiler (tested with gfortran, ifort and Cray's ftn compilers) and a working MPI library.

Scientific Benefits

The value of Incompact3D is increased for the research community by adding capabilities for variable-density flows. It is not currently known if this has impacted upon user numbers but will certainly enhance the appeal of the software to a wider community.

QuasIncompact3D is enabling new science by making available to the ARCHER community and the wider fluid dynamics research community a flow solver suitable for scale-resolving simulations of variable density flows built upon robust and accurate numerical methods.

In addition to the impact it is already having (with a paper submitted for peer-review and a planned follow-up manuscript), it has already lead to collaborative efforts within Imperial College to investigate mixing of variable density fluids. These investigations will enable future research into modelling pollutant transport amongst other potential applications.