Tuning FHI-Aims for complex simulations on Cray HPC platforms

Key Personnel

PI/Co-I: Dr Scott Woodley, Prof Richard Catlow, Dr Alexey Sokol, Dr Andrew Logsdail - University College London

Technical: Dr Matthew Farrow - University College London

Relevant documents

eCSE Technical Report: Tuning FHI-Aims for complex simulations on CRAY HPC platforms

Project summary

This project was concerned with porting and optimising the electronic structure code FHI-aims for use on the UK national supercomputer, ARCHER. The FHI-aims software has been developed specifically for highly accurate modelling of a variety of system types, including solids and nanoparticles, using state-of-the-art Density Functional Theory (DFT), which focuses on the behaviour of electrons and atoms in a chemical compound. The methodology adopted in the FHI-aims code uses numerical atomic orbitals, or electron trajectories, which, in theory, are extremely efficient for calculations on many processors simultaneously. Therefore, the code is very advantageous for usage on massive supercomputers such as ARCHER. Our work thus enables and encourages the undertaking of new research using current and emerging UK computer facilities in a wide range of fields including materials and life sciences, chemistry, physics, and engineering.

We have compiled and benchmarked the software on ARCHER and created a module for current users to load conveniently an optimised executable. This work has immense benefit for potential academic and industrial user communities. Currently, users of FHI-aims have to install the software themselves, which requires specialist knowledge and is time consuming. Moreover, the users are unlikely to be using the latest version of the software which is more efficient and has previously-discovered problems resolved. This module allows all users to be using the same code base, which is important for scientific reproducibility.

In our project, we have used profiling tools to determine which portions of the software take the longest time. For example, the performance analysis flagged one routine that took an unexpected time to complete, which we have addressed, and this resulted in an optimised routine.

We highlight the newly available self-consistent meta-GGA density functionals, which allows for very accurate chemical calculations and was previously implemented in a post-processing, and thus approximate, manner. Hence, new predictive studies can now be expected in the description of complex chemical reactions.

In the last part of the project and in support of KLMC runs, FHI-aims has been optimised to facilitate cheaper pre-screening on DFT energy landscapes when more traditional interatomic potentials are not available. Making meta-GGA functionals available in ChemShell enables new scientific applications on one hand and speeds up significantly more challenging reactivity problems. However, there is still a significant scope for further code optimisation in FHIaims for the embedding approach implemented within ChemShell. Support for further work in both directions would be very beneficial for various applications and, therefore, will form the basis of future proposals.

Achievement of objectives

The main objective was to "tune FHI-aims for optimal efficiency on ARCHER's hardware, both as a standalone code and as a key component within two software packages - KLMC and ChemShell - for the benefit of members of the Materials Chemistry Consortium (MCC) and other new or current users. Improving the scalability for a wide range of applications to enable new science is the overarching target".

We fulfilled this objective by porting an optimal installation of the FHI-aims code to the ARCHER platform, which is available to both KLMC and ChemShell, and by the creation of a module that future users can exploit by simply typing the phrase "module load aims". This command will provide future users of ARCHER, who have a license to run FHI-aims, access to the latest version of the code. To optimise the FHI-aims code for the ARCHER platform, we have tuned the optimal compiler settings; these settings are published on the FHI-aims wiki page for users of FHI-aims and were used in the compilation of the executable that is made available to ARCHER users via the "module load aims" command.

As typically found for codes newly ported onto a high-end compute (HEC) platform, the FHI-aims software also suffered from a number of bottlenecks when initially ported onto ARCHER. There are different types of jobs the user can use FHI-aims to perform; therefore, the bottlenecks are of course dependent upon the chosen task. We identified the bottlenecks of FHI-aims when used to perform typical simulations that are of interest to the materials chemistry community. These bottlenecks were brought to the attention of the core developers; patches to the code that addresses these bottlenecks will be made available in future releases of the software.

A large proportion of ARCHER resource is consumed by hybrid DFT based calculations and, in particular, performed using VASP. The objective in the largest of the four work packages was to provide a more efficient approach. The implementation of meta-generalised gradient approximation (meta-GGA) density functionals is an important and essential enhancement of current capabilities, which resulted from the eCSE and EPSRC support of this work. These functionals are an order of magnitude cheaper than hybrid functionals and offer an improved accuracy compared to the more traditional functionals that hybrid functionals have replaced.

The relationship between the core developers of FHI-aims and the UCL group has also been strengthened by the success of this project. Further performance enhancements are still underway at UCL within our group; we are finding and implementing new algorithmic solutions to the basic SCF code infrastructure. The eCSE work packages were exploratory in nature and ambitious, and have opened new avenues to pursue, for which we will seek further eCSE support.

Summary of the software

The software is held in a git repository accessible from:

https://aimsclub.fhi-berlin.mpg.de/index.php

On ARCHER, access can be obtained for the "aims" module by providing the ARCHER helpdesk with proof of licence. In general, the FHI-aims software is obtained directly from the developer (https://aimsclub.fhi-berlin.mpg.de) and a licence is required for its use.

Support for FHI-aims usage on ARCHER will be provided in the future by our team via the Materials Chemistry Consortium (MCC) framework.