Large scale CASTEP calculations to interpret solid-state NMR and Vibrational Spectroscopy Experiments

Key Personnel

PI/Co-I: Professor Jonathan Yates - University of Oxford

Technical: Professor Keith Refson - Royal Holloway, University of London; Dr. Dominik Jochym - STFC

Relevant Documents

eCSE Technical Report: Large scale CASTEP calculations to interpret solid-state NMR and Vibrational Spectroscopy Experiments

Project summary

In recent years it has become clear that in the broad discipline of Materials Science the power of modern experimental spectroscopic techniques is inextricably linked to the availability of advanced computational tools and the infrastructure to run such simulations. This is particularly true in the fields of solid-state Nuclear Magnetic Resonance (NMR) and vibrational spectroscopy (neutron and X-ray scattering, Raman and IR spectroscopy), where deployment of first-principles methods to predict experimental observables has enabled many research groups to employ a combined experimental and computational approach.

Solid-state NMR spectra are a rich source of information about atomic-level structure and dynamics, capable of providing information on wide range of materials both with and without long range order. Planewave pseudopotential approaches developed in the last 10 years have been demonstrated to be an invaluable tool for predicting and interpreting solid-state NMR observables. Notably, the GIPAW (gauge-including projector augmented wave) method (and in particular its implementation in the CASTEP code) gives excellent agreement with experiment and has been rapidly taken up by the experimental solid-state NMR community. Many experimentalists rightly describe the impact of GIPAW calculations on the field as revolutionary. Practically all solid-state NMR research groups in the UK make use of GIPAW calculations in their research accessing a range of departmental and institutional level HPC resources.

Similarly, inelastic neutron scattering is one of the most important experimental probes of lattice vibrations, with world-class high resolution instruments such as TOSCA and the new LET hosted at the ISIS facility. It is complemented by IXS, with STFC funding for the ID28 instrument at ESRF, which can be applied to tiny single crystals, dramatically extending the range of materials which can be studied. First principles lattice dynamics calculations (at which CASTEP excels) have revolutionised the ability to interpret spectra of complex materials. Indeed both experimental proposals and publications now risk failing peer review without a modelling component.

In this project we have significantly improved the parallel scaling of the spectroscopic modules in CASTEP (specifically NMR and vibrational) by introducing an additional layer parallelism into the code - parallelism over perturbations. Many spectroscopic parameters can be considered as the response of the electronic structure to a perturbation. For example the NMR J-coupling is the response of a system to the magnetic field generated by excitation of a nuclear spin; the NMR magnetic shielding is the response of a system to an applied external magnetic field; lattice dynamics is the response of the system to the movement of the ionic cores. Utilising this new level of parallelism means that for large core counts the parallel FFT is only computed within a small subset of the participating cores - dramatically increasing the scaling performance of the code. We give an indicative example for the simplest case - that of the NMR J-coupling: For a moderately large system (108 atoms) there are 108 independent perturbations. Previously these were computed sequentially i.e. one after the other. As shown in Figure 1 the scaling of the parallel 3d FFT limits the number of nodes that can be used effectively to about 4 with each perturbation taking about 4 hours to compute. Having implemented the automatic parallelisation over perturbation sites the calculation shows nearly linear scaling up to 108*4 nodes (10368 cores) (Figure 2). This dramatically decreases the time to science.

Achievement of objectives

The project had four main objectives:

1. Perturbation parallel for NMR J-coupling tensors:

   Objective fully met. We have demonstrated close to linear scaling with the number of perturbation farms, extending to over 10k cores. We anticipate the software will scale to even larger core counts for larger problem sizes. The new functionality is included in the development version of CASTEP, and is ready to be made available to Archer users.

2. Perturbation parallel for NMR Magnetic Shielding tensors:

   Using the experience gained in NMR J-coupling tensors, further analysis revealed that this development would give only marginal performance improvements. For this reason this objective was not pursued.

3. Perturbation parallel for Phonon/vibrational properties:

   Objective fully met. We have demonstrated very good scaling of the perturbation part of the calculation with the number of farms - extending to over 4k cores. Again we anticipate larger core counts would enable scaling to extend to larger number of cores. However, tests show a less good scaling of the total calculation time. This is due to the calculation of the e-field response, which does not scale with the number of farms. A simple re-factoring of the code (c. 1 month) will remove this bottleneck, at which point the code will be made available to the ARCHER community.

4. A further objective is the development of heuristics to inform users of the best parallel strategy for a particular job:

   We have extended the phonon_kpoint tool to take account of the new parallel distributions - this identifies optimal core counts for the specific job.

Summary of the software

CASTEP is a UK-based state-of-the-art implementation of DFT and a flagship code for UK HPC. It was designed from the ground up according to sound software engineering principles and with HPC in mind. CASTEP describes the electronic states ("bands") using a plane-wave (Fourier) basis, necessitating the heavy use of parallel fast Fourier transforms (FFTs) for efficiency. This implementation, while general, is very well-suited to solid-state applications including defects and interfaces. The robustness and reliability of the code has strongly contributed to its widespread adoption by the experimental communities detailed above.

In recent years the scalability of CASTEP has improved dramatically through the efforts of the CASTEP Developers Group, STFC-Daresbury (via the SLA) and a series of dCSE grants. CASTEP is a code that can be used on a variety of sizes of HPC systems, depending on the physical size and complexity of the material being studied. This can range from a single core PC suitable for the study of a few 10s of atoms, up to a few thousand cores for systems with 1000s of atoms. For large problem sizes the scaling limit of CASTEP arises from the need to perform distributed 3d FFTs.