Implementation of XIOS in the Atmospheric Component of the Unified Model

Key Personnel

PI/Co-I: Grenville Lister, Bryan Lawrence - University of Reading, Michele Weiland - EPCC

Technical: Jeff Cole - University of Reading, Rupert Nash - EPCC

Relevant Documents

eCSE Technical Report: Implementation of XIOS in the Atmospheric Component of the Unified Model

Project summary

Developments in climate modelling typically follow one of three directions (although ideally all three simultaneously): improvements of model fidelity; increases in model resolution; larger ensemble sizes. Each comes with its own challenges, but common to all is the need to better manage model diagnostic output with a recognition of the desirability to separate data processing from data production. In the Met Office Unified Model, used extensively in the ac.uk community, the diagnostic system, while very rich in capability, is tightly integrated with the model dynamics and physics. Increasing use of couplers (as for example between NEMO (an ocean model) and the atmospheric UM) leads naturally to the idea that the diagnostic system itself could (should) be a coupled component of a climate model.

Our operational familiarity with XIOS and NEMO led us to propose XIOS as a candidate a coupled diagnostic component for the UM. One of two objectives of our project was to implement this proposal. Five main areas of development were identified in the work plan to this; (i) develop code in the UM to manage XIOS and UM communicators, (ii) develop code in the UM to set up the domains, axes, grids, calendars…; (iii) develop utilities to convert UM STASH request into XML; (iv) develop code in the UM to intercept diagnostic fields to send to XIOS; (v) modify the job submission system to understand use of XIOS. Figure 1 depicts how XIOS fits into the UM infrastructure such that its look and feel is not altered (this is an important consideration to ensure user uptake of the new developments).

We have adapted the data reduction capabilities already available in XIOS to work in a UM ensemble environment, whereby, we eliminate the need to write data for each ensemble member in order to create an ensemble average; XIOS itself calculates the ensemble average and writes the result. We also allow the possibility to write out diagnostics for select ensemble members (or all of them).

We developed a method to achieve our goal through the use of an ensemble axis and determined how to configure the XML control file to direct XIOS to perform the reductions (max, min, and average). With this knowledge, we built on our UM-XIOS work to develop UM ensembles in XIOS, We undertook five main areas; (i) modify UM code to manage communications splitting and the ensemble axis, (ii) develop UM-ensemble configuration capability in Rose/Cylc; (iii) further modify job submission system to handle MPMD, (iv) modify UM environments to enable simultaneous execution of multiple instances of the model. Figure 2 describes key components of the ensemble configuration and work flow./p>

Achievement of objectives

As increasing computer power has become available, the weather and climate community have put effort into increasing the spatial resolution within the simulation domain, increasing the domain of simulation (spatial and/or temporal), expanded use of, or complexity of, data assimilation to improve initial conditions, and increasing the number of simulations within ``ensembles''. Such ensembles are collections of simulations which address the same problem, but where some key aspect of each simulation differs from the others. Our traditional methods of ensemble analysis, which entail writing out data for each ensemble member, to be processed later, will not scale with increasing ensemble size and model resolution. Hence, our objective Objective (i): Deploy simple XIOS based statistical analysis capability for ensembles of UM simulations

It has long been recognised that modularity is a highly desirable feature in a complex software system. Climate models are no exception; in the Unified model (UM), the atmosphere and ocean components are quite distinct and run independently, only to exchange fields through a coupler at some defined frequency. There is no reason why such coupling should be limited to physics schemes, and indeed since IO is an important consideration in climate modelling, IO servers have been a focus of much attention. The atmosphere component of the UM has an efficient IO server, which, nevertheless, is tightly integrated with the structure of the UM. NEMO (the ocean component of the UM climate model) uses XIOS to process and write its diagnostics. The NEMO-XIOS interface is minimal, simply call to send data to XIOS, which then has the responsibility to filter, and otherwise manipulate the raw fields, and write out the processed diagnostics. Our second objective Objective (ii): Replace the current Unified Model (UM) diagnostic system with XIOS (XML-IO-Server).

Metrics

Objective (i): Deploy simple XIOS based statistical analysis capability for ensembles of UM simulations

We have successfully run a 100-member ensemble of low resolution (N96) UMs through XIOS to generate ensemble averages of up to 30 fully 3-d fields – our test cases generated data at rates in excess of what is typical in a climate simulation (note N96 is the resolution at which a very large number of simulations will run at to form part of NERC’s contribution to CMIP6 and will be widely used throughout the community for many years to come). We also successfully ran a 10-member ensemble of high-resolution (N512) UMs, again with extreme data rates. The 10-member ensemble ran on ~54 000 processors, representing a significant and stringent test of current XIOS capability.

Objective (ii): Replace the current Unified Model (UM) diagnostic system with XIOS (XML-IO-Server)

We have implemented XIOS in the atmosphere component of the UM and developed a utility to free the user from concerns about the generation of XML from the UM description of diagnostics. The tool converts user defined STASH requests into XML and is run as a task prior to job submission. We can handle the 3672 fields available on time steps (prognostics and diagnostics) out of the 4017 available in the standard atmosphere model. Some fields are defined to be available on specialized times (radiation, or convection, or coupling time step which typically differ from the normal model time step (a multiple thereof) – we do not consider those cases.

We cannot claim to have replaced the entire UM diagnostic system but we have made significant progress toward that gaol. Our scheme does handle the vast majority of typical climate diagnostic output. The UM retains its STASH functionality – for testing and to support fields incompatible with XIOS. Check pointing is handled in the traditional manner.

Summary of the software

Our work has two naturally distinct areas of development associated with distinct parts of the workflow. We have developed stash2xiosxml which acts on Rose configuration files to convert STASH requests into XML for use by XIOS. Figure 3 outlines the workflow; stash2xioxxml ingests Rose configuration files to generate iodef.xml and its component XML files.

Our second major development is in the UM codebase. Figure 4 indicates where our developments impact the UM. Routines with prefix um_xios_ represent new code. Diagnostics are intercepted in the existing routine, stash, and directed to xios through um_xios_send_field. The new XIOS functionality detracts nothing from existing UM capability.

Code developed under this eCSE project is available to all UM users with access to the Met Office Science Repository Service (MOSRS). It is a UM branch at main:/branches/dev/jeffcole/vn10.7_xios_output. Rose suites are also available from MOSRS.