-

4. Application Development Environment

• 4.1 Information on the available modules
• 4.2 Loading, unloading and swapping modules
• 4.3 Things to watch out for
• 4.4 Compiler wrapper scripts
• 4.5 Available compiler suites
  • 4.5.1 Switching compiler suites
  • 4.5.2 Changing compiler versions
  • 4.5.3 Useful compiler options
  • 4.5.4 Cray compiler for C++ 11
  • 4.5.5 Intel compiler and GCC for C++ 11
• 4.6 Using dynamic linking/libraries
• 4.7 Compiling for Postprocessing/Serial Nodes
  • 4.7.1 Including Libraries in Serial Builds
• Switching to older/newer programming environment releases
• 4.9 Compiling in batch jobs using the ppn queue

The application development environment on ARCHER is primarily controlled through the modules environment. By loading and switching modules you control the compilers, libraries and software available.

This means that for compiling on ARCHER you typically swap to the correct Programming Environment ("PrgEnv-" modules, one for each different compiler suite), load all the required library modules (e.g. numerical libraries, IO format libraries) and compile your code using the compiler wrapper script (described in detail below). The combination of the modules environment and the wrapper scripts ensures that all the correct headers and library files are included for you.

By default, all users on ARCHER start with the Cray programming environment loaded (PrgEnv-cray) and the MPI (cray-mpich) and Cray LibSci (cray-libsci includes BLAS, LAPACK and ScaLAPACK) libraries included in their environment.

Details of the ARCHER software and application development environment are available online through the Software Catalogue.

Basic usage of the module command on ARCHER is covered below. For full documentation please see:

• Linux manual page on modules

4.1 Information on the available modules

Finding out which modules (and hence which compilers, libraries and software) are available on the system is performed using the module avail command:

user@system:~> module avail
...

This will list all the names and versions of the modules available on the service. Not all of them will work in your account though due to, for example, licencing restrictions. You will notice that for many modules we have more than one version, each of which is identified by a version number. One of these versions is marked as the default. As the service develops the default version will change.

You can list all the modules of a particular type by providing an argument to the module avail command. For example, to list all available versions of the Cray Compiler Environment (CCE) type:

user@system:~> module avail cce
...

If you want more info on any of the modules, you can use the module help command:

user@system:~> module help cce
...

The simple module list command will give the names of the modules and their versions you have presently loaded

user@system:~> module list           
...

4.2 Loading, unloading and swapping modules

To load a module to use module add or module load. For example, to load the Fastest Fourier Transform in the West (FFTW) library into the development environment:

module add fftw

This will load the default version of the FFTW Library. If you need a specfic version of the module, you can add more information:

module add fftw/2.1.5.2

will load version 2.1.5.2 for you, regardless of the default. If you want to clean up, module remove will remove a loaded module:

module remove fftw

(or module rm fftw or module unload fftw) will unload what ever version of fftw (even if it is not the default) you might have loaded. There are many situations in which you might want to change the presently loaded version to a different one, such as trying the latest version which is not yet the default or using a legacy version to keep compatibility with old data. This can be achieved most easily by using module swap oldmodule newmodule. Suppose you have loaded version 3.3.0.1, say, of FFTW, the following command will change to version 2.1.5.2:

module swap fftw fftw/2.1.5.2

This swapping mechanism is often used to select a diffent compiler suite from the default on the system (which is the Cray compiler: "PrgEnv-cray"). For example, to switch to the GNU compilers:

module swap PrgEnv-cray PrgEnv-gnu

Details of the available compiler environments are provided below.

4.3 Module conflicts and dependencies

Modules can depend or conflict with each other. A conflict between modules may be detected and reported. For example you might see something like the following:

user@eslogin008:~> module load PrgEnv-intel
PrgEnv-intel/5.1.29(10):ERROR:150: Module 'PrgEnv-intel/5.1.29' conflicts with the currently loaded module(s) 'PrgEnv-cray/5.1.29'
PrgEnv-intel/5.1.29(10):ERROR:102: Tcl command execution failed: conflict PrgEnv-cray

if you have failed to unload the existing programming environment before attempting to load another. To resolve this use module swap PrgEnv-cray PrgEnv-intel instead.

4.4 Compiler Wrapper Scripts

Code compiled on ARCHER should use the compiler wrapper scripts to ensure that all the correct libraries and header files are included in both the compile and link stages. Note: loading a module will automatically set the necessary paths and link flags for that software, eliminating the need to specify them manually using the LDFLAGS variable or another mechanism. For example, the cray-libsci module is loaded by default on log in and sets the the wrappers to link in BLAS, LAPACK and ScaLAPACK functionality without the need for the user to provide an explicit "-llib_sci".

The compiler wrapper scripts are available for Fortran, C, and C++:

ftn

Fortran compiler

cc

C compiler

CC

C++ compiler

The wrapper scripts can be used to compile both sequential and parallel codes. Further information on the wrapper scripts can be obtained from the man pages, for example:

man ftn

4.5 Available Compiler Suites

ARCHER has 3 compiler suites available:

PrgEnv-cray

The Cray Compiling Environment (CCE). This is the default on ARCHER.

PrgEnv-gnu

The GNU Compiler Collection (GCC).

PrgEnv-intel

The Intel compiler suite.

The manual pages for the different compiler suites are available once the programming environment has been switched in and are:

Cray

Fortran man crayftn ,

C man craycc ,

C++ man crayCC

GCC

Fortran man gfortran (info gfortran is available for gcc/4.*.* and gcc/6.3.0 and later),

C/C++ man gcc (info gcc is available for gcc/4.*.* and gcc/6.3.0 and later)

Intel

Fortran man ifort ,

C/C++ man icc

4.5.1 Switching compiler suites

You use the module swap command along with the "PrgEnv-" modules to switch between the compiler suites on ARCHER. The application development environment (modules and wrapper scripts) then ensures that the correct paths are used for all the libraries and compilers when compiling and linking your code.

For example, to switch from the default Cray compiler suite to the GNU compiler suite:

module swap PrgEnv-cray PrgEnv-gnu

Please note: changing between compiler suites (e.g. Cray to GNU, etc.) must be carried out via the PrgEnv modules. Along with selecting the compiler, the programming environment also determines which numerical and MPI libraries are picked up. Attempts to change compiler suite without changing the programming environment may result in the wrong numerical or MPI libraries being used and your code may not compile or may fail to execute as a result.

4.5.2 Changing compiler versions

You can also change compiler versions within each compiler suite if required; for example, you may wish to use an older version of the compiler for compatibility reasons. This process is also controlled by modules. The names of the compiler modules used to switch versions depends on the compiler suite (PrgEnv-) module you have loaded:

Cray

cce module

GNU

gcc module

Intel

intel module

For example, to switch to an older version of the Cray compiler you would use something like:

module swap cce cce/8.2.6

Note: you must change to the correct compiler suite before changing the compiler version.

4.5.3 Useful compiler options

Whilst difference codes will benefit from compiler optimisations in different ways, for reasonable performance on ARCHER, at least initially, we suggest the following compiler options:

Cray

The default options are sufficient

Intel

-O2

GNU

-O2 -ftree-vectorize -funroll-loops -ffast-math

When you have a code that you are happy is working correctly and has reasonable performance you may wish to investigate some more aggressive compiler optimisations. Below is a list of some further optimisations that you can try on your application (Note: these optimisations may result in incorrect output for programs that depend on an exact implementation of IEEE or ISO rules/specifications for math functions):

Cray

-O3 -hfp3

Intel

-fast

GNU

-Ofast -funroll-loops

Vectorisation, which is one of the important compiler optimisations for ARCHER, is enabled by default as follows:

Cray

At -O1 and above

Intel

At -O2 and above

GNU

At -O3 and above or when using -ftree-vectorize

To promote integer and real variables from four to eight byte precision for FORTRAN codes the following compiler flags can be used:

Cray

-s real64 -s integer64

Intel

-real-size 64 -integer-size 64 -xAVX

(Sometimes the Intel compiler incorrectly generates AVX2 instructions if the -real-size 64 or -r8 options are set. Using the -xAVX option prevents this.)

GNU

-freal-4-real-8 -finteger-4-integer-8

4.5.4 Cray compiler for C++ 11

Compiling C++ code to C++ 11 standards is possible with the Cray compiler on ARCHER. Use the flag

-h std=c++11

4.5.5 Intel compiler and GCC for C++ 11

Compiling C++ code to C++11 standards is possible with the GNU and Intel compilers on ARCHER. Both use the GNU flag -std=c++11, however the Intel compiler relies on the GNU header files for C++11 support which means that you need to load the gcc module as well as the Intel Programming Environment on ARCHER to perform such compilations. This can be done using the command module load gcc with the PrgEnv-intel module loaded to ensure the Intel compilers are used. The gcc module version to load depends on the intel module version loaded:

Compatible for C++11?	gcc/4.8.1	gcc/4.9.2	gcc/4.9.3	gcc/5.1.0 (default until 9 August 2017)	gcc/5.3.0	gcc/6.1.0	gcc/6.3.0 (default from 9 August 2017)
intel/14.0.4.211	no	yes	yes	no	no	no	no
intel/15.0.2.164 (default until 9 August 2017)	no	yes	yes	no	no	no	no
intel/16.0.2.181	no	yes	yes	yes	yes	no	no
intel/17.0.0.098 (default from 9 August 2017)	no	yes	yes	yes	yes	yes	yes

For example, for Intel 16 use

module switch PrgEnv-cray PrgEnv-intel
module switch intel/16.0.2.181
module load gcc/5.3.0

From 9 August 2017, the default Intel and Gnu modules will be compatible and you will only need to use

module switch PrgEnv-cray PrgEnv-intel
module load gcc

4.6 Using dynamic linking/libraries

By default, executables on ARCHER are built using static libraries (that is, all of the object code of referenced libraries are contained in the executable file) when using the wrapper scripts. This has the advantage that once an executable is created, whenever it is run in the future, it will always use the same object code and thus give the same results from the same input. However, executables compiled with static libraries have the potential disadvantage that when multiple instances are running simultaneously multiple copies of the libraries used are held in memory. This can lead to large amounts of memory being used to hold the executable and not application data.

Alternatively, applications can be compiled to use shared/dynamic libraries (i.e. libraries which are loaded at run-time as and when needed by the application). This may be because static versions of certain libraries are unavailable, or to reduce the amount of memory executables take by sharing common sections of object codes between applications which use the same library.

To create an application that uses shared/dynamic libraries you must pass extra flags during compilation, or set an environment variable. You can either:

1. Use the -dynamic flag when invoking the compiler for linking.
2. Set the environment variable CRAYPE_LINK_TYPE=dynamic without any extra compilation/linking options.

By default an application compiled this way to use shared libraries will use the default version of the library installed on the system (just like any other Linux executable), even if the system modules were set differently at compile time. This means that the application may potentially be using slightly different object code each time the application runs as the defaults may change. This is usually the desired behaviour for many applications as any fixes or improvements to the default linked libraries are used without having to recompile the application, however some users may feel this is not the desired behaviour for their applications.

Instead, users may choose to hard code the location of the libraries used during compilation and linking into the resulting executable by setting the CRAY_ADD_RPATH=yes environment variable. The same object code, from the original shared libraries, will be used each time the executable is run, as long as the library files exist. This is very similar to how statically linked executables behave.

Finally, users may wish to choose which version of a library is used through the standard module environment. This is done prepending the value of CRAY_LD_LIBRARY_PATH to the standard LD_LIBRAY_PATH environment variable once all the modules are loaded. This will place all the paths to libraries specified in the modules files ahead of the defaults and allow the Operating System to select the specified versions from the modules in place of the defaults.

Use the UNIX command ldd exe_file to check whether you are using an executable that depends on shared libraries. This utility will also report the shared libraries this executable will use with the current value of LD_LIBRARY_PATH (not CRAY_LD_LIBRARY_PATH ).

4.7 Compiling for Postprocessing/Serial Nodes

If you see an "Illegal Instruction" error when trying to use the postprocessing/serial nodes then it means you are trying to use code compiled for the compute nodes. Please follow the instructions below to compile the code for the postprocessing/serial nodes.

If you wish to use serial executables you compile yourself on the postprocessing/serial nodes (accessed through the Serial Jobs mechanism detailed in the Job Submission System chapter) then you need to compile on the postprocessing/serial nodes themselves using the bare compiler commands rather than the compiler wrapper scripts (i.e. cc, ftn, CC) provided in the Cray Application Developer Environment. This is beacuse the processor architecture used on these nodes is different from that used on the ARCHER login, service and compute nodes.

You can access the postprocessing/serial nodes for compilation either by submitting a serial job with the compile commands in the script or, if you prefer to compile interactively, by submitting an interactive job to the serial queues. Instructions on how to write postprocessing/serial job scripts and on how to run interactive jobs can be found in the Job Submission System chapter at:

• Example Serial Job Submission Script
• Interactive Serial Jobs

The compiler commands you need depend on the programming environment you wish to use. Note, that you must still switch programming environments as described above to be able to access the bare compiler commands. The table below lists the compiler commands for the different programming environments.

Note that the Cray compiler environment does not support compiling code for the architecture on the PP nodes so you should only use the GNU or Intel programming environments.

Note AVX instructions are not available on the serial/PP nodes.

Compiler Suite	Fortran	C	C++	Notes
PrgEnv-gnu	gfortran	gcc	g++
PrgEnv-intel	ifort	icc	icc
PrgEnv-cray	Unsupported	Unsupported	Unsupported	Not supported on PP nodes.

For example, to compile a single source file interactively using the GNU compilers (remember to change the budget code to your own one):

user@eslogin005:~> ssh espp2

user@esPP002:~> module swap PrgEnv-cray PrgEnv-gnu
user@esPP002:~> gfortran -o simple_prog.x simple_prog.f90

4.7.1 Including Libraries in Serial Builds

Please note that header and library paths set by the module enviornment do not apply to the bare compilers. You may see errors relating to missing includes such as:

user@eslogin006:~> module load cray-netcdf
user@eslogin006:~> gcc test.c
test.c:2:20: fatal error: netcdf.h: No such file or directory
 #include <netcdf.h>
                    ^
compilation terminated.

or

user@eslogin006:~> module load cray-netcdf
user@eslogin006:~> icc test.c
test.c(2): catastrophic error: cannot open source file "netcdf.h"
  #include <netcdf.h>
                     ^

compilation aborted for test.c (code 4)

You can resolve this by explicitly specifying the paths when compiling. Model your commands on the following:

gcc -I/opt/cray/netcdf/4.3.0/GNU/48/include -L/opt/cray/netcdf/4.3.0/GNU/48/lib -lnetcdff -lnetcdf test.c

icc -I/opt/cray/netcdf/4.3.0/INTEL/130/include -L/opt/cray/netcdf/4.3.0/INTEL/130/lib -lnetcdff -lnetcdf test.c

To locate the paths you need, "module show " can be used to give a listing of all environment changes a module makes when loaded. Here:

user@eslogin006:~> module show cray-netcdf
...
prepend-path     CRAY_LD_LIBRARY_PATH /opt/cray/netcdf/4.3.0/INTEL/130/lib
-------------------------------------------------------------------

indicates the cray-netcdf library path for the Intel compiler. These paths may change depending on the currently loaded PrgEnv module.

4.8 Switching to older/newer programming environment releases

The default programming environment (compilers, libraries, etc.) is periodically updated on ARCHER by changing the default versions of many modules. Users are informed in advance of these changes via e-mail and Twitter. Sometimes you may wish to switch back to an old programming environment release (as you code has a dependency on a particular compiler/library version that you have not yet been able to fix) or switch to a newer programming environment release to test compatibility before the defaults on the system are changed.

You can do this by loading special cdt modules. For example, to switch to the Dec 2018 programming environment you would use:

user@eslogin006:~> module load cdt/18.12

Once this command has been issued, module list should reveal that you now have the new versions of modules loaded. You can see which Programming Enviroments are available by searching for the cdt modules:

user@eslogin006:~> module avail cdt


----------------------------------------------- /opt/cray/modulefiles -----------------------------------------------
cdt/17.05 cdt/17.12 cdt/18.12

------------------------------------------- /opt/modules/packages-archer --------------------------------------------
cdt/15.11

4.9 Compiling in batch jobs using the ppn queue

Sometimes you may wish to compile in a batch job; for example, the compile process may take a long time or the compile process is part of the research workflow and can be coupled to the production job. You can use the ppn queue on ARCHER to do this. The nodes assigned to the ppn queue are compute nodes repurposed for serial processes (such as compilation). Unlike standard compute nodes:

• The /home file system is available
• Multiple users can share a node
• You should not use the aprun command to launch processes onto the node
• You must specify the qsub option -q ppn to access the nodes

An example job submission script for a compile job using make (assuming the Makefile is in the same directory as the job submission script) would be:

#!/bin/bash
#
#PBS -N ppn_example
#PBS -l select=1
#PBS -l walltime=0:20:0
#PBS -q ppn
# Replace with your budget code
#PBS -A t01

cd $PBS_O_WORKDIR

make clean
make 

3. Resource Management | Contents | 5. Running Jobs