Compiling OpenFOAM 16.12 on ARCHER

This page outlines the procedure we used for building OpenFOAM 16.12 on ARCHER. We downloaded the OpenFOAM-1612+ and ThirdParty-1612+ sources from the website and unpacked them on the /work filesystem on ARCHER.

Modifications

To build OpenFOAM we needed to make some small modifications to the configuration files. Firstly, we change the following line in etc/bashrc:

Set this to the location of your OpenFOAM directory
FOAM_INST_DIR=

Add a file, in the etc directory, named prefs.sh, with the following entries:

export WM_COMPILER=Gcc
export WM_MPLIB=CRAY-MPICH

We ensured that the following was set in etc/config.sh/CGAL:

boost_version=boost-system
cgal_version=CGAL-4.9

export BOOST_ARCH_PATH=/work/y07/y07/cse/boost/1.60_MPI/GNU/boost
export CGAL_ARCH_PATH=$WM_THIRD_PARTY_DIR/platforms/$WM_ARCH$WM_COMPILER/$cgal_version

We ensured that the following was set in etc/config.sh/FFTW:

fftw_version=fftw-system

We added the following in to etc/config.sh/mpi before the existing MPICH) entry:

 CRAY-MPICH)
    export FOAM_MPI=cray-mpich
    export MPI_ARCH_PATH=$MPICH_DIR
    _foamAddLib     $MPI_ARCH_PATH/lib
    ;;

We added the following in to etc/config.sh/settings after the case "$WM_COMPILER_TYPE" in section:

if [ "$FOAM_MPI:$WM_COMPILER" = "cray-mpich:Gcc" ]
then
    WM_CC="cc"
    WM_CXX="CC"
fi 

Next we changed the c and c++ files in wmake/rules/linuxGcc replacing gcc with cc and g++ with CC.

The same changes were made in the wmake/rules/linux64Gcc directory.

Some changes were also necessary in ThirdParty-16.12+ directory. We updated ThirdParty-16.12+/etc/wmakeFiles/scotch/Makefile.inc.i686_pc_linux2.shlib-OpenFOAM replacing gcc with cc and mpicc with cc.

We also had to change the build rules in ThirdParty-16.12+/etc/tools/ThirdPartyFunctions to make sure too many processes were not being used. We had issues with errors like this, caused by make trying to use as many cores as the machine has available for compilation (which is not compatible with the process limits users have on login or compile nodes):

Error: Problem invoking pkg-config.
sh: fork: Resource temporarily unavailable
....
Error invoking pkg-config!
make[2]: *** Waiting for unfinished jobs....

Therefore, we change export WM_NCOMPPROCS in ThirdParty-16.12+/etc/tools/ThirdPartyFunctions to be export WM_NCOMPPROCS=8 to fix the number of processes the build process could use, allowing the scotch library to be built.

Finally, one source file needed to be changed to fix an issue with a clash between C and C++ MPI library definitions. We updated OpenFOAM-v1612+/src/parallel/decompose/ptscotchDecomp/ptscotchDecomp.C, changing this:

extern "C"
{
    #include 
    #include 
    #include "ptscotch.h"
}

To this:

#include 

extern "C"
{
    #include 
    #include "ptscotch.h"
}

Build OpenFOAM

We used the following modules and environment variables when building:

module load PrgEnv-gnu
module load zlib
module load boost/1.60
module load cmake/3.2.3
module load make
module load cray-tpsl
module load fftw
export WM_64=ON
export CRAYPE_LINK_TYPE=dynamic

To build OpenFOAM you then need to source to etc/bashrc in the OpenFOAM directory:

source etc/bashrc

And then run Allwmake from the OpenFOAM directory:

./Allwmake