FEniCS

Details on the FEniCS package. How to access and compile the package.

Useful Links

• FEniCS web page
• FEniCS documentation
• General instructions on building FEniCS from source

Licensing

FEniCS is open source software.

Access

As it is open source software, the FEniCS source and binaries are available to all users.

Versions and Modules

Note that when switching between FEniCS versions, you must clear your Instant cache directory. Default: /work/group/group/user/.instant. Failure to do so may cause your jobs to crash with errors similar to:

ImportError:No module named ffc_form_0451d2343048e2eaab5477f621855f2199efedc9;
AssertionError: Failed to import module found in cache. Modulename: 'ffc_form_0451d2343048e2eaab5477f621855f2199efedc9';

We have the following versions of FEniCS installed on ARCHER:

• 1.5.0
• 1.6.0 (default)

These are available as the following modules on ARCHER:

• 1.5.0: fenics/1.5.0
• 1.6.0: fenics or fenics/1.6.0

Compiling FEniCS

The rest of this page discusses the steps we followed to built FEniCS 1.4.0 on ARCHER.

Set up environment

FEniCS was built with the GNU compiler on ARCHER.

Modules loaded:

Currently Loaded Modulefiles:
  1) modules/3.2.6.7
  2) eswrap/1.1.0-1.010400.915.0
  3) switch/1.0-1.0501.47124.1.93.ari
  4) craype-network-aries
  5) craype/2.1.1
  6) pbs/12.1.400.132424
  7) craype-ivybridge
  8) cray-mpich/6.3.1
  9) packages-archer
 10) budgets/1.1
 11) checkScript/1.1
 12) bolt/0.5
 13) serialJobs/1.0
 14) python/2.7.6
 15) tkdiff/4.2
 16) nano/2.2.6
 17) imagemagick/6.8.8-2
 18) epcc-tools/1.0
 19) arpack/1.0
 20) gcc/4.8.2
 21) cray-libsci/12.2.0
 22) udreg/2.3.2-1.0501.7914.1.13.ari
 23) ugni/5.0-1.0501.8253.10.22.ari
 24) pmi/5.0.3-1.0000.9981.128.2.ari
 25) dmapp/7.0.1-1.0501.8315.8.4.ari
 26) gni-headers/3.0-1.0501.8317.12.1.ari
 27) xpmem/0.1-2.0501.48424.3.3.ari
 28) job/1.5.5-0.1_2.0501.48066.2.43.ari
 29) csa/3.0.0-1_2.0501.47112.1.91.ari
 30) dvs/2.4_0.9.0-1.0501.1672.2.122.ari
 31) alps/5.1.1-2.0501.8507.1.1.ari
 32) rca/1.0.0-2.0501.48090.7.46.ari
 33) atp/1.7.2
 34) PrgEnv-gnu/5.1.29
 35) cray-hdf5-parallel/1.8.12
 36) cray-petsc/3.4.3.1
 37) eigen/3.2.1
 38) ply/3.4
 39) numpy/1.8.0
 40) boost/1.55
 41) swig/3.0.2
 42) petsc4py/3.4
 43) slepc/3.4.4
 44) cray-tpsl/1.4.0
 45) scientific-python/2.9.3
 

Building CMake

The FEniCS module provides its own version of CMake, accessible on the compute nodes to enable Just-In-Time compilation. Please note: a version of CMake other than v3.0.0 must be used due to a bug affecting the FEniCS build process.

cd cmake-3.0.1
CC=cc CXX=CC ./bootstrap --prefix=/path/to/fenics/install/directory && make && make install

Python Packages

FEniCS depends on the FFC, FIAT, Instant and UFL Python modules. Each is available from the project main site and all follow the same install process:

python setup.py install --prefix=/path/to/fenics/install/directory

DOLFIN

DOLFIN is the main FEniCS library and should be installed following all other dependencies. Some modification is required to enable compilation on ARCHER. Please note: all line numbers reference FEniCS v1.4.0.

Changes to cmake/modules/FindPETSc.cmake

Add:

set(PETSC_LIB_BASIC "-lcraypetsc_gnu_real")

as line 155, i.e. directly following line:

petsc_get_variable(PETSC_LIB_BASIC PETSC_LIB_BASIC)

and add:

set(PETSC_VERSION "3.4")
set(PETSC_VERSION_MAJOR "3")
set(PETSC_VERSION_MINOR "4")

(assuming use of cray-petsc/3.4.3.1) as lines 210-212, i.e. directly following:

set(PETSC_VERSION "UNKNOWN")

Changes to CMakeLists.txt

Add:

add_definitions( -DNUMPY_VERSION_MINOR=8 )

as line 6, i.e. directly following header comments and the cmake version check.

Compilation of DOLFIN

Create and run the following script in the DOLFIN source root directory:

#!/bin/sh
#
# Configure for a local install. Build files will be generated in the
# directory $toplevel/build and installation will go in the directory
# $toplevel/local.
#
# This is useful to be able to run demos from within the source tree
# without needing to specify an installation directory.
#
# This script is what most developers use to build and rebuild DOLFIN.

# Number of processes to use during build
: ${PROCS:=4}

# Extract any extra argument
CMAKE_EXTRA_ARGS=$@

# Set installation and build directory
INSTALLDIR=/path/to/fenics/install/directory
BUILDDIR=$PWD/build
echo "Build and install in $BUILDDIR, $INSTALLDIR."
echo

# Configure
mkdir -p $BUILDDIR
cd $BUILDDIR
cmake -DCMAKE_INSTALL_PREFIX=$INSTALLDIR \
      -DDOLFIN_SKIP_BUILD_TESTS=ON \
      -DDOLFIN_AUTO_DETECT_MPI=false \
      -DDOLFIN_ENABLE_CGAL=false \
      -DDOLFIN_ENABLE_VTK=false \
      -DDOLFIN_ENABLE_PASTIX=false \
      -DDOLFIN_ENABLE_SPHINX=false \
      -DDOLFIN_ENABLE_UMFPACK=false \
      -DDOLFIN_ENABLE_CHOLMOD=false \
      -DDOLFIN_ENABLE_TRILINOS=false \
      -DDOLFIN_ENABLE_UNIT_TESTS=false \
      -DDOLFIN_ENABLE_BENCHMARKS=false \
      -DCMAKE_CXX_FLAGS_RELEASE=-Wno-literal-suffix \
      -DCMAKE_BUILD_TYPE=Release \
      -DDOLFIN_DEPRECATION_ERROR=true \
      -Wno-dev \
      $CMAKE_EXTRA_ARGS \
      ..

# Build
make -j$PROCS && make install -j$PROCS

Run Environment

These variables should be set before launching FEniCS. e.g. In the application module file.

Instant cache and error directories must be redirected to a location available on the compute nodes:

export INSTANT_CACHE_DIR=/work/group/group/user/.instant
export INSTANT_ERROR_DIR=/work/group/group/user/.instant

The method used by the instant module to call to external programs (swig, cmake, etc.) must be changed from the default SUBPROCESS (pipes) to OS_SYSTEM (temp files). This is needed to address problems relating to the Aries interconnect seen when running certain FEniCS codes on >1 ARCHER node.

export INSTANT_SYSTEM_CALL_METHOD="OS_SYSTEM"

More details at:

• Thread on FEniCS Mailing List
• Related Pull Request