ARCHER User Guide

This guide explains how to make use of the ARCHER facility. It is designed with the needs of new users in mind, but will also be a useful reference for everyone.

The following screencast demonstrates how to access help in the ARCHER terminal once you are connected:

Contents

1. Introduction

An introduction to the ARCHER facility. This includes links to information on the hardware and software installed on the system and basic information on the use of the system.

• 1.1 Acknowledging ARCHER
• 1.2 Hardware
• 1.3 Disconnected tty/terminal processes will be terminated
• 1.4 Outgoing SSH connections will be terminated
• 1.5 bash is the only supported shell on ARCHER

2. Connecting and Transferring Data

How to access the ARCHER facilities and transfer data to and from ARCHER.

• 2.1 Interactive access
  • 2.1.1 Interactive access to Post Processing nodes
• 2.2 Non-interactive SSH access
• 2.3 Transferring data to and from ARCHER
  • 2.3.1 Performance considerations
• 2.4 Making access more convenient using the SSH configuration file

3. Resource Management

Information on using budgets (for CPU time), the ARCHER filesystems and managing your resources using the ARCHER SAFE.

• 3.1 The ARCHER administration web site (SAFE)
• 3.2 Checking your CPU-time (kAU) quota
• 3.3 ARCHER Filesystems
• 3.3.1 /home Filesystem
• 3.3.2 /work Filesystem
• 3.3.3 Research Data Facility
• 3.4 Disk quotas
• 3.4.1 Checking disk quotas
• 3.4.2 Allocating quotas to specific groups and users
• 3.5 File permissions and security
• 3.6 Sharing data with other ARCHER users
  • 3.6.1 Sharing data with users in your project
  • 3.6.2 Sharing data with all users
• 3.7 Sharing data between systems
• 3.8 File IO Performance Guidelines
• 3.9 Data archiving
• 3.10 Use of /tmp
• 3.11 Backup policies

4. Application Development Environment

Using the application development environment on ARCHER: using modules to control compilers and libraries, compiling and linking code, accessing debugging and performance analysis tools.

• 4.1 Information on the available modules
• 4.2 Loading, unloading and swapping modules
• 4.3 Things to watch out for
• 4.4 Compiler wrapper scripts
• 4.5 Available compiler suites
• 4.5.1 Switching compiler suites
• 4.5.2 Changing compiler versions
• 4.5.3 Useful compiler options
• 4.5.4 Cray compiler for C++ 11
• 4.5.5 Intel compiler and GCC for C++ 11
• 4.6 Using dynamic linking/libraries
• 4.7 Compiling for Postprocessing/Serial Nodes
  • 4.7.1 Including Libraries in Serial Builds
• Switching to older/newer programming environment releases
• 4.9 Compiling in batch jobs using the ppn queue

5. Running Jobs

How to run jobs on ARCHER facilities and more information on the job submission system.

• 5.1 Using PBS Pro
  • 5.8.1 Using checkQueue
• 5.2 Output from PBS jobs
• 5.3 bolt: Job submission script creation tool
  • 5.3.1 Basic usage
  • 5.3.2 Further help
• 5.4 Running Parallel Jobs
  • 5.4.1 PBS Submission Options
  • 5.4.2 Parallel job launcher aprun
  • 5.4.3 Task affinity for "unpacked" jobs
  • 5.4.4 Example: job submission script for MPI parallel job
  • 5.4.5 Example: job submission script for MPI parallel job on large memory nodes
  • 5.4.6 Example: job submission script for OpenMP parallel job
  • 5.4.7 Example: job submission script for MPI+OpenMP (mixed mode) parallel job
  • 5.4.8 Interactive Jobs
• 5.5 Array Jobs
• 5.6 Sharing Nodes with OpenMP/Threaded Jobs
• 5.7 Python Task Farm
• 5.8 Job Submission System Layouts and Limits
  • 5.8.1 Scheduling System Priorities and Logic
• 5.9 checkScript: Script validation tool
• 5.10 Setting a time limit for aprun
• 5.11 Low Priority Access
• 5.12 Long Queue Access
• 5.13 Short (debug) Queue Access
• 5.14 Reservations
• 5.15 Serial Queues
  • 5.15.1 Example Serial Job Submission Script
  • 5.15.2 Interactive Postprocessing Job
• 5.16 OOM (Out of Memory) Error Messages

6. Using Python on ARCHER

This chapter describes how to use the Python programming language on ARCHER on both the login and post-processing nodes and on the compute nodes.

• 6.1 Deciding which modules to use
• 6.2 Python for data analysis: the Anaconda distributions
  • 6.2.1 Packages included in Anaconda distributions
  • 6.2.2 Adding packages to the Anaconda distribution
• 6.3 Python for HPC: the native distributions
  • 6.3.1 Python packages for native distributions
  • 6.3.2 Adding packages to the native distribution
  • 6.3.3 Anaconda on the compute nodes
• 6.4 Custom environments
  • 6.4.1 Installing Microconda
  • 6.4.2 Installing packages into Miniconda
  • 6.4.3 Custom Python environments with MPI

7. Linux Help

This chapter gives some help on using Linux on ARCHER.

• 7.1 Linux Quick Reference Sheet
• 7.2 Finding Linux Help and Documentation from the Command Line
• 7.3 Linux on ARCHER - some hints and tips
  • 7.3.1 File meta-data