Running Jobs on Bassi

On this page

• Introduction
• MPI Codes
• OpenMP
• MPI I/O

Related pages

• Job Launch Overview
• Communication Protocols
• Runtime Configuration and Options
• Checkpoint/restart considerations
• Interactive Jobs
• Batch Jobs
• Memory Considerations
• Class Info & Policies
• Monitoring Jobs
• IBM Error Messages

Introduction

NERSC's IBM POWER 5 p575 system, named Bassi, has 111 SMP "compute" nodes with 8 processors per node for a total of 888 processors. Each node has a common pool of 32 GBytes of memory.

The nodes are connected via a high-speed network known as HPS (also called Federation). Each node has a "dual-link" HPS adapter, which attaches to a dual-plane network. Each link can support 2 GB/s of data transfer in each direction.

Bassi supports both interactive and batch processing. Interactive jobs are limited to 4 nodes and 30 minutes of wallclock time. Parallel programs can be run either using distributed memory message passing, shared memory threading, or some combination of the two.

MPI Codes

Most programs running on Bassi execute in parallel and use the Message Passing Interface, or MPI, to communicate among separate tasks. The default computing environment is configured to support MPI jobs as transparently as possible. The discussions on these pages assumes you are running an MPI code unless otherwise noted.

OpenMP Codes

Interactive jobs that use OpenMP must run with the the environment variable MP_TASK_AFFINITY unset. As of August 2, 2006 (Parallel Environment 3.3.2.4) jobs run through LoadLeveler batch scripts ignore this environment variable. Make sure that you do not use the "rset" keyword to request task affinity in your batch script.

THESE SETTINGS ARE WRONG FOR OPENMP CODES. DO NOT USE FOR OPENMP CODES.
#@ rset = rset_mcm_affinity
#@ mcm_affinity_options = mcm_distribute, mcm_mem_pref, mcm_sni_none
THESE SETTINGS ARE WRONG FOR OPENMP CODES. DO NOT USE FOR OPENMP CODES.

For interactive jobs use the appropriate statement:

unsetenv MP_TASK_AFFINITY               (for csh, tcsh)

unset    MP_TASK_AFFINITY               (for sh, ksh, bash)

The default number of OpenMP threads per task (process) is 8. If you are running a pure OpenMP code, you should run using one task per node. If you are using more than one task per node, you should reduce OMP_NUM_THREADS so that the product of tasks x threads is 8. A combination of tasks and threads significantly in exess of 8 may result in poor performance.

OpenMP code will probably performer better with the following environment variable setting:

export XLSMPTOPTS=SPINS=0;YIELDS=0 (for sh-like shells)

setenv XLSMPTOPTS 'SPINS=0;YIELDS=0' (for csh-like shells)

If you are calling malloc() or ALLOCATE in OpenMP threads, you may want to set this environment variable:

export	MALLOCMULTIHEAP=true (sh-like shells)

setenv MALLOCMULTIHEAP true (csh-like shells)

Otherwise, memory allocations will be serialized on a node.

MPI-I/O Codes

Jobs that use MPI-I/O or explict MPI threading routines must run with the the environment variable MP_SINGLE_THREAD unset. As of August 10, 2006 MP_SINGLE_THREAD is no longer set in the default Bassi environment.