NWChem at NERSC

NWChem is a computational chemistry package that is designed to run on high-performance parallel supercomputers as well as conventional workstation clusters. It aims to be scalable both in its ability to treat large problems efficiently, and in its usage of available parallel computing resources.

NWChem was developed by the High-Performance Computational Chemistry group in the Environmental Molecular Sciences Laboratory (EMSL) at the Pacific Northwest National Laboratory (PNNL).

NWChem Input

The NWChem User's manual contains a detailed description of NWChem input. It is available for online browsing or downloadable as postscript.

Running NWChem at NERSC

Usually, a number of different versions of NWChem are installed on the systems. Each version can be made available by loading a different module. To use the default and recommended version, use the command:

% module load nwchem

To see where the nwchem executables reside (the bin directory) and what environment variables it defines, type

% module show nwchem

nid04580 z/zz217> module show nwchem
-------------------------------------------------------------------
/usr/common/usg/Modules/modulefiles/nwchem/5.1:

module-whatis    NWChem Computational Chemistry Package 
setenv           NWCHEM_DIR /usr/common/usg/nwchem/5.1 
setenv           NWCHEM_NWPW_LIBRARY /u0/z/zz217/USG/nwchem/5.1/data/ 
setenv           NWCHEM_BASIS_LIBRARY /u0/z/zz217/USG/nwchem/5.1/data/libraries/ 
setenv           XT_SYMMETRIC_HEAP_SIZE 1G 
setenv           XT_LINUX_SHMEM_HEAP_SIZE 1G 
prepend-path     PATH /usr/common/usg/nwchem/5.1/bin 
-------------------------------------------------------------------

When you first start to use NWChem, you should copy a default ".nwchemrc" file to your home directory. This file is used to set certain default options for the program:

% cp ${NWCHEM_DIR}/default.nwchemrc ~/.nwchemrc

Running on Franklin

There are two ways of running nwchem on franklin, submitting a batch job, or running interactively in an interactive batch session.
1) Sample batch script to run nwchem on franklin using single core:

#PBS -S /usr/bin/tcsh 
#PBS -N test_nwchem
#PBS -q debug
#PBS -l mppwidth=32
#PBS -l mppnppn=1
#PBS -l walltime=00:30:00
#PBS -j eo
#PBS -V

cd $PBS_O_WORKDIR
module load nwchem
setenv XT_SYMMETRIC_HEAP_SIZE 1G
setenv XT_LINUX_SHMEM_HEAP_SIZE 1G

aprun -n 32 -N 1 nwchem test1.inp > &! test1.out

% qsub  test_nwchem.pbs 

If you wish to use dual core, sample script as follows:

#PBS -S /usr/bin/tcsh
#PBS -N test_nwchem
#PBS -q debug
#PBS -l mppwidth=32
#PBS -l walltime=00:30:00
#PBS -j eo
#PBS -V

cd $PBS_O_WORKDIR
module load nwchem
setenv XT_SYMMETRIC_HEAP_SIZE 500000000
setenv XT_LINUX_SHMEM_HEAP_SIZE 500000000 

aprun -n 32 nwchem test1.inp > &! test1.out

% qsub  -I -V -q interactive -l mppwidth=32,mppnppn=1 

%  cd $PBS_O_WORKDIR 
%  module load nwchem 
%  aprun -n 32 -N 1 nwchem 

% qsub  -I -V -q interactive -l mppwidth=32 

%  cd $PBS_O_WORKDIR 
%  module load nwchem 
%  setenv XT_SYMMETRIC_HEAP_SIZE 500000000 
%  setenv XT_LINUX_SHMEM_HEAP_SIZE 500000000 
%  aprun -n 32 nwchem 

Running on Bassi

Again, there are two ways of running nwchem on bassi, interactively or by sumitting a batch job.

% nwchem -procs procs -nodes nodes input > output

procs is the number of processors and nodes is the number of nodes (usually 8 processors per node on bassi you want to run on, input is the name of the input file (if the file has an extension of .nw, this can be omitted), and output is the output file.

Note: the input filename is a command line argument, it is not read as standard input.

By default, NWChem will write scratch files into the current directory of the running job. To change this via NWChem directives, see section 5.2 of the user manual.

Here is an example batch script for Bassi. Note that this script changes to your scratch directory before running.

For Bassi:

#@ job_name        = myjob
#@ output          = myjob.out
#@ error           = myjob.err
#@ job_type        = parallel
#@ notification    = complete
#@ network.LAPI=csss,shared,us
#@ network.MPI=csss,shared,us
#@ environment     = COPY_ALL
#@ environment="MP_CSS_INTERRUPT=YES; RT_GRQ=ON"
#@ class           = regular
#@ total_tasks     = 32
#@ node            = 4
#@ rset = RSET_MCM_AFFINITY
#@ mcm_affinity_options=mcm_distribute,mcm_mem_pref,mcm_sni_none
#@ bulkxfer=yes
#@ wall_clock_limit= 00:30:00
#@ queue

module load nwchem
cd $SCRATCH
poe nwchem $HOME/nwchem/b1.in > $HOME/nwchem/b1.out
ls -l

There are many other parameters to the Bassi batch system. For more details see The Bassi Batch System.

Running on Jacquard

On jacquard, we recommend launching an interactive job and running nwchem as shown in the batch script below.

Here is an example batch script for the jacquard. Note that this script changes to your scratch directory before running.

For jacquard:

#!/bin/bash -l 
#PBS -l nodes=8:ppn=2,walltime=30:00
#PBS -N medium 
#PBS -j oe 
#PBS -q batch
 
cd $SCRATCH
module load nwchem
nwchem=$(which nwchem)
mpirun -np 16 $nwchem ~/medium.nw > medium.out
ls -l

There are many other parameters to the PBS batch system. For more details see Running Jobs on Jacquard.

Disk Usage for Direct and Semi-Direct Calculations

The default setting for SCF calculations is to cache integrals in memory. If this is not possible, then integrals are written to disk. By default, there is a quota of 100Gb in /scratch on the SP. For this reason, it is a good idea to use the scf block subdirective semidirect to specify how big the integral files can be. For example:

scf
semidirect filesize 1000000
end 

sets the maximum size of the integral file to 1000000 words (8000000 bytes) per processor. For more details see section 10.12 of the user manual.

On Bassi, the filesystem /scratch (GPFS disk) offers much better performance than /scr (local SCSI disk). /scratch also has considerably more space.