Python software on Archer

For running Python on work nodes on Archer, there are python-compute (native) and anaconda-compute modules available. Users are discouraged from using anaconda-compute because it is not optimised for running on Archer. There are some preinstalled packages for python-compute and they begin with pc-. For installation of additional packages, virtual environments are encouraged.

For more info see: http://www.archer.ac.uk/documentation/user-guide/python.php

When compiling Python software (like PySCF) Archer by default builds all libraries as static libraries. This leads to errors like:

 ImportError Cannot import name ...

or

 File "/work/y07/y07/cse/numpy/1.9.2-libsci/lib/python2.7/site-packages/numpy/ctypeslib.py", line 128, in load_library
   raise OSError("no file with expected extension")
 OSError: no file with expected extension

and other errors.

To prevent dynamic libraries from becoming static you must:

 export CRAYPE_LINK_TYPE=dynamic

before the compilation. For more information see: http://www.archer.ac.uk/documentation/user-guide/development.php#sec-4.6

In submission script itself, do not forget to:

1) Load python and all modules supplied centrally

 module load python-compute
 module load pc-numpy
 module load pc-scipy

2) Enter virtual environment containing all additional modules (AFTER loading python and the central modules)

 source /work/e507/e507/ap837/code/venv_pyscf/bin/activate

3) Use aprun to run the job, otherwise it will run only on the shared job launcher node and not on the compute nodes.

4) Create a tmp folder in the work directory and set the environmental variable to pint to it. The compute nodes do not have access to the regular tmp folders.

 export TMPDIR=/work/e507/e507/ap837/tmp

5) If code uses intel libraries: (errors like:

 OSError: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
 OSError: libiomp5.so: cannot open shared object file: No such file or directory

) do:

 source /opt/intel/bin/compilervars.sh intel64

6) When using numpy, the following error may occur:

 Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so

Tha solution is to do:

 export LD_PRELOAD=/opt/intel/mkl/lib/intel64/libmkl_core.so:/opt/intel/mkl/lib/intel64/libmkl_sequential.so

as found here http://debugjournal.tumblr.com/post/98401758462/intel-mkl-dynamic-link-library-error.

7) For PySCF, since we are using old numpy, you need to comment out warning in __init__.py in the pyscf folder

 #if LooseVersion(numpy.__version__) <= LooseVersion('1.8.0'):
 #    raise SystemError("You're using an old version of Numpy (%s). "
 #                      "It is recommended to upgrad numpy to 1.8.0 or newer. \n"
 #                      "You still can use all features of PySCF with the old numpy by removing this warning msg. "
 #                      "Some modules (DFT, CC, MRPT) might be affected because of the bug in old numpy." %
 #                      numpy.__version__)

Setting up virtual environment for PySCF

In this order:

1) Load modules

  module load python-compute pc-numpy pc-scipy

[2) Create virtual environment]

  virtualenv --system-site-packages pyscfEnv

3) Enter virtual environment

  source /work/e507/e507/$USER/pyscfEnv/bin/activate
  export CRAYPE_LINK_TYPE=dynamic

4) Update pip

  pip install pyscf

3) Install h5py, ase or other necessary modules