Thom Group Wiki - User contributions [en]

Python software on Archer

2017-09-05T13:41:26Z

Ap837:

For running Python on work nodes on Archer, there are python-compute (native) and anaconda-compute modules available. Users are discouraged from using anaconda-compute because it is not optimised for running on Archer. There are some preinstalled packages for python-compute and they begin with pc-. For installation of additional packages, virtual environments are encouraged.

For more info see: http://www.archer.ac.uk/documentation/user-guide/python.php

When compiling Python software (like PySCF) Archer by default builds all libraries as static libraries. This leads to errors like:

ImportError Cannot import name ...

or
File "/work/y07/y07/cse/numpy/1.9.2-libsci/lib/python2.7/site-packages/numpy/ctypeslib.py", line 128, in load_library
raise OSError("no file with expected extension")
OSError: no file with expected extension

and other errors.

To prevent dynamic libraries from becoming static you must:

export CRAYPE_LINK_TYPE=dynamic

before the compilation. For more information see: http://www.archer.ac.uk/documentation/user-guide/development.php#sec-4.6

== In submission script itself, do not forget to: ==

1) Load python and all modules supplied centrally
module load python-compute
module load pc-numpy
module load pc-scipy
2) Enter virtual environment containing all additional modules (AFTER loading python and the central modules)
source /work/e507/e507/ap837/code/venv_pyscf/bin/activate

3) Use aprun to run the job, otherwise it will run only on the shared job launcher node and not on the compute nodes.

4) Create a tmp folder in the work directory and set the environmental variable to pint to it. The compute nodes do not have access to the regular tmp folders.
export TMPDIR=/work/e507/e507/ap837/tmp

5) If code uses intel libraries: (errors like:
OSError: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
OSError: libiomp5.so: cannot open shared object file: No such file or directory
) do:
source /opt/intel/bin/compilervars.sh intel64

6) When using numpy, the following error may occur:
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so
Tha solution is to do:
export LD_PRELOAD=/opt/intel/mkl/lib/intel64/libmkl_core.so:/opt/intel/mkl/lib/intel64/libmkl_sequential.so
as found here http://debugjournal.tumblr.com/post/98401758462/intel-mkl-dynamic-link-library-error.

7) For PySCF, since we are using old numpy, you need to comment out warning in __init__.py in the pyscf folder

#if LooseVersion(numpy.__version__) <= LooseVersion('1.8.0'):
# raise SystemError("You're using an old version of Numpy (%s). "
# "It is recommended to upgrad numpy to 1.8.0 or newer. \n"
# "You still can use all features of PySCF with the old numpy by removing this warning msg. "
# "Some modules (DFT, CC, MRPT) might be affected because of the bug in old numpy." %
# numpy.__version__)

== Setting up virtual environment for PySCF ==
In this order:

1) Load python, numpy, scipy

[2) Create virtual environment]

3) Enter virtual environment

4) Update pip

3) Install h5py, ase or other necessary modules

Python software on Archer

2017-09-05T13:41:03Z

Ap837:

Python software on Archer

2017-08-31T09:25:34Z

Ap837:

Python software on Archer

2017-08-30T15:43:39Z

Ap837:

For running Python on work nodes on Archer, there are python-compute (native) and anaconda-compute modules available. Users are discouraged from using anaconda-compute because it is not optimised for running on Archer. There are some preinstalled packages for python-compute and they begin with pc-. For installation of additional packages, virtual environments are encouraged.

For more info see: http://www.archer.ac.uk/documentation/user-guide/python.php

When compiling Python software (like PySCF) Archer by default builds all libraries as static libraries. This leads to errors like:

ImportError Cannot import name ...

or
File "/work/y07/y07/cse/numpy/1.9.2-libsci/lib/python2.7/site-packages/numpy/ctypeslib.py", line 128, in load_library
raise OSError("no file with expected extension")
OSError: no file with expected extension

and other errors.

To prevent dynamic libraries from becoming static you must:

export CRAYPE_LINK_TYPE=dynamic

before the compilation. For more information see: http://www.archer.ac.uk/documentation/user-guide/development.php#sec-4.6

== In submission script itself, do not forget to: ==

1) Load python and all modules supplied centrally
module load python-compute
module load pc-numpy
module load pc-scipy
2) Enter virtual environment containing all additional modules (AFTER loading python and the central modules)
source /work/e507/e507/ap837/code/venv_pyscf/bin/activate
3) If code uses intel libraries: (errors like:
OSError: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
OSError: libiomp5.so: cannot open shared object file: No such file or directory
) do:
source /opt/intel/bin/compilervars.sh intel64

4) When using numpy, the following error may occur:
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so
Tha solution is to do:
export LD_PRELOAD=/opt/intel/mkl/lib/intel64/libmkl_core.so:/opt/intel/mkl/lib/intel64/libmkl_sequential.so
as found here http://debugjournal.tumblr.com/post/98401758462/intel-mkl-dynamic-link-library-error.

5) For PySCF, since we are using old numpy, you need to comment out warning in __init__.py in the pyscf folder

#if LooseVersion(numpy.__version__) <= LooseVersion('1.8.0'):
# raise SystemError("You're using an old version of Numpy (%s). "
# "It is recommended to upgrad numpy to 1.8.0 or newer. \n"
# "You still can use all features of PySCF with the old numpy by removing this warning msg. "
# "Some modules (DFT, CC, MRPT) might be affected because of the bug in old numpy." %
# numpy.__version__)

Python software on Archer

2017-08-30T15:43:07Z

Ap837:

For running Python on work nodes on Archer, there are python-compute (native) and anaconda-compute modules available. Users are discouraged from using anaconda-compute because it is not optimised for running on Archer. There are some preinstalled packages for python-compute and they begin with pc-. For installation of additional packages, virtual environments are encouraged.

For more info see: http://www.archer.ac.uk/documentation/user-guide/python.php

When compiling Python software (like PySCF) Archer by default builds all libraries as static libraries. This leads to errors like:

ImportError Cannot import name ...

or
File "/work/y07/y07/cse/numpy/1.9.2-libsci/lib/python2.7/site-packages/numpy/ctypeslib.py", line 128, in load_library
raise OSError("no file with expected extension")
OSError: no file with expected extension

and other errors.

To prevent dynamic libraries from becoming static you must:

export CRAYPE_LINK_TYPE=dynamic

before the compilation. For more information see: http://www.archer.ac.uk/documentation/user-guide/development.php#sec-4.6

In submission script itself, do not forget to:

1) Load python and all modules supplied centrally
module load python-compute
module load pc-numpy
module load pc-scipy
2) Enter virtual environment containing all additional modules (AFTER loading python and the central modules)
source /work/e507/e507/ap837/code/venv_pyscf/bin/activate
3) If code uses intel libraries: (errors like:
OSError: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
OSError: libiomp5.so: cannot open shared object file: No such file or directory
) do:
source /opt/intel/bin/compilervars.sh intel64

4) When using numpy, the following error may occur:
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so
Tha solution is to do:
export LD_PRELOAD=/opt/intel/mkl/lib/intel64/libmkl_core.so:/opt/intel/mkl/lib/intel64/libmkl_sequential.so
as found here http://debugjournal.tumblr.com/post/98401758462/intel-mkl-dynamic-link-library-error.

5) For PySCF, since we are using old numpy, you need to comment out warning in __init__.py in the pyscf folder

#if LooseVersion(numpy.__version__) <= LooseVersion('1.8.0'):
# raise SystemError("You're using an old version of Numpy (%s). "
# "It is recommended to upgrad numpy to 1.8.0 or newer. \n"
# "You still can use all features of PySCF with the old numpy by removing this warning msg. "
# "Some modules (DFT, CC, MRPT) might be affected because of the bug in old numpy." %
# numpy.__version__)

Python software on Archer

2017-08-30T15:42:21Z

Ap837:

For running Python on work nodes on Archer, there are python-compute (native) and anaconda-compute modules available. Users are discouraged from using anaconda-compute because it is not optimised for running on Archer. There are some preinstalled packages for python-compute and they begin with pc-. For installation of additional packages, virtual environments are encouraged.

For more info see: http://www.archer.ac.uk/documentation/user-guide/python.php

When compiling Python software (like PySCF) Archer by default builds all libraries as static libraries. This leads to errors like:

ImportError Cannot import name ...

or
File "/work/y07/y07/cse/numpy/1.9.2-libsci/lib/python2.7/site-packages/numpy/ctypeslib.py", line 128, in load_library
raise OSError("no file with expected extension")
OSError: no file with expected extension

and other errors.

To prevent dynamic libraries from becoming static you must:

export CRAYPE_LINK_TYPE=dynamic

before the compilation. For more information see: http://www.archer.ac.uk/documentation/user-guide/development.php#sec-4.6

In submission script itself, do not forget to:

1) Load python and all modules supplied centrally
module load python-compute
module load pc-numpy
module load pc-scipy
2) Enter virtual environment containing all additional modules (AFTER loading python and the central modules
source /work/e507/e507/ap837/code/venv_pyscf/bin/activate
3) If code uses intel libraries: (errors like:
OSError: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
OSError: libiomp5.so: cannot open shared object file: No such file or directory
) do:
source /opt/intel/bin/compilervars.sh intel64

4) When using numpy, the following error may occur:
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so
Tha solution is to do:
export LD_PRELOAD=/opt/intel/mkl/lib/intel64/libmkl_core.so:/opt/intel/mkl/lib/intel64/libmkl_sequential.so
as found here http://debugjournal.tumblr.com/post/98401758462/intel-mkl-dynamic-link-library-error.

5) For PySCF, since we are using old numpy, you need to comment out warning in __init__.py in the pyscf folder

#if LooseVersion(numpy.__version__) <= LooseVersion('1.8.0'):
# raise SystemError("You're using an old version of Numpy (%s). "
# "It is recommended to upgrad numpy to 1.8.0 or newer. \n"
# "You still can use all features of PySCF with the old numpy by removing this warning msg. "
# "Some modules (DFT, CC, MRPT) might be affected because of the bug in old numpy." %
# numpy.__version__)

Python software on Archer

2017-08-29T14:04:50Z

Ap837:

For running Python on work nodes on Archer, there are python-compute (native) and anaconda-compute modules available. Users are discouraged from using anaconda-compute because it is not optimised for running on Archer. There are some preinstalled packages for python-compute and they begin with pc-. For installation of additional packages, virtual environments are encouraged.

For more info see: http://www.archer.ac.uk/documentation/user-guide/python.php

When compiling Python software (like PySCF) Archer by default builds all libraries as static libraries. This leads to errors like:

ImportError Cannot import name ...

or

File "crystal.py", line 3, in <module>
import pyscf.pbc
File "/work/e507/e507/ap837/code/pyscf/__init__.py", line 118, in <module>
from pyscf import gto
File "/work/e507/e507/ap837/code/pyscf/pyscf/__init__.py", line 118, in <module>
from pyscf import gto
File "/work/e507/e507/ap837/code/pyscf/gto/__init__.py", line 7, in <module>
from pyscf.gto import mole
File "/work/e507/e507/ap837/code/pyscf/gto/mole.py", line 16, in <module>
from pyscf import lib
File "/work/e507/e507/ap837/code/pyscf/lib/__init__.py", line 11, in <module>
from pyscf.lib import numpy_helper
File "/work/e507/e507/ap837/code/pyscf/lib/numpy_helper.py", line 17, in <module>
_np_helper = misc.load_library('libnp_helper')
File "/work/e507/e507/ap837/code/pyscf/lib/misc.py", line 41, in load_library
return numpy.ctypeslib.load_library(libname, _loaderpath)
File "/work/y07/y07/cse/numpy/1.9.2-libsci/lib/python2.7/site-packages/numpy/ctypeslib.py", line 128, in load_library
raise OSError("no file with expected extension")
OSError: no file with expected extension

and other errors.

To prevent dynamic libraries from becoming static you must:

export CRAYPE_LINK_TYPE=dynamic

For more information see: http://www.archer.ac.uk/documentation/user-guide/development.php#sec-4.6

When using numpy, the following error may occur:
Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so
Tha solution is to do:
export LD_PRELOAD=/opt/intel/mkl/lib/intel64/libmkl_core.so:/opt/intel/mkl/lib/intel64/libmkl_sequential.so
as found here http://debugjournal.tumblr.com/post/98401758462/intel-mkl-dynamic-link-library-error.

Python software on Archer

2017-08-29T13:20:14Z

Ap837:

Python software on Archer

2017-08-29T13:19:43Z

Ap837:

Python software on Archer

2017-08-29T12:58:17Z

Ap837: Created page with "For running Python on work nodes on Archer, there are python-compute (native) and anaconda-compute modules available. Users are discouraged from using anaconda-compute because..."

Main Page

2017-08-29T12:44:19Z

Ap837: /* Useful Information */

Good Evening, Guten Abend, and Willkommen to ze new Thom Group Wiki
= Contents =
== [[Calendar | Calendar]] ==
https://calendar.google.com/calendar/render?mode=day&date=20160601T153539#main_7%7Cday-1+23745+23745+23745

{{Special:IframePage}}

== Group Meetings ==

===Research Period===
{|
|+
|'''Date''' || '''Mini-talk'''
|-
|15th August 2016 || Jonathan
|-
|30th August 2016 || Charlie
|-
|12th September 2016 || James
|-
|26th September 2016 || CANCELLED
|}

===Michaelmas Term===
{|
|'''Date''' || '''Mini-talk''' ||| '''Cake'''
|-
|10th October 2016 || Charlie || Hugh
|-
|24th October 2016 || Verena || Sav
|-
|7th November 2016 || Choi || Alex
|-
|21st November 2016 || Ruth || Bang & Verena
|-
|5th December 2016 || [AJWT away]
|-
|19th December 2016 || Hugh || Choi
|}

===Lent Term 2017===
{|
|'''Date''' || '''Mini-talk''' ||| '''Cake'''
|-
|9th January || Sav || Charlie & Verena
|-
|23rd January || Henry & Alex C || Ruth
|-
|13th February || Choi & Ben || Alex T
|-
|27th February || Bang || Alex C
|-
|13th March || Verena || Ben
|}

===Easter Term 2017===
{|
|'''Date''' || '''Mini-talk''' ||| '''Cake'''
|-
|24th April || Sav || Alex T
|-
|8th May || ||
|-
|22nd May || [AJWT away] || [Perhaps try the beer festival in the evening?]
|-
|5th June || ||
|}

== [[Group_List | Group List]] ==
(see also http://www.ch.cam.ac.uk/group/thom)
- An up to date list of group members

== [[Computer_List | Computer List]] ==
- Group computers available.
== [[Introduction | Introduction to basic shell commands]] ==
- There are a number of tutorials available which document basic operations that are useful including:

# General bash commands
# Setting up cygwin
# Setting up ssh keys and general bash commands what are helpful
# using [[GIT]]
# Useful cerebro queue commands
# Some [[vim]] tidbits

Currently undergoing construction.

== Useful Software ==
# Using QChem
# Using [[QCMagic]]
# [[SimpleDMC]]

== Useful Information ==
# Guidelines on [[Code Review]]
# How to do things relating to [[HANDE]] releases
# How to run PySCF or other [[Python software on Archer]]

== [[Archiving_data | Archiving data for the university repository]] ==
== [[Website_to_do | To-do list for the Thom Group Website]] ==

= To Do List =

-a pretty picture

--A "How to:" page on setting up cygwin, ssh keys and general bash commands what are helpful

--A "How to:" page on using qchem

--A Pretty picture for the $wgLogo

Main Page

2017-08-29T12:43:08Z

Ap837: /* Useful Information */

Good Evening, Guten Abend, and Willkommen to ze new Thom Group Wiki
= Contents =
== [[Calendar | Calendar]] ==
https://calendar.google.com/calendar/render?mode=day&date=20160601T153539#main_7%7Cday-1+23745+23745+23745

{{Special:IframePage}}

== Group Meetings ==

===Research Period===
{|
|+
|'''Date''' || '''Mini-talk'''
|-
|15th August 2016 || Jonathan
|-
|30th August 2016 || Charlie
|-
|12th September 2016 || James
|-
|26th September 2016 || CANCELLED
|}

===Michaelmas Term===
{|
|'''Date''' || '''Mini-talk''' ||| '''Cake'''
|-
|10th October 2016 || Charlie || Hugh
|-
|24th October 2016 || Verena || Sav
|-
|7th November 2016 || Choi || Alex
|-
|21st November 2016 || Ruth || Bang & Verena
|-
|5th December 2016 || [AJWT away]
|-
|19th December 2016 || Hugh || Choi
|}

===Lent Term 2017===
{|
|'''Date''' || '''Mini-talk''' ||| '''Cake'''
|-
|9th January || Sav || Charlie & Verena
|-
|23rd January || Henry & Alex C || Ruth
|-
|13th February || Choi & Ben || Alex T
|-
|27th February || Bang || Alex C
|-
|13th March || Verena || Ben
|}

===Easter Term 2017===
{|
|'''Date''' || '''Mini-talk''' ||| '''Cake'''
|-
|24th April || Sav || Alex T
|-
|8th May || ||
|-
|22nd May || [AJWT away] || [Perhaps try the beer festival in the evening?]
|-
|5th June || ||
|}

== [[Group_List | Group List]] ==
(see also http://www.ch.cam.ac.uk/group/thom)
- An up to date list of group members

== [[Computer_List | Computer List]] ==
- Group computers available.
== [[Introduction | Introduction to basic shell commands]] ==
- There are a number of tutorials available which document basic operations that are useful including:

# General bash commands
# Setting up cygwin
# Setting up ssh keys and general bash commands what are helpful
# using [[GIT]]
# Useful cerebro queue commands
# Some [[vim]] tidbits

Currently undergoing construction.

== Useful Software ==
# Using QChem
# Using [[QCMagic]]
# [[SimpleDMC]]

== Useful Information ==
# Guidelines on [[Code Review]]
# How to do things relating to [[HANDE]] releases
# How to run PySCF or other Python software on Archer

== [[Archiving_data | Archiving data for the university repository]] ==
== [[Website_to_do | To-do list for the Thom Group Website]] ==

= To Do List =

-a pretty picture

--A "How to:" page on setting up cygwin, ssh keys and general bash commands what are helpful

--A "How to:" page on using qchem

--A Pretty picture for the $wgLogo