Python software on Archer - Revision history

Van26 at 11:24, 15 August 2018

2018-08-15T11:24:04Z

← Older revision		Revision as of 11:24, 15 August 2018
Line 21:		Line 21:


	== In ~~submission~~ ~~script~~ ~~itself,~~ do ~~not~~ ~~forget~~ to: ==		== When installing pyscf and in the submission script: ==


	1) Load python and all modules supplied centrally		1) Load python and all modules supplied centrally (and load them in your submission script)
	module load python-compute		module load python-compute
	module load pc-numpy		module load pc-numpy
	module load pc-scipy		module load pc-scipy
			module load pc-ase

	2) Enter virtual environment containing all additional modules (AFTER loading python and the central modules)		2) Enter virtual environment containing all additional modules (AFTER loading python and the central modules)
	source /work/e507/e507/ap837/code/venv_pyscf/bin/activate		source /work/e507/e507/ap837/code/venv_pyscf/bin/activate
	[for AJWT this was in /fs2/e507/e507/...]		[for AJWT this was in /fs2/e507/e507/...]

			3) Now use pip install for the rest (only needed once, when setting up):
⚫	3) Use aprun to run the job, otherwise it will run only on the shared job launcher node and not on the compute nodes.
			export CRAYPE_LINK_TYPE=dynamic
		⚫	pip install pyscf
		⚫	pip install unittest2

	4) Create a tmp folder in the work directory and set the environmental variable to ~~pint~~ to it. The compute nodes do not have access to the regular tmp folders.		4) Create a tmp folder in the work directory and set the environmental variable to point to it in the submission script. The compute nodes do not have access to the regular tmp folders.
	export TMPDIR=/work/e507/e507/ap837/tmp		export TMPDIR=/work/e507/e507/ap837/tmp

		⚫	5) Use aprun to run the job, otherwise it will run only on the shared job launcher node and not on the compute nodes.
⚫	5) If code uses intel libraries: (errors like:

		⚫	6) If code uses intel libraries: (errors like:
	OSError: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory		OSError: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
	OSError: libiomp5.so: cannot open shared object file: No such file or directory		OSError: libiomp5.so: cannot open shared object file: No such file or directory
Line 43:		Line 50:
	source /opt/intel/bin/compilervars.sh intel64		source /opt/intel/bin/compilervars.sh intel64

		⚫	7) When using numpy, the following error may occur:
⚫	~~[5b~~ ~~AJWT needed to~~ pip install unittest2
⚫	6) When using numpy, the following error may occur:
	Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so		Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so
	Tha solution is to do:		Tha solution is to do:
	export LD_PRELOAD=/opt/intel/mkl/lib/intel64/libmkl_core.so:/opt/intel/mkl/lib/intel64/libmkl_sequential.so		export LD_PRELOAD=/opt/intel/mkl/lib/intel64/libmkl_core.so:/opt/intel/mkl/lib/intel64/libmkl_sequential.so
	as found here http://debugjournal.tumblr.com/post/98401758462/intel-mkl-dynamic-link-library-error.		as found here http://debugjournal.tumblr.com/post/98401758462/intel-mkl-dynamic-link-library-error.

	7) For PySCF, since we are using old numpy, you need to comment out warning in __init__.py in the pyscf folder

	#if LooseVersion(numpy.__version__) <= LooseVersion('1.8.0'):
	# raise SystemError("You're using an old version of Numpy (%s). "
	# "It is recommended to upgrad numpy to 1.8.0 or newer. \n"
	# "You still can use all features of PySCF with the old numpy by removing this warning msg. "
	# "Some modules (DFT, CC, MRPT) might be affected because of the bug in old numpy." %
	# numpy.__version__)


	== Setting up virtual environment for PySCF ==		== Setting up virtual environment for PySCF ==
Line 79:		Line 75:
	source /work/e507/e507/$USER/pyscfEnv/bin/activate		source /work/e507/e507/$USER/pyscfEnv/bin/activate
	export CRAYPE_LINK_TYPE=dynamic		export CRAYPE_LINK_TYPE=dynamic

	4) Update pip

⚫	pip install pyscf

	3) Install h5py, ase or other necessary modules

Ajwt3 at 13:25, 9 August 2018

2018-08-09T13:25:30Z

← Older revision		Revision as of 13:25, 9 August 2018
Line 30:		Line 30:
	2) Enter virtual environment containing all additional modules (AFTER loading python and the central modules)		2) Enter virtual environment containing all additional modules (AFTER loading python and the central modules)
	source /work/e507/e507/ap837/code/venv_pyscf/bin/activate		source /work/e507/e507/ap837/code/venv_pyscf/bin/activate
			[for AJWT this was in /fs2/e507/e507/...]

	3) Use aprun to run the job, otherwise it will run only on the shared job launcher node and not on the compute nodes.		3) Use aprun to run the job, otherwise it will run only on the shared job launcher node and not on the compute nodes.
Line 42:		Line 43:
	source /opt/intel/bin/compilervars.sh intel64		source /opt/intel/bin/compilervars.sh intel64

			[5b AJWT needed to pip install unittest2
	6) When using numpy, the following error may occur:		6) When using numpy, the following error may occur:
	Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so		Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so

Ajwt3: /* Setting up virtual environment for PySCF */

2018-08-09T09:36:30Z

Setting up virtual environment for PySCF

← Older revision		Revision as of 09:36, 9 August 2018
Line 67:		Line 67:
	[2) Create virtual environment]		[2) Create virtual environment]

			cd ~/work
	virtualenv --system-site-packages pyscfEnv		virtualenv --system-site-packages pyscfEnv

			This will print out a directory for the virtualenv installation which will hopefully be like the one below (with $USER changed)
			/work/e507/e507/$USER/pyscfEnv/bin/activate

	3) Enter virtual environment		3) Enter virtual environment

Ajwt3: /* Setting up virtual environment for PySCF */

2018-08-09T09:30:48Z

Setting up virtual environment for PySCF

← Older revision		Revision as of 09:30, 9 August 2018
Line 63:		Line 63:
	1) Load modules		1) Load modules

	module load python-compute ~~pc-numpy pc-scipy~~		module load python-compute

	[2) Create virtual environment]		[2) Create virtual environment]

Ajwt3 at 09:20, 9 August 2018

2018-08-09T09:20:25Z

← Older revision		Revision as of 09:20, 9 August 2018
Line 61:		Line 61:
	In this order:		In this order:

			1) Load modules
⚫	1) ~~Load~~ python-compute, pc-numpy, pc-scipy

		⚫	module load python-compute pc-numpy pc-scipy

	[2) Create virtual environment]		[2) Create virtual environment]

	virtualenv --system-site-packages pyscfEnv		virtualenv --system-site-packages pyscfEnv

	3) Enter virtual environment		3) Enter virtual environment

	source /work/e507/e507/$USER/pyscfEnv/bin/activate		source /work/e507/e507/$USER/pyscfEnv/bin/activate
	export CRAYPE_LINK_TYPE=dynamic		export CRAYPE_LINK_TYPE=dynamic

	4) Update pip		4) Update pip

	pip install pyscf		pip install pyscf

	3) Install h5py, ase or other necessary modules		3) Install h5py, ase or other necessary modules

Ajwt3: /* Setting up virtual environment for PySCF */

2018-08-09T09:19:38Z

Setting up virtual environment for PySCF

← Older revision		Revision as of 09:19, 9 August 2018
Line 64:		Line 64:

	[2) Create virtual environment]		[2) Create virtual environment]
			virtualenv --system-site-packages pyscfEnv

	3) Enter virtual environment		3) Enter virtual environment
			source /work/e507/e507/$USER/pyscfEnv/bin/activate
			export CRAYPE_LINK_TYPE=dynamic

	4) Update pip		4) Update pip
			pip install pyscf

	3) Install h5py, ase or other necessary modules		3) Install h5py, ase or other necessary modules

Ajwt3 at 09:14, 9 August 2018

2018-08-09T09:14:35Z

← Older revision		Revision as of 09:14, 9 August 2018
Line 61:		Line 61:
	In this order:		In this order:

	1) Load python, numpy, scipy		1) Load python-compute, pc-numpy, pc-scipy

	[2) Create virtual environment]		[2) Create virtual environment]

Ap837 at 13:41, 5 September 2017

2017-09-05T13:41:26Z

← Older revision		Revision as of 13:41, 5 September 2017
Line 69:		Line 69:
	4) Update pip		4) Update pip

	3) Install h5py, ase or other necessary		3) Install h5py, ase or other necessary modules

Ap837 at 13:41, 5 September 2017

2017-09-05T13:41:03Z

← Older revision		Revision as of 13:41, 5 September 2017
Line 56:		Line 56:
	# "Some modules (DFT, CC, MRPT) might be affected because of the bug in old numpy." %		# "Some modules (DFT, CC, MRPT) might be affected because of the bug in old numpy." %
	# numpy.__version__)		# numpy.__version__)


			== Setting up virtual environment for PySCF ==
			In this order:

			1) Load python, numpy, scipy

			[2) Create virtual environment]

			3) Enter virtual environment

			4) Update pip

			3) Install h5py, ase or other necessary

Ap837 at 09:25, 31 August 2017

2017-08-31T09:25:34Z

← Older revision		Revision as of 09:25, 31 August 2017
Line 30:		Line 30:
	2) Enter virtual environment containing all additional modules (AFTER loading python and the central modules)		2) Enter virtual environment containing all additional modules (AFTER loading python and the central modules)
	source /work/e507/e507/ap837/code/venv_pyscf/bin/activate		source /work/e507/e507/ap837/code/venv_pyscf/bin/activate

⚫	3) If code uses intel libraries: (errors like:
			3) Use aprun to run the job, otherwise it will run only on the shared job launcher node and not on the compute nodes.

			4) Create a tmp folder in the work directory and set the environmental variable to pint to it. The compute nodes do not have access to the regular tmp folders.
			export TMPDIR=/work/e507/e507/ap837/tmp

		⚫	5) If code uses intel libraries: (errors like:
	OSError: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory		OSError: libmkl_intel_lp64.so: cannot open shared object file: No such file or directory
	OSError: libiomp5.so: cannot open shared object file: No such file or directory		OSError: libiomp5.so: cannot open shared object file: No such file or directory
Line 36:		Line 42:
	source /opt/intel/bin/compilervars.sh intel64		source /opt/intel/bin/compilervars.sh intel64

	4) When using numpy, the following error may occur:		6) When using numpy, the following error may occur:
	Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so		Intel MKL FATAL ERROR: Cannot load libmkl_avx.so or libmkl_def.so
	Tha solution is to do:		Tha solution is to do:
Line 42:		Line 48:
	as found here http://debugjournal.tumblr.com/post/98401758462/intel-mkl-dynamic-link-library-error.		as found here http://debugjournal.tumblr.com/post/98401758462/intel-mkl-dynamic-link-library-error.

	5) For PySCF, since we are using old numpy, you need to comment out warning in __init__.py in the pyscf folder		7) For PySCF, since we are using old numpy, you need to comment out warning in __init__.py in the pyscf folder

	#if LooseVersion(numpy.__version__) <= LooseVersion('1.8.0'):		#if LooseVersion(numpy.__version__) <= LooseVersion('1.8.0'):