NECI-testcode

This is a new page to post info on how to migrate the test code between machines, as was done between discovery and scepter recently. I will be adding to this as and when I have spare time. --js615 12:52, 12 October 2009 (BST)

Setting up the testcode

1. Check out the components

This script written by JSS sets the correct permissions for the CLEAN directory. The 2775 sets the directory to have rwx permissions to the user and group and rx permissions for everyone else. The setgid bit is set so that anything created within the CLEAN directory inherits the group ownership of the CLEAN directory rather than the default group of the user (which is normally a group containing just that user). Please set umask 0002 in your ~/.bashrc before changing files in the CLEAN directory so that the group retains write permissions on files modified by you within the CLEAN directory.

 #!/bin/bash -x
 umask 0002
 SVN=https://wwmm.ch.cam.ac.uk/svn2/groups/alavi
 mkdir CLEAN
 chgrp alavi CLEAN
 chmod 2775 CLEAN
 cd CLEAN
 svn co $SVN/CPMD/branches/QMC/trunk CPMD
 git clone scepter:~ajwt3/NECI.git NECI
 svn co $SVN/testcode testcode

Currently, svn and git are supported by the testcode.

CPMD then needs to be told where NECI is:

 js615@scepter:~/CLEAN/CPMD> cat .compileconf 
 NECIsrc=/home/js615/CLEAN/NECI

The testcode also needs to be told the locations of and version control systems used for CPMD and NECI. Do this by copying userconfig-template and setting the obvious variables to your personal settings. The automated compilation and testing script, checktest.py, needs to know the location of the test suite. This is defined in main_testcode/tools/checktest_config file residing in the test suite (the testcode directory).

testcode options can also be specified within checktest_config on a compiler-level basis. One of the more useful options is to specify how many cores the parallel tests should use. Note that new benchmarks need to be produced if the number of cores changes as certain FCIMC options are given on a per-core basis and so changing the number of cores used actually changes the calculation. See the testcode suite documentation for more details of the possible options.

Add the pseudopotential library to your ~/.bashrc as well.

 export PP_LIBRARY_PATH=~/CLEAN/CPMD/PP_LIBRARY

2. Compiling the libraries

Certain compilers require certain libraries to be available when compiling CPMD and NECI. Which libraries are required can be investigated in the file checktest_config.

 js615@scepter:~/CLEAN/testcode/main_testcode/tools> cat checktest_config 

This contains blocks like this.

 [PC-PGI]
 modules = pgi/32/8.0/5 use.own fftw3.2.2/32/pgi
 job_types = neci
 job_categories = NECI

Although not all versions are used, it is convenient to compile lapack and fftw across a range of compilers so that they are accessible to the test code. JSS has written a set of scripts to do this. Extract scepter:~jss43/setup_libraries.tar.gz and run setup_libraries.sh. This installs libraries to ~/local/lib: you can adjust this location in the scripts if you want. It also sets up private environment modules for your own use.

It's important to note that the current version of fftw (3.2.2) doesn't compile with pathscale. (setup_libraries.tar.gz includes fftw 3.2 for this reason.)

See Lapack compilation for more information on compiling lapack.

You'll also need python libraries.

3. Setting up the libraries as modules

The command

 module load use.own

allows use of local libraries via environment modules in the folder ~/privatemodules. See Environment modules for more information. The test code requires each library to be set up in this manner.

The procedure to set up a module is as follows:

 js615@scepter:~/privatemodules> mkdir -p fftw3.2/64
 js615@scepter:~/privatemodules> vi fftw3.2/64/pathscale

And then use the following input:

 #%Module1.0#####################################################################
 ##
 ## 
 ##
 set root /home/js615/local/lib/fftw-3.2
 set version 3.2.2
 module-whatis "Loads version $version of the fftw library, compiled in 64-bit."
 prepend-path LIBRARY_PATH $root/lib
 prepend-path LD_LIBRARY_PATH $root/lib
 prepend-path MANPATH $root/manpages/man/

Change the root to reflect where you've actually got this version of fftw.

The following statement should be added to the end of modules compiled with pathscale:

 if { [ llength [ array get env PSC_GENFLAGS ] ] } {
       set mypscflags  "-L $root/lib $env(PSC_GENFLAGS)"
 } else {
       set mypscflags "-L $root/lib"
 }
 setenv PSC_GENFLAGS $mypscflags

(JSS: note that this causes serious issues if multiple pathscale libraries are being used. Avoid.)

Now the same module files need to be made for lapack. These require a slightly different header:

 js615@scepter:~/privatemodules/lapack3.1.1/32> cat ifort
 #%Module1.0#####################################################################
 ##
 ## 
 ##
 set root /home/js615/local/lib/lapack-3.1.1/32/ifort
 prepend-path LIBRARY_PATH $root/lib
 prepend-path LIBRARY_PATH $root/include
 prepend-path LD_LIBRARY_PATH $root/lib
 prepend-path LD_LIBRARY_PATH $root/include
 prepend-path MANPATH $root/manpages/man/

Note that in the current set-up there is slightly more information in these files, including (proc ModulesHelp and module-whatis). These have been ommitted here for clarity. People who are curious can go to the directory above and 'cat' the current file.

There is a more elegant solution to creating multiple modules for different compilers, using .base files (as done in setup_libraries---see http://www-theor.ch.cam.ac.uk/IT/software/modules.html). Furthermore, there is a better way to set PSC_GENFLAGS for pathscale. The build above is sufficient for the test code. To see these designs, go to:

 scepter:/home/jss43/privatemodules

and also look at Environment modules.

4. Link the libraries to the test code.

The test code has a file called checktest_config which tells it which modules to load for each compiler. This is currently at:

 js615@scepter:~/CLEAN/testcode/main_testcode/tools/checktest_config 

These need to match the directory structure of your privatemodules set-up. To test this, take a block of checktest_config:

 [PC-PGI]
 modules = pgi/32/8.0/5 use.own fftw3.2.2/32/pgi
 job_types = neci
 job_categories = NECI

and try to load everything on the "modules =" line (a "module purge" might be needed to avoid conflicts). E.g.

 js615@scepter:~/CLEAN/testcode/main_testcode/tools> module load pgi/32/8.0/5 use.own fftw3.2.2/32/pgi

The test code should now work.

5. Setting up cron (for nightly testing)

Use the command:

 crontab -e

from anywhere to set up cron. The following setup is used at the moment for this file:

 PP_LIBRARY_PATH=/home/js615/CLEAN/CPMD/PP_LIBRARY
 30   01  * * *  umask 0002 && /home/js615/CLEAN/testcode/main_testcode/tools/checktest.py 

Re-stating the pseudopotential library path and umask 0002 are required since cron doesn't source your ~/.bashrc. The following line runs checktest.py with the default settings at 1.30am.

6. Setting permissions

If the CLEAN directory isn't made with the statements in 1:

 mkdir CLEAN
 chgrp alavi CLEAN
 chmod 2775 CLEAN

Then it will not have the correct permissions. The best way to fix this is to start again...

Though JSS has suggested:

 #!/bin/bash
 set_perms_on_dir=~/test
 chgrp -R alavi $set_perms_on_dir
 for file in $(find $set_perms_on_dir); do
   # Get user's permissions
   perm=$(getfacl $file 2> /dev/null | grep user | sed -e 's/user:://')
   chmod g=$perm $file
   if [ -d $file ]; then
       chmod g+s $file
   fi
 done

Other Users on scepter

To use the test code on scepter, simply put umask 2 in your .bashrc, and then you can use testcode.py. If you want to use checktest.py, this will require re-compilation of the components, so will require you to have links to the libraries. You can either compile your own, or use mine. To use mine, simply copy my privatemodules folder (in /home/js615/) to your home directory. Then you should be able to use checktest.py.

Setting up other testcodes

I've just set another testcode up with vasp (with group permissions) at:

 scepter:/home/js615/CLEANVASP

The steps taken were:

1. Made a clean directory and checked out code as above

2. Added this block to the userconfig file:

 [vasp]
 exe = /home/js615/CLEANVASP/VASP/vasp
 cleanexe = /home/js615/CLEANVASP/VASP/vasp
 timing_output = cpu elapsed
 inp = INCAR
 repository_location = https://wwmm.ch.cam.ac.uk/svn2/groups/alavi/VASP/vasp.5/trunk
 compile = ./croncompile PLATFORM
 vcs = svn
 submit_template = /home/js615/CLEAN/testcode/main_testcode/tools/neci-submit.sh
 extract = /home/js615/CLEANVASP/testcode/main_testcode/tools/extractvasp.pl

(not all of these features I'm using, but this is basically just a duplicate of the NECI block slightly altered).

3. Then defined new jobs in jobconfig:

 [VASP_test]
 type = vasp

 # Form job categories.
 [categories]
 VASP = VASP_test

4. Made a croncompile script, which is very simple since I'm only wanting to test VASP on ifort/mpi (so no dealing with PLATFORM variables, and no loading of modules)

 #!/bin/bash
 cp ump2_aG.F ump2.F
 cd ../NECI/
 make neci-vasp
 stat=$?
 if [ $stat -ne 0 ] ; then
       exit $stat
 fi
 cd ../VASP/
 make vasp

5. Made a very simple data extraction script just for vasp DFT/HF output (more to be added)

 js615@scepter:~/CLEANVASP/testcode/main_testcode/tools> vi extractvasp.pl 

 #!/usr/bin/perl -w
 $out=$ARGV[0];
 `cat OUTCAR >> $out`;
 $energy=`grep TOTEN $out | head -1 | tr -d "[:alpha:][:blank:]="`;
 print "energy\n";
 print "$energy";

6. Made my test (VASP_test) and benchmarked it:

 js615@scepter:~/CLEANVASP/testcode> ./testcode.py --runbenchmark

7. Added one platform block to the checktest_config file:

 [PC-ifort64-MPI]
 modules = ifort/64/11.0/074 mpi/openmpi/64/intel11/1.3.1 
 options = --cores=1

And modified the emailing.

8. Ran checktest.py.