Difference between revisions of "Other useful scripts"

From CUC3
Jump to navigation Jump to search
import>Jss43
import>Jss43
 
(8 intermediate revisions by 2 users not shown)
Line 1: Line 1:
−
A place to share non-research specific scripts that could be useful to others.
 +
A place to share non-research specific code that could be useful to others.
   
 
= scratchrestart.pl =
  
= scratchrestart.pl =
   
−
A common issue is running over quota on the clusters, due to each calculation generating a large file (e.g. CPMD RESTART files). [[Media:scratchrestart.txt | scratchrestart.pl]] attempts to solve this by searching recursively through the directory tree (from the working directory) and moving all files matching a regular expression to a scratch disk, leaving behind a symbolic link to the file's new location and leaving all other files untouched. Warning: you will be asked for confirmation *only* once.
 +
A common issue is running over quota on the clusters, due to each calculation generating a large file (e.g. CPMD RESTART files). [[Media:scratchrestart2.txt | scratchrestart.pl]] attempts to solve this by searching recursively through the directory tree (from the working directory) and moving all files matching a regular expression to a scratch disk, leaving behind a symbolic link to the file's new location and leaving all other files untouched. Warning: you will be asked for confirmation *only* once.
   
 
<pre>
  
<pre>
Line 21: Line 21:
 
= testcode =
  
= testcode =
   
−
[http://www-alavi.ch.cam.ac.uk/tools/testcode.tar testcode] Test that code at least reproduces previous results by running a set of test calculations and comparing to "benchmarks": useful to test for regressions, compilation issues and optimisations. Requires a script that analyses output of the calculations. Integrated with subversion (but this is not required). The Alavi group use this for automated nightly testing of their software and during development work.
+
Test that code at least reproduces previous results by running a set of test calculations and comparing to "benchmarks": useful to test for regressions, compilation issues and optimisations. The Alavi group use this for automated nightly testing of their software and during development work.
   
 
Features:
  
Features:
 
* Flexible job specification, so that a few jobs or job categories can be tested at a time, allowing for both quick tests when developing a small area of the code and a more comprehensive set of tests.
  
* Flexible job specification, so that a few jobs or job categories can be tested at a time, allowing for both quick tests when developing a small area of the code and a more comprehensive set of tests.
  +
* Multiple tests in the same job directory.
−
* Ability to test more than one codebase in one test suite. This is not particularly appropriate for very disparate codes: see the benchmarks section for further details.
 +
* Ability to test more than one codebase in one test suite. This is not particularly appropriate for very disparate codes.
 
* Easy to adapt: all the configuration is given in a simple syntax in config files.
  
* Easy to adapt: all the configuration is given in a simple syntax in config files.
 
* Can produce benchmarks for future comparisons.
  
* Can produce benchmarks for future comparisons.
  +
* Can run the test jobs either locally or by submission to a cluster.
−
* Can be used in conjunction with checktest.py (not released yet) and cron to regularly check an svn repository code.
 +
* Can be used in conjunction with checktest.py and cron to regularly checkout a code from a version control repository, compile and then test it.
   
−
See the README file for more information.
 +
See the documentation for more information.
  +
  +
Available for download from [http://www.cmth.ph.ic.ac.uk/people/j.spencer/code.php].
  +
  +
= Memory Manager =
  +
  +
Fortran module to log and check memory allocations. The debugging mode prints out all memory log activities. Outputs the peak memory usage and information about the largest arrays.
  +
  +
<pre>
  +
jss43@keiko:~/testmem> ./testmem.x
  +
  +
Welcome.
  +
  +
Memory Manager initialised with 1024.0 MB
  +
  +
This is a test.
  +
  +
================================================================
  +
Memory usage
  +
Maximum memory defined is (MB) : 1024.0
  +
Maximum memory used is (MB) : 381.5
  +
  +
Large memory allocations:
  +
  +
Name Allocated in Deallocated in Size
  +
----------------------------------------------------------------
  +
arr3 subroutine2 subroutine2 381.5MB
  +
arr2 subroutine1 subroutine2 156.3KB
  +
arr1 subroutine2 not deallocated 39.1KB
  +
arr1 subroutine1 subroutine1 39.1KB
  +
arr1 subroutine1 subroutine1 39.1KB
  +
================================================================
  +
  +
All done.
  +
Goodbye.
  +
</pre>
  +
  +
Available for download from [http://www.cmth.ph.ic.ac.uk/people/j.spencer/code.php].

Latest revision as of 13:38, 1 October 2009

A place to share non-research specific code that could be useful to others.

scratchrestart.pl

A common issue is running over quota on the clusters, due to each calculation generating a large file (e.g. CPMD RESTART files). scratchrestart.pl attempts to solve this by searching recursively through the directory tree (from the working directory) and moving all files matching a regular expression to a scratch disk, leaving behind a symbolic link to the file's new location and leaving all other files untouched. Warning: you will be asked for confirmation *only* once. 

# scratchrestart.pl --help
usage: scratchrestart.pl [regular expression]

Search all files matching the regular expression (default: RESTART) contained in the
the current directory and all subdirectories and move them to scratch, leaving behind a 
symbolic link to the file now in scratch.  The directory structure is preserved in scratch. 
e.g. The file /home/jss43/RUN/RESTART.1 is moved to /scratch/jss43/RUN/RESTART.1.
Files are only moved if they are not symbolic links, so it is safe to do multiple times 
(a run only moves the files that have been generated since the last run).

This is used for moving files to /storage on tardis: you will need to change the value of the $scratch variable to be the name of your scratch disk.

testcode

Test that code at least reproduces previous results by running a set of test calculations and comparing to "benchmarks": useful to test for regressions, compilation issues and optimisations. The Alavi group use this for automated nightly testing of their software and during development work.

Features:

  • Flexible job specification, so that a few jobs or job categories can be tested at a time, allowing for both quick tests when developing a small area of the code and a more comprehensive set of tests.
  • Multiple tests in the same job directory.
  • Ability to test more than one codebase in one test suite. This is not particularly appropriate for very disparate codes.
  • Easy to adapt: all the configuration is given in a simple syntax in config files.
  • Can produce benchmarks for future comparisons.
  • Can run the test jobs either locally or by submission to a cluster.
  • Can be used in conjunction with checktest.py and cron to regularly checkout a code from a version control repository, compile and then test it.

See the documentation for more information.

Available for download from [1].

Memory Manager

Fortran module to log and check memory allocations. The debugging mode prints out all memory log activities. Outputs the peak memory usage and information about the largest arrays. 

jss43@keiko:~/testmem> ./testmem.x

 Welcome.

 Memory Manager initialised with   1024.0 MB

 This is a test.

 ================================================================
 Memory usage
 Maximum memory defined is (MB) :    1024.0
 Maximum memory used is    (MB) :     381.5
 
 Large memory allocations:
 
 Name              Allocated in       Deallocated in         Size
 ----------------------------------------------------------------
 arr3              subroutine2        subroutine2         381.5MB
 arr2              subroutine1        subroutine2         156.3KB
 arr1              subroutine2        not deallocated      39.1KB
 arr1              subroutine1        subroutine1          39.1KB
 arr1              subroutine1        subroutine1          39.1KB
 ================================================================

 All done.
 Goodbye.

Available for download from [2].

Retrieved from "http://wikis.ch.cam.ac.uk/cuc3/wiki/index.php?title=Other_useful_scripts&oldid=2581"