James Spencer

James is a third year PhD student working with Ali Alavi. He's probably hacking something...

Python documentation locally:

pydoc -p 1774 &

and then go to http://localhost:1774. Google gets less traffic these days...

Without source code management life would be rather unpleasant: SVN Page.

I use ipython all the time: ipython homepage.

Code testing

I have written a package (two actually: this is the second generation) that runs a set of test calculations and compares it to previous results. More details: Other_useful_scipts#testcode.

Every night the Alavi group code is downloaded from the trunk of the repository, compiled and tested, with any errors sent out via email. This enables us to keep on top of any bugs that have been accidentally committed (accidental as all code is tested before it's committed to the trunk, of course!).

Cleaning up scratch

My submit script doesn't tidy up after itself after my job has finished, as I have had problems in the past with files being corrupted during the copy back, which has led to a lot of wasted time. Instead I have a python script to do the tidying, which I run periodically. The following is for tardis, but it is easy to adapt to clusters which don't have a nodescratch (replace the rm -rv /nodescratch/%s/%s/*/ bit with rsh %s rm -rv /scratch/%s/*/). You might also have to change the name of the nodes appropriate to the cluster. You will have to change the crsid...

#!/usr/bin/python
'''Clean up jss43's nodescratch'''

import subprocess

crsid='jss43'

def running_on_nodes(user):
    '''Obtain the set of nodes currently running jobs of the user.'''
    running_nodes=set()
    p=subprocess.Popen('qstat -nru %s' % user,shell=True,stdout=subprocess.PIPE,close_fds=True)
    child_stdout=p.communicate()[0].splitlines
    for line in child_stdout():
        if 'node' in line:
            running_nodes.update([node.strip() for node in line.split('+')]) # I love sets.
    return running_nodes

def cleanup(user=crsid,no_nodes=69):
    '''Delete all files on the scratch space of each node not currently being used by the user.'''
    avoid_nodes=running_on_nodes(user)
    for i in range(1,no_nodes+1):
        this_node='node%03d' % i
        if this_node not in avoid_nodes:
            cmd='rm -rv /nodescratch/%s/%s/*/' % (this_node,user)
            p=subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT,close_fds=True)
            (child_stdout_stderr)=p.communicate()
            print this_node
            print 'STDOUT:\n',child_stdout_stderr[0]
            print 'STDERR:\n' ,child_stdout_stderr[1],'\n'
    print 'You are running jobs on the following nodes and I couldn\'t clean up scratch on them:'
    for node in sorted([a_n for a_n in avoid_nodes]):
        print node


if __name__=='__main__':
    cleanup()

(Less trivial code contains more comments!)

Useful links

sed magic

bash scripting

much fun