Jenkins CI
Jenkins CI is an open source tool for continuous integration. We use it to ensure GMIN, OPTIM and PATHSAMPLE are always working. Each night Jenkins downloads the latest code from the svn repository and compiles each of the targets with each of the compilers. You can see the current state of the code by going to the following website https://wales-jenkins.ch.cam.ac.uk/ using your AD password.
Job details
If you go to the Jenkins page, you'll see that there are quite a few jobs. CheckoutJob is the job which downloads the most recent version of the code from the svn. All the rest of the jobs are spawned if CheckoutJob finishes without error. GMIN and OPTIM perform a "matrix" of jobs. Each target is compiled with all four compilers. You can see the details of each sub-job by clicking on the job name. If any target is known to not work with a given compiler (e.g. CHARMM only works with ifort and pgi), those jobs will be filtered out and not run.
CHARMM with OPTIM and GMIN are compiled separately because part of the CHARMM compilation fails if file names are too long. These jobs compile in a temporary directory to avoid long file names.
GMIN_DUPLICATE_TESTING and the corresponding one for OPTIM are special jobs which exist only to test for duplicate symbols in the object files. This generally means two subroutines for functions with the same name. Duplicates are not detected in the default build because the object files (.o files) built from the source files are first compiled into an archive, e.g. optimlib.a.
Seeing console output
You can see the console output of any job that was run on Jenkins. This is particularly useful for finding the reason why a job failed. Typically you would click on a job, then on a sub-job in the matrix, then on a particular build (left panel). At this point there should be a link for console output in the left panel.
This also contains information about the compilers used, so that you can see which compilers may have succeeded or failed in building the code.
e.g.
-- Check for working Fortran compiler: /usr/local/shared/ubuntu-12.04/x86_64/pgi/linux86-64/14.3/bin/pgf90 -- works
This line indicates that Jenkins is using PGI version 14.3 on Ubuntu 12.04.
Where is it built?
The web server for Jenkins is hosted at wales-jenkins.ch.private.cam.ac.uk . The web server doesn't have enough juice to run the actual jobs, so they are run on other computers via slave nodes. Currently all the jobs are run on phantom.
OPTIM Test Suite
In addition to the compilation tests, Jenkins also runs a set of test OPTIM jobs and compares the output with predefined "expected output" files to make sure nothing unexpected has changed. At the moment, the tests have only minimal coverage of OPTIM's huge range of keywords and functionalities. The tests are also quite primitive, so definitely won't catch every bug. Please feel free to contribute new tests and to improve the old ones!
The test suite is run every time the Jenkins OPTIM project completes successfully (i.e. all builds compile correctly). A separate pass/fail record is generated for each combination of test, system and compiler. This allows us to quickly see which bits of the code have broken.
In the svn root directory is a folder called TESTS/OPTIM, within which are separate folders for the different tests being run. These folders contain a subdirectory for each different system to be tested. Each of these subdirectories contain the input files and expected_output directory for that particular test. Jenkins takes a copy of this directory for each OPTIM executable and runs the programme in that location.
Each test directory also contains a script to read and compare the relevant output and expected_output files. At the time of writing these are all python scripts, but they don't need to be. Finally, there's a bash script called output_parser.sh which is what Jenkins actually runs and which invokes the python script to check the output. The exit code of this script determines whether the Jenkins test passes.
Note that a "failure" of one of these tests doesn't necessarily mean something is broken. It just means that the new output files are different from the expected output files - that could mean there's a bug, or it could mean that the amount and format of output printing has changed and broken backward-compatibility. It could even mean that one of the test routines has been improved so the output no longer matches! If you are absolutely certain that your changes have made the code better, you should replace the expected_output folder with your new output files and commit the changes.
The tests that I've set up give a limited report of their success/failure state to the Jenkins console output (see above), and email it to the Jenkins administrators when there's a failure. If you need more detail on the most recent test output, look at (for example): phantom:/scratch/jenkins-slave/workspace/OPTIM_Tests/Compiler/ifort/Target/SD/Test/PATH/test
Feel free use the test input in the svn directory to test your code before committing it - but please make sure you don't accidentally commit any output and auxiliary files created this way.
Future projects that you could help with:
- Adding more documentation to the svn TESTS folder
- Adding a bash script to run all tests quickly with a specified OPTIM executable
- Expanding the OPTIM tests to cover more of the programme's functionality and system types
- Adding a similar test suite for GMIN and/or PATHSAMPLE