SVN setup
The group code is now managed under Subversion (SVN) version control. What follows is a quick setup guide to get you up and running with the Wales Group codes - GMIN, OPTIM and PATHSAMPLE. Once you are set up with the source code and are happy compiling it, you should read the best practice tips on the group version control page.
Getting an account on the SVN server
You should email or speak to David (dw34) or Rosie (rgm38) to request access to the Wales Group repository. Make sure you let us know your your CRSID, the bit before @cam.ac.uk in your email address.
Once they reply, you should be able to log in using your Active Directory (AD)/Admitto password. If you're not sure what your AD password is, or have lost it - you can get one here To check, try logging in here. If you can't, you've done something wrong! Get help!
Setting up your SVN details
Open a shell on the machine where you plan on using the group code. To make subversion easy to use, you need to specify some locations in your .bashrc file as environment variables. We do this as follows:
vi ~/.bashrc
This opens the vi editor. Press SHIFT+G to go to the bottom of the file, then press I to enter insert mode. Add the following to the bottom:
export SVN=https://svn.ch.cam.ac.uk/svn/wales/groups/wales export PAPERSSVN=https://svn.ch.cam.ac.uk/svn/wales/groups/djwpapers export MYSVN=https://svn.ch.cam.ac.uk/svn/CRSID
where CRSID is the user name you use to log into Raven with. The PAPERSSVN repository contains papers in preparation, while the SVN repository contains the group codes. When you're done, we need to save he changes, press ESC to exit insert mode (check 'Insert' isn't still showing at the bottom!) and then type:
:wq
and press ENTER. This should exit the editor, and return you to the directory you were in before. These variables will now be automatically set when you open a new shell, but for now, we need to load them manually. We do this using the source command:
source ~/.bashrc
We're now ready to check if you have access to the repository from your chosen machine. Lets try listing the contents of the trunk directory. Type the following ($SVN is the environmental variable we set to the location of the Wales Group repository above):
svn ls $SVN/trunk
You may be prompted to validate a security certificate so press p and then ENTER to accept it permanently. Now enter the password you set above (you should only need to do this once!). You should see a list of the directories in the repository, something like this:
AMBER/ AMBERTOOLS/ BLAS/ CAMSHIFTDATA/ CHARMM31/ CHARMM35/ CMakeModules/ COPYING DISCONNECT/ DOC/ GMIN/ INCLUDE/ LAPACK/ NAB/ OPTIM/ PATHSAMPLE/ PLUGINS/ SCRIPTS/ THESES/ UTILS/
NOTE: The 'trunk' contains the non-developmental version of the group codes.
Finally, before you download the code, you need to set up the log file template. This ensures that everyone submitting changes to the repository writes their log files in a similar fashion. Firstly, we need to edit the SVN config file
vi ~/.subversion/config
There is a section in the config file that begins as follows:
# [helpers] # editor-cmd = editor (vi, emacs, notepad, etc.)
This should be changed to the following:
[helpers] editor-cmd = vim +"r ~/template"
This tell subversion to open the file /home/CRSID/template when it asks for a commit log message to be written.
Interlude: while you're editing the config file, there are some other useful changes you may want to make. To have svn ignore various types of unversioned file (e.g. the products of a compilation) in the report from 'svn status', find the section labelled
[miscellany]
(it may be commented out with a # currently). Make sure it is uncommented (remove the #) and has no preceeding spaces on the line. Then, add a line underneath it like this:
global-ignores = *.o *.mod lib*.a
You can use wildcards and give all the suffixes you want to ignore, or even give explicit filenames if you want, separated by spaces as above.
Let's create the commit template file now
vi ~/template
Enter insert mode by pressing i and paste the following into vi:
<one line summary> PROGRAM ------- SUMMARY OF CHANGES ------------------ SUBROUTINES AFFECTED -------------------- KEYWORDS AFFECTED ----------------- KNOWN ISSUES ------------
Save the file by pressing ESC and then typing :wq. When committing, the first line of the template should be replaced with a one line summary of your changes. You're now good to download the group code :)
'Checking out' the code
The first time you want to access the code, you need to 'check it out'. This will pull the code from the repository into a local directory for your use.
WARNING: It has been found that if you put the svn directory too far from your /home/CRSID/, you will not be able to compile the CHARMM source. This is most likely due to the variable CHARMM uses to contain the locations of files is of a limited length! We hope to fix this problem, but for now you need to ensure it is close enough.
This is the furthest the CHARMM31 directory within the repository can be from your home, without causing errors. Adding one more 'x' to the svn directory name will cause problems.
/home/csw34/svnxxxxxxxxxxx/CHARMM31
As a result, it is HIGHLY recommended that you transfer the code to ~/svn as shown below. This also prevents you from having to edit Makefiles more than is absolutely necessary. The code is checked out as follows:
svn co $SVN/trunk ~/svn
Notice that I'm transferring the code to:
/home/csw34/svn
This path is clearly shorter than that above, and so we should have no problem compiling CHARMM if we choose to do so. You'll need to wait a while for the code to transfer! Once it has finished, you'll see a message like this:
Checked out revision 9853.
The number given is the version number you have just obtained. Whenever changes are make to the code and uploaded to the repository, this number is incremented. It might be increased by more than one between versions as the counter is common for all software on the SVN server, not just ours! That's it! You now have a copy of the group codes. Here are a few notes on its structure for new users.
- GMIN/source contains the GMIN source code itself.
- GMIN/bin contains the GMIN binaries when they are compiled.
- PATHSAMPLE contains the PATHSAMPLE code. It has the same directory structure as GMIN.
- OPTIM contains the OPTIM code. It has the same directory structure as GMIN also.
- AMBER contains the AMBER9 source code.
- NAB contains the AMBER9 nucleic acid builder (NAB) source code.
- LAPACK and BLAS contain the maths libraries used by all the group code.
- CHARMM31 contains the CHARMM source code. You need to compile CHARMM before you compile GMIN with it, and you must use the same compiler in both cases. If you don't you'll get errors. The OPTIM compilation automatically recompiles CHARMM if necessary.
- DOC contains the documentation for GMIN, OPTIM and PATHSAMPLE. Whenever you introduce a new keyword or feature, it should be documented.
- SCRIPTS contains useful scripts developed by members of the group. For example, perm-pdb.py generates an OPTIM perm.allow file from an input PDB.
- AMBERTOOLS contains the full version of Amber Tools 1.2 with our updated LeaP files to fix forcefield problems we have encountered using AMBER9.
Note: If you want to use OPTIM with the new C++ implementation of the NEB routine, you will need to obtain the source code for that separately. See here for instructions.
Note: If you want to use AMBER you will have to copy the libraries for the N-terminal (everything with nt) from AMBERTOOLS to $AMBERHOME/dat/leap/lib. Otherwise, you will potentially use unsymmetrised force field parameters.
Now that you have the source codes, you should try to compile and test it. You can find a tutorial on using cmake to compile GMIN here. If you'd rather not use cmake, or are having problems, a brief tutorial on compiling GMIN with CHARMM just using make can be found here.
Useful svn commands and good practice
Info on the commands you'll need to use to add your changes to the code, and retrieve the updates from the repository can be found on the Theory Sector SVN Page.
WARNING: svn commit is very powerful and it can be easy to misuse it, therefore it is strongly encouraged to ask for advice from more experienced users when using for the first time or doing something different.
GOOD PRACTICE: keep your code up to date with the trunk by regularly updating and committing changes as this will make your life easier and breakages and errors will be spotted more quickly.
- revert a file in the repository to revision xx
svn merge -c -xx <fname> svn commit <fname>
- revert the entire repository to revision xx with yy being the last revision to be removed
svn merge -r yy:xx $SVN/trunk svn commit
So for example to revert to revision 33122 and removing all revision up to 33134:
svn merge -r 33134:33122 $SVN/trunk svn commit
The 'release' branch
When you follow the instructions above, you are checking out the group development branch. This version of the code is what we are all working on together. As a result, it is possible that at any time, it might be somehow broken, we're only human after all!
If you are not going to be altering the code yourself and would like a stable (if possibly slightly outdated) version of the codes that should at least always compile, you can check out the current release branch code. We use this to produce the code tarballs on the website for collaborators without access to the repository. To do this, you need to follow the instructions above, but when it comes to checking out the code, replace this command:
svn co $SVN/trunk ~/svn
with this:
svn co $SVN/release ~/svn
You can then compile the codes as normal. You should NEVER commit anything to the release branch. The branch is updated by merging changes in from the main trunk at regular intervals once the code has been checked. If you want to modify the code yourself, check out the 'trunk' as normal above. If you do accidentally commit a change - please revert it. If you are unsure how to do this, tell Chris or Jo ASAP!
Another alternative is to create your own personal development branch as detailed below. This can be useful if you'd like to be able to commit intermediate bits of your new code without having to worry about breaking things for everyone else. Please make sure that you merge in your changes when you are sure everything is working :)
Personal development branches
At the same level as $SVN/trunk/ in the repository is a directory $SVN/branches/ . This is the place where the development branches live. A dev branch is a copy of a particular revision of trunk/ (or the subdirectory in trunk/ for one of the programs if the work is going to be that localized...). They are made with the 'svn copy ...' command, and mean that potentially disruptive work can be under version control whilst leaving the trunk stable and well-behaved. The basic work cycle is:
- make the branch with the svn copy .... command
- check out a working copy of the new branch
- make changes in your working copy and commit them, in the usual way
- keep an eye on what's being committed to trunk during this time. Pull the changes on trunk between when you make the branch and the current head into your branch by merging. This will make sure you have the latest bug fixes etc. You can merge as many times as you need, bringing in the changes on trunk between different pairs of revision numbers as time passes. Just make sure you don't try to merge the same changeset into your branch twice.
- make more commits on your branch. After testing, you find your shiny new code is ready to go into the trunk for distribution etc.
- do one more merge of the latest trunk changesets (since the previous such merge) into your branch
- merge the changes on your branch since its creation into trunk
- do something else now... :-)
In the above list, for trunk read "trunk or the subdirectory in trunk/ for one of the programs", i.e. whichever directory you specified in the original svn copy to make the branch.
See here for more info on the svn commands for branching and merging.
RSS feeds
As an added feature, we have set up RSS feeds for the SVN logs. This means that when you commit a change, the source you submitted is automatically diff'd against the previous version, and differences uploaded to the feed, along with the corresponding log message. More info on the feed locations can be found on the group version control page.
Using git-svn
Git can interface nicely with svn.
Why use git?
Git is a more modern version control system than svn which has lots of very nice features that svn doesn't.
- You have an entire copy of the repository locally so you can use git log, git diff, and checkout and older version of the software without needing to contact the svn server with every command.
- You can make lots of commits locally before contacting the server. You only need to contact the svn server when you want to get changes from the server or push your changes to the server. This is nice because you can work on a section of the code for days, weeks or months and have all your work under local version control, but only push your changes to the server when you think it's ready for prime time.
- Making new local branches is completely trivial so you can easily work on two logically separate things (e.g. experimenting with a new potential in GMIN and adding a feature to disconnectDPS) at the same time within the same repository.
Clone the svn repository
To clone the svn repository, just copy your old stuff in ~/svn to somewhere else for safekeeping. Then,
git svn clone https://svn.ch.cam.ac.uk/svn/wales/groups/wales/trunk/ svn/
will download the repository to a new directory svn/ Sometimes that stalls midway through the download, so just run
git svn fetch
until it works.
Here is a good git cheat-sheet written for svn users
Basic Working Cycle
The basic cycle assumes you're doing all your work on the master branch. The first step is to sync your local repository with the server
git svn rebase
For git experts: it is important to use rebase here instead of merge because svn can only deal with linear histories.
Next make some changes to the files and commit the changes
git commit -a -m "here is my commit message"
Ensure that the local repository is up to date and push the commit (or commits) back to the server.
git svn rebase git svn dcommit
Useful tools
The following git commands are very useful tools and pretty self explanatory
git log git diff git grep
Also, the gui gitk is very useful for visually inspecting the history
gitk --all
Advanced working cycle
A more advanced working cycle would take advantage of the powerful branching ability of git. First make a working branch.
git branch mybranch git checkout mybranch
Make changes and commit them to mybranch. Multiple commits are fine.
Sync your branch remotes/git-svn with the svn server (this is doing in two steps what `git svn rebase` would do in one)
git svn fetch
Bring your working branch up to date
git rebase remotes/git-svn
push your commits to the svn server
git svn dcommit
Note: It is important to use git rebase rather than git merge because it maintains a purely linear commit history, which is necessary for easy communication with the svn server. Before using dcommit, ensure that the changes you want to commit are arranged in a nicely linear manner on top of remotes/git-svn.
gitk --all
is a good tool for examining the branch structure of your repository and making sure everything is linear before a dcommit.
Note that
git svn rebase
is a shorcut for the two commands
git svn fetch git rebase remotes/git-svn
Squashing commits
It is good to make many small commits when working locally, but when pushing to the trunk it you might want to just make a few, very well-commented commits. You can squash multiple git commits into one single commit and change the commit message at the same time using rebase.
To squash your last four commits together, use
git rebase -i HEAD~4
For step-by-step instructions, look here.