Docswiki - User contributions [en]

Git Workflow

2025-07-30T10:15:49Z

Jwrm2: /* Checking out an older version of the master branch */ Formatted checkout command as code.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. If you get the error message

Error: failed to call git rev-parse --git-dir: exit status 128 : fatal: .git

then try

$ git lfs install --skip-repo

instead. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/softwarewales software]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Submodules==

Our software repository has become quite large as external potentials have been added. Submodules offer a way to compartmentalise the repository and speed up clones and updates. Self-contained third-party potentials are good candidates for splitting off into submodules. We will use GDML (Gradient-Descent Machine Learning) as an example. On an initial checkout, you will see a GDML/ directory in the root of the repository, but the directory will be empty. Most users do not need GDML, so will not care or even be particularly aware that GDML has not be cloned.

Let us suppose that you actually need GDML. You can tell git that it is required by running

$ git submodule init GDML

Then the next time you run

$ git submodule update

the contents of GDML will be checked out. If GDML is subsequently updated, run the update command again to get the newest version. If at some later point you decide that you have finished with GDML and no longer need it checked out, run

$ git submodule deinit GDML

and the GDML directory will be emptied. If the submodule you are interested in has its own submodules, add the --recursive flag to the commands. If you decide that you want all the submodules run

$ git submodule update --init --recursive

and all submodules will be checked out. This command is not recommended and you must have a good reason why you need all the submodules before you consider running it.

===Creating a new submodule===

If you are creating an interface to a new large external potential, it may be appropriate to add the external potential as a submodule. It is appropriate if the potential is quite large (more than a few MB), is being placed in the root of the repository, and is not likely to require much in the way of changes after the interface is set up. Instead of adding the external potential to the softwarewales repository, create a new repository under Wales Group on Gitlab and place the files there. Within softwarewales run

$ git submodule add git@gitlab.developers.cam.ac.uk:ch/wales/my_new_repository.git

where my_new_repository is replaced with whatever you named the new repository. You could also got the appropriate URL by going onto Gitlab for the new repository, clicking the 'Clone' button and copying. This command makes the necessary changes to the .gitmodules file, which will then need to be committed and pushed.

If the external potential already exists in some other repository and you will not need to make any changes to it at all, you can add the external repository directly rather than copying and using up space on our Gitlab. LAMMPS was added in this way, with a command like

$ git submodule add https://github.com/lammps/lammps.git

This approach is only likely to work for public repositories that do not need any kind of password to clone.

There are a couple of useful options to be aware of. Commands like the above will create a new directory with the name of the repository. You can use a custom name by adding it after the repository URL. The command will use the default branch of the repository (usually 'master' or 'main'), but you can specify a different branch with the -b flag. For example

$ git submodule add -b metatomic https://github.com/metatensor/lammps.git lammps_petmad

will clone the metatomic branch of the lammps repository from metatensor, but put it in the directory lammps_petmad.

Turning an existing subdirectory into a submodule is also possible, but is slightly more complicated and is considered an advanced topic. Google is your friend. Do not mess with the Gitlab repository until you are satisfied you have made the correct adjustments locally.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Checking out an older version of the master branch==

There are all sorts of useful git tools for checking how individual files and branches have changed.
See the documentation for git diff for example. To check out the master branch current at a given date you can use:

$ git checkout `git rev-list -1 --before="Feb 11 2024" master`

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Each paper is a separate repository. To start a new paper, go to Gitlab and create a new repository by clicking the blue 'New Project' button on your home screen. Create a blank project and make sure the project URL indicates it is under ch/wales rather than in your user space. Checkout the new repository and start writing the paper in the blank directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf documents either (except perhaps for proofs created by the journal). When it comes to revisions and resubmissions, do not create a new subdirectory for the new version. Git keeps the whole history so it is always possible to revert to a previous version.

You will also want to checkout the [https://gitlab.developers.cam.ac.uk/ch/wales/bib bibliography repository] but you only need one copy of this, not one for each paper. A suitable directory structure might be to have a papers/ directory under your home directory that contains a directory for the bibliography repository and a directory for each paper you are currently working on.

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You should assign the request to yourself and anyone else who has worked on this branch. Choose the person who is going to review for you. Make sure you tick the boxes to delete the feature branch after the request is accepted. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you select as reviewer will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. However, the reviewer will not be doing extensive testing, so it remains your responsibility to follow the coding standards and make sure everything works. You should make sure your changes compile with nagfor before submitting the merge request, as that is the most particular compiler. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

$ git fetch -p && for branch in $(git branch -vv | grep ': gone]' | awk '{print $1}'); do git branch -D $branch; done

Delete all local branches that do not exist on GitLab. This command is useful to periodically clean up local branches after they have been merged and deleted on GitLab. Warning: do not run this command if you've created a new local branch and not yet pushed it to GitLab.

Git Workflow

2025-07-30T10:15:15Z

Jwrm2: /* Creating a new submodule */ More details about adding a new submodule.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. If you get the error message

Error: failed to call git rev-parse --git-dir: exit status 128 : fatal: .git

then try

$ git lfs install --skip-repo

instead. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/softwarewales software]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Submodules==

Our software repository has become quite large as external potentials have been added. Submodules offer a way to compartmentalise the repository and speed up clones and updates. Self-contained third-party potentials are good candidates for splitting off into submodules. We will use GDML (Gradient-Descent Machine Learning) as an example. On an initial checkout, you will see a GDML/ directory in the root of the repository, but the directory will be empty. Most users do not need GDML, so will not care or even be particularly aware that GDML has not be cloned.

Let us suppose that you actually need GDML. You can tell git that it is required by running

$ git submodule init GDML

Then the next time you run

$ git submodule update

the contents of GDML will be checked out. If GDML is subsequently updated, run the update command again to get the newest version. If at some later point you decide that you have finished with GDML and no longer need it checked out, run

$ git submodule deinit GDML

and the GDML directory will be emptied. If the submodule you are interested in has its own submodules, add the --recursive flag to the commands. If you decide that you want all the submodules run

$ git submodule update --init --recursive

and all submodules will be checked out. This command is not recommended and you must have a good reason why you need all the submodules before you consider running it.

===Creating a new submodule===

If you are creating an interface to a new large external potential, it may be appropriate to add the external potential as a submodule. It is appropriate if the potential is quite large (more than a few MB), is being placed in the root of the repository, and is not likely to require much in the way of changes after the interface is set up. Instead of adding the external potential to the softwarewales repository, create a new repository under Wales Group on Gitlab and place the files there. Within softwarewales run

$ git submodule add git@gitlab.developers.cam.ac.uk:ch/wales/my_new_repository.git

where my_new_repository is replaced with whatever you named the new repository. You could also got the appropriate URL by going onto Gitlab for the new repository, clicking the 'Clone' button and copying. This command makes the necessary changes to the .gitmodules file, which will then need to be committed and pushed.

If the external potential already exists in some other repository and you will not need to make any changes to it at all, you can add the external repository directly rather than copying and using up space on our Gitlab. LAMMPS was added in this way, with a command like

$ git submodule add https://github.com/lammps/lammps.git

This approach is only likely to work for public repositories that do not need any kind of password to clone.

There are a couple of useful options to be aware of. Commands like the above will create a new directory with the name of the repository. You can use a custom name by adding it after the repository URL. The command will use the default branch of the repository (usually 'master' or 'main'), but you can specify a different branch with the -b flag. For example

$ git submodule add -b metatomic https://github.com/metatensor/lammps.git lammps_petmad

will clone the metatomic branch of the lammps repository from metatensor, but put it in the directory lammps_petmad.

Turning an existing subdirectory into a submodule is also possible, but is slightly more complicated and is considered an advanced topic. Google is your friend. Do not mess with the Gitlab repository until you are satisfied you have made the correct adjustments locally.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Checking out an older version of the master branch==

There are all sorts of useful git tools for checking how individual files and branches have changed.
See the documentation for git diff for example. To check out the master branch current at a given date you can use:

git checkout `git rev-list -1 --before="Feb 11 2024" master`

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Each paper is a separate repository. To start a new paper, go to Gitlab and create a new repository by clicking the blue 'New Project' button on your home screen. Create a blank project and make sure the project URL indicates it is under ch/wales rather than in your user space. Checkout the new repository and start writing the paper in the blank directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf documents either (except perhaps for proofs created by the journal). When it comes to revisions and resubmissions, do not create a new subdirectory for the new version. Git keeps the whole history so it is always possible to revert to a previous version.

You will also want to checkout the [https://gitlab.developers.cam.ac.uk/ch/wales/bib bibliography repository] but you only need one copy of this, not one for each paper. A suitable directory structure might be to have a papers/ directory under your home directory that contains a directory for the bibliography repository and a directory for each paper you are currently working on.

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You should assign the request to yourself and anyone else who has worked on this branch. Choose the person who is going to review for you. Make sure you tick the boxes to delete the feature branch after the request is accepted. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you select as reviewer will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. However, the reviewer will not be doing extensive testing, so it remains your responsibility to follow the coding standards and make sure everything works. You should make sure your changes compile with nagfor before submitting the merge request, as that is the most particular compiler. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

$ git fetch -p && for branch in $(git branch -vv | grep ': gone]' | awk '{print $1}'); do git branch -D $branch; done

Delete all local branches that do not exist on GitLab. This command is useful to periodically clean up local branches after they have been merged and deleted on GitLab. Warning: do not run this command if you've created a new local branch and not yet pushed it to GitLab.

Git Workflow

2023-08-08T09:33:06Z

Jwrm2: /* Writing a Paper */ Fixed bib repository link.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. If you get the error message

Error: failed to call git rev-parse --git-dir: exit status 128 : fatal: .git

then try

$ git lfs install --skip-repo

instead. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/softwarewales software]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Submodules==

Our software repository has become quite large as external potentials have been added. Submodules offer a way to compartmentalise the repository and speed up clones and updates. Self-contained third-party potentials are good candidates for splitting off into submodules. We will use GDML (Gradient-Descent Machine Learning) as an example. On an initial checkout, you will see a GDML/ directory in the root of the repository, but the directory will be empty. Most users do not need GDML, so will not care or even be particularly aware that GDML has not be cloned.

Let us suppose that you actually need GDML. You can tell git that it is required by running

$ git submodule init GDML

Then the next time you run

$ git submodule update

the contents of GDML will be checked out. If GDML is subsequently updated, run the update command again to get the newest version. If at some later point you decide that you have finished with GDML and no longer need it checked out, run

$ git submodule deinit GDML

and the GDML directory will be emptied. If the submodule you are interested in has its own submodules, add the --recursive flag to the commands. If you decide that you want all the submodules run

$ git submodule update --init --recursive

and all submodules will be checked out. This command is not recommended and you must have a good reason why you need all the submodules before you consider running it.

===Creating a new submodule===

If you are creating an interface to a new large external potential, it may be appropriate to add the external potential as a submodule. It is appropriate if the potential is quite large (more than a few MB), is being placed in the root of the repository, and is not likely to require much in the way of changes after the interface is set up. Instead of adding the external potential to the softwarewales repository, create a new repository under Wales Group on Gitlab and place the files there. Within softwarewales run

$ git submodule add git@gitlab.developers.cam.ac.uk:ch/wales/my_new_repository.git

where my_new_repository is replaced with whatever you named the new repository. You could also got the appropriate URL by going onto Gitlab for the new repository, clicking the 'Clone' button and copying. This command makes the necessary changes to the .gitmodules file, which will then need to be committed and pushed.

Turning an existing subdirectory into a submodule is also possible, but is slightly more complicated and is considered an advanced topic. Google is your friend. Do not mess with the Gitlab repository until you are satisfied you have made the correct adjustments locally.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Each paper is a separate repository. To start a new paper, go to Gitlab and create a new repository by clicking the blue 'New Project' button on your home screen. Create a blank project and make sure the project URL indicates it is under ch/wales rather than in your user space. Checkout the new repository and start writing the paper in the blank directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf documents either (except perhaps for proofs created by the journal). When it comes to revisions and resubmissions, do not create a new subdirectory for the new version. Git keeps the whole history so it is always possible to revert to a previous version.

You will also want to checkout the [https://gitlab.developers.cam.ac.uk/ch/wales/bib bibliography repository] but you only need one copy of this, not one for each paper. A suitable directory structure might be to have a papers/ directory under your home directory that contains a directory for the bibliography repository and a directory for each paper you are currently working on.

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You should assign the request to yourself and anyone else who has worked on this branch. Choose the person who is going to review for you. Make sure you tick the boxes to delete the feature branch after the request is accepted. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you select as reviewer will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. However, the reviewer will not be doing extensive testing, so it remains your responsibility to follow the coding standards and make sure everything works. You should make sure your changes compile with nagfor before submitting the merge request, as that is the most particular compiler. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

$ git fetch -p && for branch in $(git branch -vv | grep ': gone]' | awk '{print $1}'); do git branch -D $branch; done

Delete all local branches that do not exist on GitLab. This command is useful to periodically clean up local branches after they have been merged and deleted on GitLab. Warning: do not run this command if you've created a new local branch and not yet pushed it to GitLab.

Git Workflow

2023-08-08T09:31:55Z

Jwrm2: /* Writing a Paper */ Updated to reflect having a repository per paper.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. If you get the error message

Error: failed to call git rev-parse --git-dir: exit status 128 : fatal: .git

then try

$ git lfs install --skip-repo

instead. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/softwarewales software]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Submodules==

Our software repository has become quite large as external potentials have been added. Submodules offer a way to compartmentalise the repository and speed up clones and updates. Self-contained third-party potentials are good candidates for splitting off into submodules. We will use GDML (Gradient-Descent Machine Learning) as an example. On an initial checkout, you will see a GDML/ directory in the root of the repository, but the directory will be empty. Most users do not need GDML, so will not care or even be particularly aware that GDML has not be cloned.

Let us suppose that you actually need GDML. You can tell git that it is required by running

$ git submodule init GDML

Then the next time you run

$ git submodule update

the contents of GDML will be checked out. If GDML is subsequently updated, run the update command again to get the newest version. If at some later point you decide that you have finished with GDML and no longer need it checked out, run

$ git submodule deinit GDML

and the GDML directory will be emptied. If the submodule you are interested in has its own submodules, add the --recursive flag to the commands. If you decide that you want all the submodules run

$ git submodule update --init --recursive

and all submodules will be checked out. This command is not recommended and you must have a good reason why you need all the submodules before you consider running it.

===Creating a new submodule===

If you are creating an interface to a new large external potential, it may be appropriate to add the external potential as a submodule. It is appropriate if the potential is quite large (more than a few MB), is being placed in the root of the repository, and is not likely to require much in the way of changes after the interface is set up. Instead of adding the external potential to the softwarewales repository, create a new repository under Wales Group on Gitlab and place the files there. Within softwarewales run

$ git submodule add git@gitlab.developers.cam.ac.uk:ch/wales/my_new_repository.git

where my_new_repository is replaced with whatever you named the new repository. You could also got the appropriate URL by going onto Gitlab for the new repository, clicking the 'Clone' button and copying. This command makes the necessary changes to the .gitmodules file, which will then need to be committed and pushed.

Turning an existing subdirectory into a submodule is also possible, but is slightly more complicated and is considered an advanced topic. Google is your friend. Do not mess with the Gitlab repository until you are satisfied you have made the correct adjustments locally.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Each paper is a separate repository. To start a new paper, go to Gitlab and create a new repository by clicking the blue 'New Project' button on your home screen. Create a blank project and make sure the project URL indicates it is under ch/wales rather than in your user space. Checkout the new repository and start writing the paper in the blank directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf documents either (except perhaps for proofs created by the journal). When it comes to revisions and resubmissions, do not create a new subdirectory for the new version. Git keeps the whole history so it is always possible to revert to a previous version.

You will also want to checkout the [git@gitlab.developers.cam.ac.uk:ch/wales/bib.git bibliography repository] but you only need one copy of this, not one for each paper. A suitable directory structure might be to have a papers/ directory under your home directory that contains a directory for the bibliography repository and a directory for each paper you are currently working on.

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You should assign the request to yourself and anyone else who has worked on this branch. Choose the person who is going to review for you. Make sure you tick the boxes to delete the feature branch after the request is accepted. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you select as reviewer will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. However, the reviewer will not be doing extensive testing, so it remains your responsibility to follow the coding standards and make sure everything works. You should make sure your changes compile with nagfor before submitting the merge request, as that is the most particular compiler. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

$ git fetch -p && for branch in $(git branch -vv | grep ': gone]' | awk '{print $1}'); do git branch -D $branch; done

Delete all local branches that do not exist on GitLab. This command is useful to periodically clean up local branches after they have been merged and deleted on GitLab. Warning: do not run this command if you've created a new local branch and not yet pushed it to GitLab.

Git Workflow

2023-08-08T09:22:06Z

Jwrm2: /* Initial Checkout */ Updated Gitlab link.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. If you get the error message

Error: failed to call git rev-parse --git-dir: exit status 128 : fatal: .git

then try

$ git lfs install --skip-repo

instead. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/softwarewales software]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Submodules==

Our software repository has become quite large as external potentials have been added. Submodules offer a way to compartmentalise the repository and speed up clones and updates. Self-contained third-party potentials are good candidates for splitting off into submodules. We will use GDML (Gradient-Descent Machine Learning) as an example. On an initial checkout, you will see a GDML/ directory in the root of the repository, but the directory will be empty. Most users do not need GDML, so will not care or even be particularly aware that GDML has not be cloned.

Let us suppose that you actually need GDML. You can tell git that it is required by running

$ git submodule init GDML

Then the next time you run

$ git submodule update

the contents of GDML will be checked out. If GDML is subsequently updated, run the update command again to get the newest version. If at some later point you decide that you have finished with GDML and no longer need it checked out, run

$ git submodule deinit GDML

and the GDML directory will be emptied. If the submodule you are interested in has its own submodules, add the --recursive flag to the commands. If you decide that you want all the submodules run

$ git submodule update --init --recursive

and all submodules will be checked out. This command is not recommended and you must have a good reason why you need all the submodules before you consider running it.

===Creating a new submodule===

If you are creating an interface to a new large external potential, it may be appropriate to add the external potential as a submodule. It is appropriate if the potential is quite large (more than a few MB), is being placed in the root of the repository, and is not likely to require much in the way of changes after the interface is set up. Instead of adding the external potential to the softwarewales repository, create a new repository under Wales Group on Gitlab and place the files there. Within softwarewales run

$ git submodule add git@gitlab.developers.cam.ac.uk:ch/wales/my_new_repository.git

where my_new_repository is replaced with whatever you named the new repository. You could also got the appropriate URL by going onto Gitlab for the new repository, clicking the 'Clone' button and copying. This command makes the necessary changes to the .gitmodules file, which will then need to be committed and pushed.

Turning an existing subdirectory into a submodule is also possible, but is slightly more complicated and is considered an advanced topic. Google is your friend. Do not mess with the Gitlab repository until you are satisfied you have made the correct adjustments locally.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add. You will need to add them individually however, with
$ git add -f filename
and cannot add all files at the same time (i.e. using git add -A).

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You should assign the request to yourself and anyone else who has worked on this branch. Choose the person who is going to review for you. Make sure you tick the boxes to delete the feature branch after the request is accepted. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you select as reviewer will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. However, the reviewer will not be doing extensive testing, so it remains your responsibility to follow the coding standards and make sure everything works. You should make sure your changes compile with nagfor before submitting the merge request, as that is the most particular compiler. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

$ git fetch -p && for branch in $(git branch -vv | grep ': gone]' | awk '{print $1}'); do git branch -D $branch; done

Delete all local branches that do not exist on GitLab. This command is useful to periodically clean up local branches after they have been merged and deleted on GitLab. Warning: do not run this command if you've created a new local branch and not yet pushed it to GitLab.

Git Workflow

2023-08-08T09:21:18Z

Jwrm2: /* Initial Checkout */ Changed from papers to software.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. If you get the error message

Error: failed to call git rev-parse --git-dir: exit status 128 : fatal: .git

then try

$ git lfs install --skip-repo

instead. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/softwarewales papers]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Submodules==

Our software repository has become quite large as external potentials have been added. Submodules offer a way to compartmentalise the repository and speed up clones and updates. Self-contained third-party potentials are good candidates for splitting off into submodules. We will use GDML (Gradient-Descent Machine Learning) as an example. On an initial checkout, you will see a GDML/ directory in the root of the repository, but the directory will be empty. Most users do not need GDML, so will not care or even be particularly aware that GDML has not be cloned.

Let us suppose that you actually need GDML. You can tell git that it is required by running

$ git submodule init GDML

Then the next time you run

$ git submodule update

the contents of GDML will be checked out. If GDML is subsequently updated, run the update command again to get the newest version. If at some later point you decide that you have finished with GDML and no longer need it checked out, run

$ git submodule deinit GDML

and the GDML directory will be emptied. If the submodule you are interested in has its own submodules, add the --recursive flag to the commands. If you decide that you want all the submodules run

$ git submodule update --init --recursive

and all submodules will be checked out. This command is not recommended and you must have a good reason why you need all the submodules before you consider running it.

===Creating a new submodule===

If you are creating an interface to a new large external potential, it may be appropriate to add the external potential as a submodule. It is appropriate if the potential is quite large (more than a few MB), is being placed in the root of the repository, and is not likely to require much in the way of changes after the interface is set up. Instead of adding the external potential to the softwarewales repository, create a new repository under Wales Group on Gitlab and place the files there. Within softwarewales run

$ git submodule add git@gitlab.developers.cam.ac.uk:ch/wales/my_new_repository.git

where my_new_repository is replaced with whatever you named the new repository. You could also got the appropriate URL by going onto Gitlab for the new repository, clicking the 'Clone' button and copying. This command makes the necessary changes to the .gitmodules file, which will then need to be committed and pushed.

Turning an existing subdirectory into a submodule is also possible, but is slightly more complicated and is considered an advanced topic. Google is your friend. Do not mess with the Gitlab repository until you are satisfied you have made the correct adjustments locally.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add. You will need to add them individually however, with
$ git add -f filename
and cannot add all files at the same time (i.e. using git add -A).

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You should assign the request to yourself and anyone else who has worked on this branch. Choose the person who is going to review for you. Make sure you tick the boxes to delete the feature branch after the request is accepted. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you select as reviewer will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. However, the reviewer will not be doing extensive testing, so it remains your responsibility to follow the coding standards and make sure everything works. You should make sure your changes compile with nagfor before submitting the merge request, as that is the most particular compiler. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

$ git fetch -p && for branch in $(git branch -vv | grep ': gone]' | awk '{print $1}'); do git branch -D $branch; done

Delete all local branches that do not exist on GitLab. This command is useful to periodically clean up local branches after they have been merged and deleted on GitLab. Warning: do not run this command if you've created a new local branch and not yet pushed it to GitLab.

Git Workflow

2023-08-08T09:19:47Z

Jwrm2: /* Working on the group code */ Instructed to compile with nagfor before merging.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. If you get the error message

Error: failed to call git rev-parse --git-dir: exit status 128 : fatal: .git

then try

$ git lfs install --skip-repo

instead. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/paperswales papers]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/paperswales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Submodules==

Our software repository has become quite large as external potentials have been added. Submodules offer a way to compartmentalise the repository and speed up clones and updates. Self-contained third-party potentials are good candidates for splitting off into submodules. We will use GDML (Gradient-Descent Machine Learning) as an example. On an initial checkout, you will see a GDML/ directory in the root of the repository, but the directory will be empty. Most users do not need GDML, so will not care or even be particularly aware that GDML has not be cloned.

Let us suppose that you actually need GDML. You can tell git that it is required by running

$ git submodule init GDML

Then the next time you run

$ git submodule update

the contents of GDML will be checked out. If GDML is subsequently updated, run the update command again to get the newest version. If at some later point you decide that you have finished with GDML and no longer need it checked out, run

$ git submodule deinit GDML

and the GDML directory will be emptied. If the submodule you are interested in has its own submodules, add the --recursive flag to the commands. If you decide that you want all the submodules run

$ git submodule update --init --recursive

and all submodules will be checked out. This command is not recommended and you must have a good reason why you need all the submodules before you consider running it.

===Creating a new submodule===

If you are creating an interface to a new large external potential, it may be appropriate to add the external potential as a submodule. It is appropriate if the potential is quite large (more than a few MB), is being placed in the root of the repository, and is not likely to require much in the way of changes after the interface is set up. Instead of adding the external potential to the softwarewales repository, create a new repository under Wales Group on Gitlab and place the files there. Within softwarewales run

$ git submodule add git@gitlab.developers.cam.ac.uk:ch/wales/my_new_repository.git

where my_new_repository is replaced with whatever you named the new repository. You could also got the appropriate URL by going onto Gitlab for the new repository, clicking the 'Clone' button and copying. This command makes the necessary changes to the .gitmodules file, which will then need to be committed and pushed.

Turning an existing subdirectory into a submodule is also possible, but is slightly more complicated and is considered an advanced topic. Google is your friend. Do not mess with the Gitlab repository until you are satisfied you have made the correct adjustments locally.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add. You will need to add them individually however, with
$ git add -f filename
and cannot add all files at the same time (i.e. using git add -A).

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You should assign the request to yourself and anyone else who has worked on this branch. Choose the person who is going to review for you. Make sure you tick the boxes to delete the feature branch after the request is accepted. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you select as reviewer will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. However, the reviewer will not be doing extensive testing, so it remains your responsibility to follow the coding standards and make sure everything works. You should make sure your changes compile with nagfor before submitting the merge request, as that is the most particular compiler. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

$ git fetch -p && for branch in $(git branch -vv | grep ': gone]' | awk '{print $1}'); do git branch -D $branch; done

Delete all local branches that do not exist on GitLab. This command is useful to periodically clean up local branches after they have been merged and deleted on GitLab. Warning: do not run this command if you've created a new local branch and not yet pushed it to GitLab.

Git Workflow

2023-01-26T16:30:32Z

Jwrm2: /* Submodules */ Typos

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. If you get the error message

Error: failed to call git rev-parse --git-dir: exit status 128 : fatal: .git

then try

$ git lfs install --skip-repo

instead. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/paperswales papers]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/paperswales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Submodules==

Our software repository has become quite large as external potentials have been added. Submodules offer a way to compartmentalise the repository and speed up clones and updates. Self-contained third-party potentials are good candidates for splitting off into submodules. We will use GDML (Gradient-Descent Machine Learning) as an example. On an initial checkout, you will see a GDML/ directory in the root of the repository, but the directory will be empty. Most users do not need GDML, so will not care or even be particularly aware that GDML has not be cloned.

Let us suppose that you actually need GDML. You can tell git that it is required by running

$ git submodule init GDML

Then the next time you run

$ git submodule update

the contents of GDML will be checked out. If GDML is subsequently updated, run the update command again to get the newest version. If at some later point you decide that you have finished with GDML and no longer need it checked out, run

$ git submodule deinit GDML

and the GDML directory will be emptied. If the submodule you are interested in has its own submodules, add the --recursive flag to the commands. If you decide that you want all the submodules run

$ git submodule update --init --recursive

and all submodules will be checked out. This command is not recommended and you must have a good reason why you need all the submodules before you consider running it.

===Creating a new submodule===

If you are creating an interface to a new large external potential, it may be appropriate to add the external potential as a submodule. It is appropriate if the potential is quite large (more than a few MB), is being placed in the root of the repository, and is not likely to require much in the way of changes after the interface is set up. Instead of adding the external potential to the softwarewales repository, create a new repository under Wales Group on Gitlab and place the files there. Within softwarewales run

$ git submodule add git@gitlab.developers.cam.ac.uk:ch/wales/my_new_repository.git

where my_new_repository is replaced with whatever you named the new repository. You could also got the appropriate URL by going onto Gitlab for the new repository, clicking the 'Clone' button and copying. This command makes the necessary changes to the .gitmodules file, which will then need to be committed and pushed.

Turning an existing subdirectory into a submodule is also possible, but is slightly more complicated and is considered an advanced topic. Google is your friend. Do not mess with the Gitlab repository until you are satisfied you have made the correct adjustments locally.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add. You will need to add them individually however, with
$ git add -f filename
and cannot add all files at the same time (i.e. using git add -A).

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You can also choose who to assign it to. Make sure you tick the boxes to delete the feature branch after the request is accepted, and to squash your branch into one commit. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you assign to will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

$ git fetch -p && for branch in $(git branch -vv | grep ': gone]' | awk '{print $1}'); do git branch -D $branch; done

Delete all local branches that do not exist on GitLab. This command is useful to periodically clean up local branches after they have been merged and deleted on GitLab. Warning: do not run this command if you've created a new local branch and not yet pushed it to GitLab.

Git Workflow

2022-12-19T13:43:28Z

Jwrm2: /* Creating a new submodule */ Typos

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. If you get the error message

Error: failed to call git rev-parse --git-dir: exit status 128 : fatal: .git

then try

$ git lfs install --skip-repo

instead. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/paperswales papers]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/paperswales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Submodules==

Our software repository has become quite large as external potentials have been added. Submodules offer a way to compartmentalise the reporistory and speed up clones and updates. Self-contained third-party potentials are good candidates for splitting off into submodules. We will use GDML (Gradient-Descent Machine Learning) as an example. On an initial checkout, you will see a GDML/ directory in the root of the repository, but the directpory will be empty. Most users do not need GDML, so will not care or even be particularly aware that GDML has not be cloned.

Let us suppose that you actually need GDML. You can tell git that it is required by running

$ git submodule init GDML

Then the next time you run

$ git submodule update

the contents of GDML will be checked out. If GDML is subsequently updated, run the update command again to get the newest version. If at some later point you decide that you have finihsed with GDML and no longer need it checked out, run

$ git submodule deinit GDML

and the GDML directory will be emptied. If the submodule you are interested in has its own submodules, add the --recursive flag to the commands. If you decide that you want all the submodules run

$ git submodule update --init --recursive

and all submodules will be checked out. This command is not recommended and you must have a good reason why you need all the submodules before you consider running it.

===Creating a new submodule===

If you are creating an interface to a new large external potential, it may be appropriate to add the external potential as a submodule. It is appropriate if the potential is quite large (more than a few MB), is being placed in the root of the repository, and is not likely to require much in the way of changes after the interface is set up. Instead of adding the external potential to the softwarewales repository, create a new repository under Wales Group on Gitlab and place the files there. Within softwarewales run

$ git submodule add git@gitlab.developers.cam.ac.uk:ch/wales/my_new_repository.git

where my_new_repository is replaced with whatever you named the new repository. You could also got the appropriate URL by going onto Gitlab for the new repository, clicking the 'Clone' button and copying. This command makes the necessary changes to the .gitmodules file, which will then need to be committed and pushed.

Turing an existing subdirectory into a submodule is also possible, but is slightly more complicated and is considered an advanced topic. Google is your friend. Do not mess with the Gitlab repository until you are satisfied you have made the correct adjustments locally.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add. You will need to add them individually however, with
$ git add -f filename
and cannot add all files at the same time (i.e. using git add -A).

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You can also choose who to assign it to. Make sure you tick the boxes to delete the feature branch after the request is accepted, and to squash your branch into one commit. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you assign to will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

$ git fetch -p && for branch in $(git branch -vv | grep ': gone]' | awk '{print $1}'); do git branch -D $branch; done

Delete all local branches that do not exist on GitLab. This command is useful to periodically clean up local branches after they have been merged and deleted on GitLab. Warning: do not run this command if you've created a new local branch and not yet pushed it to GitLab.

Git Workflow

2022-12-19T13:43:03Z

Jwrm2: Added submodule information.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. If you get the error message

Error: failed to call git rev-parse --git-dir: exit status 128 : fatal: .git

then try

$ git lfs install --skip-repo

instead. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/paperswales papers]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/paperswales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Submodules==

Our software repository has become quite large as external potentials have been added. Submodules offer a way to compartmentalise the reporistory and speed up clones and updates. Self-contained third-party potentials are good candidates for splitting off into submodules. We will use GDML (Gradient-Descent Machine Learning) as an example. On an initial checkout, you will see a GDML/ directory in the root of the repository, but the directpory will be empty. Most users do not need GDML, so will not care or even be particularly aware that GDML has not be cloned.

Let us suppose that you actually need GDML. You can tell git that it is required by running

$ git submodule init GDML

Then the next time you run

$ git submodule update

the contents of GDML will be checked out. If GDML is subsequently updated, run the update command again to get the newest version. If at some later point you decide that you have finihsed with GDML and no longer need it checked out, run

$ git submodule deinit GDML

and the GDML directory will be emptied. If the submodule you are interested in has its own submodules, add the --recursive flag to the commands. If you decide that you want all the submodules run

$ git submodule update --init --recursive

and all submodules will be checked out. This command is not recommended and you must have a good reason why you need all the submodules before you consider running it.

===Creating a new submodule===

If you are creating an interface to a new large external potential, it may be appropriate to add the external potential as a submodule. It is appropriate if the potential is quite large (more than a few MB), is being placed in the root of the repository, and is not likely to require much in the way of changes after the interface is set up. Instead of adding the external potential to the softwarewales repository, create a new repository under Wales Group on Gitlab and place the files there. Within softwarewales run

$ git submodule add git@gitlab.developers.cam.ac.uk:ch/wales/my_new_repository.git

where my_new_repository is replaced with whatever you named the new repository. You could also got the appropriate URL by going onto Gitlab for the new repository, clicking the 'Clone' button and copying. This command makes the necessary changes to the .gitmodules file, which will then need to be committed and pushed.

Turing an existing subdirectory into a submodule is also possible, but is slightly more complicated and is considered and advanced topic. Google is your friend. Do not mess with the Gitlab repository until you are satisfied you have made the correct adjustments locally.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add. You will need to add them individually however, with
$ git add -f filename
and cannot add all files at the same time (i.e. using git add -A).

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You can also choose who to assign it to. Make sure you tick the boxes to delete the feature branch after the request is accepted, and to squash your branch into one commit. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you assign to will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

$ git fetch -p && for branch in $(git branch -vv | grep ': gone]' | awk '{print $1}'); do git branch -D $branch; done

Delete all local branches that do not exist on GitLab. This command is useful to periodically clean up local branches after they have been merged and deleted on GitLab. Warning: do not run this command if you've created a new local branch and not yet pushed it to GitLab.

Compiling Wales Group codes using cmake

2022-09-09T14:04:06Z

Jwrm2: /* Extra command line build examples */

[http://www.cmake.org/ CMake] (Cross-platform Make) provides a simple, platform independent way for us to compile and test the group codebase. Dependencies are handled automatically, compilation can proceed in parallel to avoid long waits while testing changes and builds are done entirely outside of the source directory. It also enables us to use the [[Jenkins CI]] 'build bot' system to automatically compile and test the code on a nightly basis - helping us catch troublesome commits before they affect other users.

Although everything below refers to compiling [[GMIN]] with the Intel ''ifort'' compiler and AMBER9 - the exact same procedure works for [[OPTIM]] and [[PATHSAMPLE]].

Note that not every option for our codes is expected to actually compile with every compiler, for example, anything using CHARMM35/36 will not compile with ''nagfor'' or ''gfortran''. This is nothing to do with our code - it's a CHARMM issue. You can get an idea for what should work by looking at the automated [[Jenkins CI]] builds.

==Preparing to compile==
Before you get started, you need to ensure that the machine you are planning to compile on has cmake 2.8 or higher installed. You can check the current version like so:
<pre>
cmake --version
</pre>

The clusters have a module for cmake 3.0, which you can load using the following command:
<pre>
module load cmake/3.0.0
</pre>

You also need to create a directory to build the code in. We suggest that you create a directory for the compiler you are using within the program directory, under a subdirectory called 'builds' - for example for compiling GMIN with ifort, you would make a directory here:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/ifort
cd ~/softwarewales/GMIN/builds/ifort
</pre>
You can call these directories whatever you like - but make sure it is clear to you what they contain! You might also want to check which version of the compiler you have loaded. This is important as the different clusters and workstations may have different default versions loaded, some of which might not work properly. You can check the compiler version currently loaded using the same '--version' flag we used for ''cmake'' above:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ifort --version
ifort (IFORT) 12.1.3 20120212
Copyright (C) 1985-2012 Intel Corporation. All rights reserved.
</pre>

To load a different compiler, you can use the ''module load'' or ''module swap'' commands. A list of all available modules can be accessed using:
<pre>
module av
</pre>
If you are having problems compiling, one of the first things to check is whether it works with a different version of the compiler!

'''Note: When compiling GMIN, if you are getting the error that there is no implicit type for ERFC in ewald.f90, try using a newer version of your compiler. This should be the built-in complementary error function.'''

==Compiling using the ccmake GUI interface to set options==
[[Image:Ccmake.png|thumb|ccmake set up to compile A9GMIN|200px|right]]

One advantage using cmake has over make is that we can use the simple ccmake GUI. This interface lets us set options like compiling with AMBER9, or CHARMM35, toggle between 'Release' and 'Debug' builds (see below) - and examine and alter the flags being uses for the compilation if we wish. Before we can run ccmake, we need to specify the compiler and run cmake in our build directory (e.g. softwarewales/GMIN/builds/ifort). We specify the '''F'''ortran '''C'''ompiler by setting the '''$FC''' environment variable (in this case the Intel Fortran compiler, ifort), and then run ''cmake'' (on the command line), passing it the relative location of the [[GMIN]] source directory:
<pre>
FC=ifort cmake ../../source
</pre>

If you run ''ls'', you will see some cmake files have been generated:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ls
CMakeCache.txt CMakeFiles cmake_install.cmake Makefile modules
</pre>

You can now run ''ccmake'' to open the GUI:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ccmake .
</pre>

To navigate between options, use the arrow keys. Options can be toggled by pressing Return. To compile [[GMIN]] with AMBER9 (A9GMIN), we need to toggle the ''WITH_AMBER'' option ''ON''. Once you have done this, you need to configure and generate appropriate cmake info. This is done by pressing 'c' to configure, 'e' to exit and then 'g' to generate.

'''Note: for some builds (CHARMM with DFTB and CUDAGMIN), you might need to configure, exit and generate twice to set all necessary options'''

You can now compile A9GMIN in parallel as follows:
<pre>
make -j8
</pre>

The '-j8' flag here tells make to use up to 8 'threads' when building. For optimal performance, you should keep this slightly greater than the number of cores (CPUs) the node you are working on has. If all goes well, you should now have an A9GMIN binary in your build directory - congratulations!
<pre>
Linking Fortran executable A9GMIN
[100%] Built target A9GMIN

--------------------------------------------------------------------------------------------- 15:23:45

csw34@sinister:~/softwarewales/GMIN/builds/ifort> ls
A9GMIN cmake_install.cmake libcudadummylib.a libmylapack.a NAB
AMBER display_version.f90 libdummylib.a Makefile nab_binaries_built
CMakeCache.txt GMIN libgminlib.a modules porfuncs.f90
CMakeFiles libamber12dummylib.a libmyblas.a n
</pre>
Plain [[GMIN]] is also built at the same time should you need it. You can move this into your ~/bin directory if you like, or anywhere else in your ''$PATH'' to make running it simple.

'''Note: If you want to use OPTIM with the new C++ implementation of the NEB routine, you will need to obtain the source code for that separately. See [https://wikis.ch.cam.ac.uk/wales/wiki/index.php/OPTIM here] for instructions.'''

==Compiling by setting options on the command line==
If you know the options you'd like to set already (you can see them all in ccmake), you can save some time by passing them directly to ''cmake'' on the command line, bypassing the need for ''ccmake''. For example, to compile A9GMIN (GMIN with the AMBER9 interface) using the Intel ifort compiler, you would run the following commands:
<pre>
FC=ifort cmake -DWITH_AMBER=1 ../../source
make -j8
</pre>
where '../../source' is the relative location of the GMIN source directory. You can find some more examples of compiling from the command line below.

'''Note: Sometimes you may get error''' (for example, Fatal Error: Can't open module file 'someModule.mod' for reading at (1): No such file or directory) when following this procedure. In that case there are three things you could try: make sure you are building in a new directory, if that does not help run `make VERBOSE=1` instead of `make -j8` or simply switch to using ccmake.

==Compiling with MPI==
To compile with MPI support add the following flags when running cmake on the command line:
<pre>
FC=mpif90 CC=mpicc cmake ../source -DCOMPILER_SWITCH=pgi -DWITH_MPI=yes
</pre>
Here -DCOMPILER_SWITCH=pgi assumes you're using the Portland ''pgi'' compiler. Make sure you have the correct modules loaded (in this case ''pgi'' and ''mpi-pgi''), and that the particular mpi you want (in this case ''mpi-pgi'') is listed before any other mpi's loaded (so that it has the highest priority). The modules can be loaded by typing:
<pre>
module load pgi/64/
module load mpi/openmpi/pgi/64/1.6.3
</pre>

and you can check which modules are loaded and in which order/priority by the ''module list'' command. You may need to ''module unload <name>'' any other mpi's that are higher up in the list than the one you want. You can of course set the COMPILER_SWITCH and WITH_MPI flags in ''ccmake'' if you prefer.

Note: It has been observed that pgi/64/15.1 leads to compilation errors, and for now, it is best to use pgi/64/14.9

==Advanced mode - changing compiler flags with ccmake==
[[Image:Ccmakeadvanced.png|thumb|ccmake advanced mode|200px|right]]

Although initially the ''ccmake'' GUI looks very simple, there is a lot going on under the hood. By pressing 't' you can enter 'Advanced mode' which will show you all of the hidden options, for example the compiler flags that are being passed to ''make'' when you compile the code. You can also make changes to the flags here, for example if you would like to add '-p' to do profiling.

As with changing the build type, you simply select the field you'd like to change using the arrow keys, press Return, make your changes and press Return again to save them. When you subsequently configure and generate as above, those altered flag will be used for the subsequent compilation.

Note that these changes only apply in the build directory in which you make them.

==Debugging runtime problems using gdb or valgrind==
If you are getting a segmentation fault, crash or other unexpected behaviour, you might want to run your job through a debugger like [http://www.gnu.org/software/gdb/ gdb] or [http://valgrind.org/ valgrind]. In order to maximise your chances of getting useful output, you should build a 'Debug' version of the program you are having trouble. To do this, you can either change the ''CMAKE_BUILD_TYPE'' in ''ccmake'' to 'Debug' (press Return, change 'Release' to 'Debug' and press Return again), or on the command line like so for GMIN with AMBER 9 using the Intel ifort compiler:
<pre>
FC=ifort cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_AMBER=1 ../../source
make -j8
</pre>

You can then run the binary ''through'' gdb or valgrind as follows:
<pre>
gdb A9GMIN
</pre>
or
<pre>
valgrind A9GMIN
</pre>

I won't cover debugging with these tools here as it's a science in itself! Do some Googling and ask for help as needed :)

==Debugging compilation problems==
There are many ways to try and track down why your code is not compiling. Before you start changing compilers, building a 'Debug' version or changing machines, you might want to try running make again with the ''VERBOSE'' option enabled. This will dump a lot of potentially useful output:
<pre>
VERBOSE=1 make
</pre>

One possible gotcha: all .f and .f90 files in the relevant source directories will be compiled and added to a library. This is quite different from the old Makefile way of doing things, where source files were explicitly specified for compilation (via their corresponding .o file). So, if you are testing something by for instance copying code.f90 to code.myhack.f90 and code.orig.f90, then slightly editing a line or two of code.myhack.f90 and copying it back to code.f90 for use, this will probably cause linking problems due to multiply-defined subroutines (from all three files). The solution, if you must have alternative versions of the same file hanging round, is to differentiate the filenames by a suffix AFTER the .f[90] .

Another occasional issue is the unexplained compiler bug - a problem with the version of the compiler you happen to be using. You can can an idea for which compiler versions we expect to work by checking the Jenkins build-bot output, as described in the 'Seeing console output' section of the [[Jenkins CI]] page. If you are using a different version of the compiler in question, consider swapping to the version Jenkins is using with 'module swap'.

If the error message you are getting doesn't make sense to you after some Googling, go and ask someone - we all have these problems. Things you can try first include trying a different compiler version, or an entirely different compiler e.g. pgi rather than ifort for example. You should bear in mind that as mentioned above, not all versions of each code will compile with every compiler. Make sure you're not trying to build something that isn't expected to work.

==Extra command line build examples==
The below commands are absolutely not an exhaustive list, but should give you an idea of what is possible. You can use ''ccmake'' as described above to discover which variables (e.g. WITH_AMBER) can be manipulated on the command line like this. All of these examples assume your git repository is set up in ''/home/CRSID/softwarewales'' - make the appropriate modifications if you have it elsewhere.

===GMIN===
'''A12GMIN''' (GMIN with AMBER12) using ifort:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/ifort_amber12
cd !$
FC=ifort cmake -DWITH_AMBER12=1 ../../source
make -j8
</pre>

'''C35GMIN''' (GMIN with CHARMM 35) using pgi:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/pgi_charmm35
cd !$
FC=pgf90 cmake -DWITH_CHARMM35=1 ../../source
make -j8
</pre>

'''CUDAGMIN''' (GMIN leveraging GPU minimisation via the AMBER 12 interface) using ifort:
<pre>
module load cuda/5.5
mkdir -p ~/softwarewales/GMIN/builds/ifort_cuda
cd !$
FC=ifort cmake -DWITH_CUDA=1 ../../source
make -j8
</pre>
This will only work on machines with specific NVIDIA GPUs, for example when submitting jobs on the pat cluster. There is some additional information on the [[Using GMIN with GPUs]] page.

'''DFTBGMIN''' (GMIN with DFTBP) using Intel on nest:
<pre>
module load mkl/64/2022/0/0 cmake/3.23.2 ifort/64/2020/4/304
mkdir -p ~/softwarewales/GMIN/builds/ifort_dftbp
cd !$
FC=ifort CC=icc CXX=icc cmake ../../source -DWITH_DFTBP=yes
make -j8
</pre>

or using GCC on nest:
<pre>
module load cmake/3.23.2 gcc/12.2.0
mkdir -p ~/softwarewales/GMIN/builds/gfortran_dftbp
cd !$
cmake ../../source -DWITH_DFTBP=yes -DWITH_MYBLAS=no -DBLAS_LIBRARIES=/usr/lib64/libopenblas.so.0
make -j8
</pre>

===OPTIM===
'''A9OPTIM''' (OPTIM with AMBER9) using ifort:
<pre>
mkdir -p ~/softwarewales/OPTIM/builds/ifort_amber
cd !$
FC=ifort cmake -DWITH_AMBER9=1 ../../source
make -j8
</pre>

'''C35OPTIM''' (OPTIM with CHARMM 35) using pgi:
<pre>
mkdir -p ~/softwarewales/OPTIM/builds/pgi_charmm35
cd !$
FC=pgf90 cmake -DWITH_CHARMM35=1 ../../source
make -j8
</pre>

'''CUDAOPTIM''' (OPTIM leveraging GPU via the AMBER 12 interface) using ifort:
<pre>
module load cuda/5.5
mkdir -p ~/softwarewales/OPTIM/builds/ifort_cuda5.5
cd !$
FC=ifort CC=icc cmake -DWITH_CUDA=1 ../../source
make -j8
</pre>
This will only work on machines with specific NVIDIA GPUs, for example when submitting jobs on the pat cluster. There is some additional information on the [[Using GMIN and OPTIM with GPUs]] page.

'''DFTBOPTIM''' (OPTIM with DFTBP) using Intel on nest:
<pre>
module load mkl/64/2022/0/0 cmake/3.23.2 ifort/64/2020/4/304
mkdir -p ~/softwarewales/OPTIM/builds/ifort_dftbp
cd !$
FC=ifort CC=icc CXX=icc cmake ../../source -DWITH_DFTBP=yes
make -j8
</pre>

or using GCC on nest:
<pre>
module load cmake/3.23.2 gcc/12.2.0
mkdir -p ~/softwarewales/OPTIM/builds/gfortran_dftbp
cd !$
cmake ../../source -DWITH_DFTBP=yes -DWITH_MYBLAS=no -DBLAS_LIBRARIES=/usr/lib64/libopenblas.so.0
make -j8
</pre>

===PATHSAMPLE===
There are very few options for [[PATHSAMPLE]] as we don't need to worry about interfacing with a particular potential. As a result, every binary is simply called ''PATHSAMPLE''.

Using nagfor (the NAG fortran compiler - check you have the module loaded - very strict!):
<pre>
mkdir -p ~/softwarewales/PATHSAMPLE/builds/nagfor
cd !$
FC=nagfor cmake ../../source
make -j8
</pre>

Using pgi (much more generous with coding slips/non-standard uses):
<pre>
mkdir -p ~/softwarewales/PATHSAMPLE/builds/pgi
cd !$
FC=pgf90 cmake ../../source
make -j8
</pre>

==Configuring defaults - for developers==

Fortran compilers and their corresponding default settings are all controlled by the file $SVN/CMakeModules/FindFORTRANCOMPILER.cmake ($SVN is your svn root directory). In particular, we may wish to edit the flags used for each set of compilers and build type. These are contained in the following block:

<pre>
if(NOT COMPILER_FLAGS_WERE_SET)
message("Setting initial values for compiler flags")
if(COMPILER_SWITCH MATCHES "pgi")
set (CMAKE_Fortran_FLAGS "-Mextend" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-O3 -Munroll -Mnoframe" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG "-Mextend -C -g -gopt -Mbounds -Mchkfpstk -Mchkptr -Mchkstk -Mcoff -Mdwarf1 -Mdwarf2 -Melf -Mpgicoff -traceback" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-Mextend -C -g -gopt -Mbounds -Mchkfpstk -Mchkptr -Mchkstk -Mcoff -Mdwarf1 -Mdwarf2 -Mdwarf3 -Melf -Mpgicoff -traceback" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-Mfree" CACHE TYPE STRING)
elseif(COMPILER_SWITCH MATCHES "gfortran")
set (CMAKE_Fortran_FLAGS "-ffixed-line-length-200 -ffree-line-length-0" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-O3" CACHE TYPE STRING FORCE)
# set (CMAKE_Fortran_FLAGS_DEBUG "-g -fbounds-check -Wuninitialized -O -ftrapv -fimplicit-none -fno-automatic" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG "-g -fbounds-check -Wuninitialized -O -ftrapv -fno-automatic" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "${CMAKE_Fortran_FLAGS_DEBUG} -fimplicit-none" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-ffree-form" CACHE TYPE STRING)
elseif(COMPILER_SWITCH MATCHES "nag")
set (CMAKE_Fortran_FLAGS "-132 -kind=byte -maxcontin=3000" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-mismatch_all -O4" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG "-g -mismatch_all -ieee=stop" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-C=all -mtrace=all -gline -g -mismatch_all -ieee=stop" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-free" CACHE TYPE STRING) # js850> is this ever used?
elseif(COMPILER_SWITCH MATCHES "ifort")
set (CMAKE_Fortran_FLAGS "-132 -heap-arrays -assume byterecl" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-O3" CACHE TYPE STRING FORCE)
# Warnings about temporary argument creation and edit descriptor widths are disabled with the final flags.
set (CMAKE_Fortran_FLAGS_DEBUG "-C -g -traceback -debug full -check all,noarg_temp_created -diag-disable 8290,8291" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-debug all -check all -implicitnone -warn unused -fp-stack-check -ftrapuv -check pointers -check bounds" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-free" CACHE TYPE STRING)
else()
message(FATAL_ERROR "unknown comiler switch: ${COMPILER_SWITCH}")
endif()
SET(COMPILER_FLAGS_WERE_SET yes CACHE TYPE INTERNAL)
endif(NOT COMPILER_FLAGS_WERE_SET)
</pre>

The main if/elseif blocks correspond to compiler switches. Inside these, there are the default flags for each of our build types (release, debug and debug_slow), which are configured using ccmake. These can be edited, if we wish to change the default behaviour (e.g. a recent addition of -check all,noarg_temp_created -diag-disable 8290,8291 to disable annoying warning messages for ifort).

Compiling Wales Group codes using cmake

2022-09-09T14:03:43Z

Jwrm2: /* Extra command line build examples */

[http://www.cmake.org/ CMake] (Cross-platform Make) provides a simple, platform independent way for us to compile and test the group codebase. Dependencies are handled automatically, compilation can proceed in parallel to avoid long waits while testing changes and builds are done entirely outside of the source directory. It also enables us to use the [[Jenkins CI]] 'build bot' system to automatically compile and test the code on a nightly basis - helping us catch troublesome commits before they affect other users.

Although everything below refers to compiling [[GMIN]] with the Intel ''ifort'' compiler and AMBER9 - the exact same procedure works for [[OPTIM]] and [[PATHSAMPLE]].

Note that not every option for our codes is expected to actually compile with every compiler, for example, anything using CHARMM35/36 will not compile with ''nagfor'' or ''gfortran''. This is nothing to do with our code - it's a CHARMM issue. You can get an idea for what should work by looking at the automated [[Jenkins CI]] builds.

==Preparing to compile==
Before you get started, you need to ensure that the machine you are planning to compile on has cmake 2.8 or higher installed. You can check the current version like so:
<pre>
cmake --version
</pre>

The clusters have a module for cmake 3.0, which you can load using the following command:
<pre>
module load cmake/3.0.0
</pre>

You also need to create a directory to build the code in. We suggest that you create a directory for the compiler you are using within the program directory, under a subdirectory called 'builds' - for example for compiling GMIN with ifort, you would make a directory here:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/ifort
cd ~/softwarewales/GMIN/builds/ifort
</pre>
You can call these directories whatever you like - but make sure it is clear to you what they contain! You might also want to check which version of the compiler you have loaded. This is important as the different clusters and workstations may have different default versions loaded, some of which might not work properly. You can check the compiler version currently loaded using the same '--version' flag we used for ''cmake'' above:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ifort --version
ifort (IFORT) 12.1.3 20120212
Copyright (C) 1985-2012 Intel Corporation. All rights reserved.
</pre>

To load a different compiler, you can use the ''module load'' or ''module swap'' commands. A list of all available modules can be accessed using:
<pre>
module av
</pre>
If you are having problems compiling, one of the first things to check is whether it works with a different version of the compiler!

'''Note: When compiling GMIN, if you are getting the error that there is no implicit type for ERFC in ewald.f90, try using a newer version of your compiler. This should be the built-in complementary error function.'''

==Compiling using the ccmake GUI interface to set options==
[[Image:Ccmake.png|thumb|ccmake set up to compile A9GMIN|200px|right]]

One advantage using cmake has over make is that we can use the simple ccmake GUI. This interface lets us set options like compiling with AMBER9, or CHARMM35, toggle between 'Release' and 'Debug' builds (see below) - and examine and alter the flags being uses for the compilation if we wish. Before we can run ccmake, we need to specify the compiler and run cmake in our build directory (e.g. softwarewales/GMIN/builds/ifort). We specify the '''F'''ortran '''C'''ompiler by setting the '''$FC''' environment variable (in this case the Intel Fortran compiler, ifort), and then run ''cmake'' (on the command line), passing it the relative location of the [[GMIN]] source directory:
<pre>
FC=ifort cmake ../../source
</pre>

If you run ''ls'', you will see some cmake files have been generated:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ls
CMakeCache.txt CMakeFiles cmake_install.cmake Makefile modules
</pre>

You can now run ''ccmake'' to open the GUI:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ccmake .
</pre>

To navigate between options, use the arrow keys. Options can be toggled by pressing Return. To compile [[GMIN]] with AMBER9 (A9GMIN), we need to toggle the ''WITH_AMBER'' option ''ON''. Once you have done this, you need to configure and generate appropriate cmake info. This is done by pressing 'c' to configure, 'e' to exit and then 'g' to generate.

'''Note: for some builds (CHARMM with DFTB and CUDAGMIN), you might need to configure, exit and generate twice to set all necessary options'''

You can now compile A9GMIN in parallel as follows:
<pre>
make -j8
</pre>

The '-j8' flag here tells make to use up to 8 'threads' when building. For optimal performance, you should keep this slightly greater than the number of cores (CPUs) the node you are working on has. If all goes well, you should now have an A9GMIN binary in your build directory - congratulations!
<pre>
Linking Fortran executable A9GMIN
[100%] Built target A9GMIN

--------------------------------------------------------------------------------------------- 15:23:45

csw34@sinister:~/softwarewales/GMIN/builds/ifort> ls
A9GMIN cmake_install.cmake libcudadummylib.a libmylapack.a NAB
AMBER display_version.f90 libdummylib.a Makefile nab_binaries_built
CMakeCache.txt GMIN libgminlib.a modules porfuncs.f90
CMakeFiles libamber12dummylib.a libmyblas.a n
</pre>
Plain [[GMIN]] is also built at the same time should you need it. You can move this into your ~/bin directory if you like, or anywhere else in your ''$PATH'' to make running it simple.

'''Note: If you want to use OPTIM with the new C++ implementation of the NEB routine, you will need to obtain the source code for that separately. See [https://wikis.ch.cam.ac.uk/wales/wiki/index.php/OPTIM here] for instructions.'''

==Compiling by setting options on the command line==
If you know the options you'd like to set already (you can see them all in ccmake), you can save some time by passing them directly to ''cmake'' on the command line, bypassing the need for ''ccmake''. For example, to compile A9GMIN (GMIN with the AMBER9 interface) using the Intel ifort compiler, you would run the following commands:
<pre>
FC=ifort cmake -DWITH_AMBER=1 ../../source
make -j8
</pre>
where '../../source' is the relative location of the GMIN source directory. You can find some more examples of compiling from the command line below.

'''Note: Sometimes you may get error''' (for example, Fatal Error: Can't open module file 'someModule.mod' for reading at (1): No such file or directory) when following this procedure. In that case there are three things you could try: make sure you are building in a new directory, if that does not help run `make VERBOSE=1` instead of `make -j8` or simply switch to using ccmake.

==Compiling with MPI==
To compile with MPI support add the following flags when running cmake on the command line:
<pre>
FC=mpif90 CC=mpicc cmake ../source -DCOMPILER_SWITCH=pgi -DWITH_MPI=yes
</pre>
Here -DCOMPILER_SWITCH=pgi assumes you're using the Portland ''pgi'' compiler. Make sure you have the correct modules loaded (in this case ''pgi'' and ''mpi-pgi''), and that the particular mpi you want (in this case ''mpi-pgi'') is listed before any other mpi's loaded (so that it has the highest priority). The modules can be loaded by typing:
<pre>
module load pgi/64/
module load mpi/openmpi/pgi/64/1.6.3
</pre>

and you can check which modules are loaded and in which order/priority by the ''module list'' command. You may need to ''module unload <name>'' any other mpi's that are higher up in the list than the one you want. You can of course set the COMPILER_SWITCH and WITH_MPI flags in ''ccmake'' if you prefer.

Note: It has been observed that pgi/64/15.1 leads to compilation errors, and for now, it is best to use pgi/64/14.9

==Advanced mode - changing compiler flags with ccmake==
[[Image:Ccmakeadvanced.png|thumb|ccmake advanced mode|200px|right]]

Although initially the ''ccmake'' GUI looks very simple, there is a lot going on under the hood. By pressing 't' you can enter 'Advanced mode' which will show you all of the hidden options, for example the compiler flags that are being passed to ''make'' when you compile the code. You can also make changes to the flags here, for example if you would like to add '-p' to do profiling.

As with changing the build type, you simply select the field you'd like to change using the arrow keys, press Return, make your changes and press Return again to save them. When you subsequently configure and generate as above, those altered flag will be used for the subsequent compilation.

Note that these changes only apply in the build directory in which you make them.

==Debugging runtime problems using gdb or valgrind==
If you are getting a segmentation fault, crash or other unexpected behaviour, you might want to run your job through a debugger like [http://www.gnu.org/software/gdb/ gdb] or [http://valgrind.org/ valgrind]. In order to maximise your chances of getting useful output, you should build a 'Debug' version of the program you are having trouble. To do this, you can either change the ''CMAKE_BUILD_TYPE'' in ''ccmake'' to 'Debug' (press Return, change 'Release' to 'Debug' and press Return again), or on the command line like so for GMIN with AMBER 9 using the Intel ifort compiler:
<pre>
FC=ifort cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_AMBER=1 ../../source
make -j8
</pre>

You can then run the binary ''through'' gdb or valgrind as follows:
<pre>
gdb A9GMIN
</pre>
or
<pre>
valgrind A9GMIN
</pre>

I won't cover debugging with these tools here as it's a science in itself! Do some Googling and ask for help as needed :)

==Debugging compilation problems==
There are many ways to try and track down why your code is not compiling. Before you start changing compilers, building a 'Debug' version or changing machines, you might want to try running make again with the ''VERBOSE'' option enabled. This will dump a lot of potentially useful output:
<pre>
VERBOSE=1 make
</pre>

One possible gotcha: all .f and .f90 files in the relevant source directories will be compiled and added to a library. This is quite different from the old Makefile way of doing things, where source files were explicitly specified for compilation (via their corresponding .o file). So, if you are testing something by for instance copying code.f90 to code.myhack.f90 and code.orig.f90, then slightly editing a line or two of code.myhack.f90 and copying it back to code.f90 for use, this will probably cause linking problems due to multiply-defined subroutines (from all three files). The solution, if you must have alternative versions of the same file hanging round, is to differentiate the filenames by a suffix AFTER the .f[90] .

Another occasional issue is the unexplained compiler bug - a problem with the version of the compiler you happen to be using. You can can an idea for which compiler versions we expect to work by checking the Jenkins build-bot output, as described in the 'Seeing console output' section of the [[Jenkins CI]] page. If you are using a different version of the compiler in question, consider swapping to the version Jenkins is using with 'module swap'.

If the error message you are getting doesn't make sense to you after some Googling, go and ask someone - we all have these problems. Things you can try first include trying a different compiler version, or an entirely different compiler e.g. pgi rather than ifort for example. You should bear in mind that as mentioned above, not all versions of each code will compile with every compiler. Make sure you're not trying to build something that isn't expected to work.

==Extra command line build examples==
The below commands are absolutely not an exhaustive list, but should give you an idea of what is possible. You can use ''ccmake'' as described above to discover which variables (e.g. WITH_AMBER) can be manipulated on the command line like this. All of these examples assume your svn repository is set up in ''/home/CRSID/softwarewales'' - make the appropriate modifications if you have it elsewhere.

===GMIN===
'''A12GMIN''' (GMIN with AMBER12) using ifort:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/ifort_amber12
cd !$
FC=ifort cmake -DWITH_AMBER12=1 ../../source
make -j8
</pre>

'''C35GMIN''' (GMIN with CHARMM 35) using pgi:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/pgi_charmm35
cd !$
FC=pgf90 cmake -DWITH_CHARMM35=1 ../../source
make -j8
</pre>

'''CUDAGMIN''' (GMIN leveraging GPU minimisation via the AMBER 12 interface) using ifort:
<pre>
module load cuda/5.5
mkdir -p ~/softwarewales/GMIN/builds/ifort_cuda
cd !$
FC=ifort cmake -DWITH_CUDA=1 ../../source
make -j8
</pre>
This will only work on machines with specific NVIDIA GPUs, for example when submitting jobs on the pat cluster. There is some additional information on the [[Using GMIN with GPUs]] page.

'''DFTBGMIN''' (GMIN with DFTBP) using Intel on nest:
<pre>
module load mkl/64/2022/0/0 cmake/3.23.2 ifort/64/2020/4/304
mkdir -p ~/softwarewales/GMIN/builds/ifort_dftbp
cd !$
FC=ifort CC=icc CXX=icc cmake ../../source -DWITH_DFTBP=yes
make -j8
</pre>

or using GCC on nest:
<pre>
module load cmake/3.23.2 gcc/12.2.0
mkdir -p ~/softwarewales/GMIN/builds/gfortran_dftbp
cd !$
cmake ../../source -DWITH_DFTBP=yes -DWITH_MYBLAS=no -DBLAS_LIBRARIES=/usr/lib64/libopenblas.so.0
make -j8
</pre>

===OPTIM===
'''A9OPTIM''' (OPTIM with AMBER9) using ifort:
<pre>
mkdir -p ~/softwarewales/OPTIM/builds/ifort_amber
cd !$
FC=ifort cmake -DWITH_AMBER9=1 ../../source
make -j8
</pre>

'''C35OPTIM''' (OPTIM with CHARMM 35) using pgi:
<pre>
mkdir -p ~/softwarewales/OPTIM/builds/pgi_charmm35
cd !$
FC=pgf90 cmake -DWITH_CHARMM35=1 ../../source
make -j8
</pre>

'''CUDAOPTIM''' (OPTIM leveraging GPU via the AMBER 12 interface) using ifort:
<pre>
module load cuda/5.5
mkdir -p ~/softwarewales/OPTIM/builds/ifort_cuda5.5
cd !$
FC=ifort CC=icc cmake -DWITH_CUDA=1 ../../source
make -j8
</pre>
This will only work on machines with specific NVIDIA GPUs, for example when submitting jobs on the pat cluster. There is some additional information on the [[Using GMIN and OPTIM with GPUs]] page.

'''DFTBOPTIM''' (OPTIM with DFTBP) using Intel on nest:
<pre>
module load mkl/64/2022/0/0 cmake/3.23.2 ifort/64/2020/4/304
mkdir -p ~/softwarewales/OPTIM/builds/ifort_dftbp
cd !$
FC=ifort CC=icc CXX=icc cmake ../../source -DWITH_DFTBP=yes
make -j8
</pre>

or using GCC on nest:
<pre>
module load cmake/3.23.2 gcc/12.2.0
mkdir -p ~/softwarewales/OPTIM/builds/gfortran_dftbp
cd !$
cmake ../../source -DWITH_DFTBP=yes -DWITH_MYBLAS=no -DBLAS_LIBRARIES=/usr/lib64/libopenblas.so.0
make -j8
</pre>

===PATHSAMPLE===
There are very few options for [[PATHSAMPLE]] as we don't need to worry about interfacing with a particular potential. As a result, every binary is simply called ''PATHSAMPLE''.

Using nagfor (the NAG fortran compiler - check you have the module loaded - very strict!):
<pre>
mkdir -p ~/softwarewales/PATHSAMPLE/builds/nagfor
cd !$
FC=nagfor cmake ../../source
make -j8
</pre>

Using pgi (much more generous with coding slips/non-standard uses):
<pre>
mkdir -p ~/softwarewales/PATHSAMPLE/builds/pgi
cd !$
FC=pgf90 cmake ../../source
make -j8
</pre>

==Configuring defaults - for developers==

Fortran compilers and their corresponding default settings are all controlled by the file $SVN/CMakeModules/FindFORTRANCOMPILER.cmake ($SVN is your svn root directory). In particular, we may wish to edit the flags used for each set of compilers and build type. These are contained in the following block:

<pre>
if(NOT COMPILER_FLAGS_WERE_SET)
message("Setting initial values for compiler flags")
if(COMPILER_SWITCH MATCHES "pgi")
set (CMAKE_Fortran_FLAGS "-Mextend" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-O3 -Munroll -Mnoframe" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG "-Mextend -C -g -gopt -Mbounds -Mchkfpstk -Mchkptr -Mchkstk -Mcoff -Mdwarf1 -Mdwarf2 -Melf -Mpgicoff -traceback" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-Mextend -C -g -gopt -Mbounds -Mchkfpstk -Mchkptr -Mchkstk -Mcoff -Mdwarf1 -Mdwarf2 -Mdwarf3 -Melf -Mpgicoff -traceback" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-Mfree" CACHE TYPE STRING)
elseif(COMPILER_SWITCH MATCHES "gfortran")
set (CMAKE_Fortran_FLAGS "-ffixed-line-length-200 -ffree-line-length-0" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-O3" CACHE TYPE STRING FORCE)
# set (CMAKE_Fortran_FLAGS_DEBUG "-g -fbounds-check -Wuninitialized -O -ftrapv -fimplicit-none -fno-automatic" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG "-g -fbounds-check -Wuninitialized -O -ftrapv -fno-automatic" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "${CMAKE_Fortran_FLAGS_DEBUG} -fimplicit-none" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-ffree-form" CACHE TYPE STRING)
elseif(COMPILER_SWITCH MATCHES "nag")
set (CMAKE_Fortran_FLAGS "-132 -kind=byte -maxcontin=3000" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-mismatch_all -O4" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG "-g -mismatch_all -ieee=stop" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-C=all -mtrace=all -gline -g -mismatch_all -ieee=stop" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-free" CACHE TYPE STRING) # js850> is this ever used?
elseif(COMPILER_SWITCH MATCHES "ifort")
set (CMAKE_Fortran_FLAGS "-132 -heap-arrays -assume byterecl" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-O3" CACHE TYPE STRING FORCE)
# Warnings about temporary argument creation and edit descriptor widths are disabled with the final flags.
set (CMAKE_Fortran_FLAGS_DEBUG "-C -g -traceback -debug full -check all,noarg_temp_created -diag-disable 8290,8291" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-debug all -check all -implicitnone -warn unused -fp-stack-check -ftrapuv -check pointers -check bounds" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-free" CACHE TYPE STRING)
else()
message(FATAL_ERROR "unknown comiler switch: ${COMPILER_SWITCH}")
endif()
SET(COMPILER_FLAGS_WERE_SET yes CACHE TYPE INTERNAL)
endif(NOT COMPILER_FLAGS_WERE_SET)
</pre>

The main if/elseif blocks correspond to compiler switches. Inside these, there are the default flags for each of our build types (release, debug and debug_slow), which are configured using ccmake. These can be edited, if we wish to change the default behaviour (e.g. a recent addition of -check all,noarg_temp_created -diag-disable 8290,8291 to disable annoying warning messages for ifort).

Compiling Wales Group codes using cmake

2022-09-09T14:02:37Z

Jwrm2: /* OPTIM */ Added DFTBP examples.

[http://www.cmake.org/ CMake] (Cross-platform Make) provides a simple, platform independent way for us to compile and test the group codebase. Dependencies are handled automatically, compilation can proceed in parallel to avoid long waits while testing changes and builds are done entirely outside of the source directory. It also enables us to use the [[Jenkins CI]] 'build bot' system to automatically compile and test the code on a nightly basis - helping us catch troublesome commits before they affect other users.

Although everything below refers to compiling [[GMIN]] with the Intel ''ifort'' compiler and AMBER9 - the exact same procedure works for [[OPTIM]] and [[PATHSAMPLE]].

Note that not every option for our codes is expected to actually compile with every compiler, for example, anything using CHARMM35/36 will not compile with ''nagfor'' or ''gfortran''. This is nothing to do with our code - it's a CHARMM issue. You can get an idea for what should work by looking at the automated [[Jenkins CI]] builds.

==Preparing to compile==
Before you get started, you need to ensure that the machine you are planning to compile on has cmake 2.8 or higher installed. You can check the current version like so:
<pre>
cmake --version
</pre>

The clusters have a module for cmake 3.0, which you can load using the following command:
<pre>
module load cmake/3.0.0
</pre>

You also need to create a directory to build the code in. We suggest that you create a directory for the compiler you are using within the program directory, under a subdirectory called 'builds' - for example for compiling GMIN with ifort, you would make a directory here:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/ifort
cd ~/softwarewales/GMIN/builds/ifort
</pre>
You can call these directories whatever you like - but make sure it is clear to you what they contain! You might also want to check which version of the compiler you have loaded. This is important as the different clusters and workstations may have different default versions loaded, some of which might not work properly. You can check the compiler version currently loaded using the same '--version' flag we used for ''cmake'' above:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ifort --version
ifort (IFORT) 12.1.3 20120212
Copyright (C) 1985-2012 Intel Corporation. All rights reserved.
</pre>

To load a different compiler, you can use the ''module load'' or ''module swap'' commands. A list of all available modules can be accessed using:
<pre>
module av
</pre>
If you are having problems compiling, one of the first things to check is whether it works with a different version of the compiler!

'''Note: When compiling GMIN, if you are getting the error that there is no implicit type for ERFC in ewald.f90, try using a newer version of your compiler. This should be the built-in complementary error function.'''

==Compiling using the ccmake GUI interface to set options==
[[Image:Ccmake.png|thumb|ccmake set up to compile A9GMIN|200px|right]]

One advantage using cmake has over make is that we can use the simple ccmake GUI. This interface lets us set options like compiling with AMBER9, or CHARMM35, toggle between 'Release' and 'Debug' builds (see below) - and examine and alter the flags being uses for the compilation if we wish. Before we can run ccmake, we need to specify the compiler and run cmake in our build directory (e.g. softwarewales/GMIN/builds/ifort). We specify the '''F'''ortran '''C'''ompiler by setting the '''$FC''' environment variable (in this case the Intel Fortran compiler, ifort), and then run ''cmake'' (on the command line), passing it the relative location of the [[GMIN]] source directory:
<pre>
FC=ifort cmake ../../source
</pre>

If you run ''ls'', you will see some cmake files have been generated:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ls
CMakeCache.txt CMakeFiles cmake_install.cmake Makefile modules
</pre>

You can now run ''ccmake'' to open the GUI:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ccmake .
</pre>

To navigate between options, use the arrow keys. Options can be toggled by pressing Return. To compile [[GMIN]] with AMBER9 (A9GMIN), we need to toggle the ''WITH_AMBER'' option ''ON''. Once you have done this, you need to configure and generate appropriate cmake info. This is done by pressing 'c' to configure, 'e' to exit and then 'g' to generate.

'''Note: for some builds (CHARMM with DFTB and CUDAGMIN), you might need to configure, exit and generate twice to set all necessary options'''

You can now compile A9GMIN in parallel as follows:
<pre>
make -j8
</pre>

The '-j8' flag here tells make to use up to 8 'threads' when building. For optimal performance, you should keep this slightly greater than the number of cores (CPUs) the node you are working on has. If all goes well, you should now have an A9GMIN binary in your build directory - congratulations!
<pre>
Linking Fortran executable A9GMIN
[100%] Built target A9GMIN

--------------------------------------------------------------------------------------------- 15:23:45

csw34@sinister:~/softwarewales/GMIN/builds/ifort> ls
A9GMIN cmake_install.cmake libcudadummylib.a libmylapack.a NAB
AMBER display_version.f90 libdummylib.a Makefile nab_binaries_built
CMakeCache.txt GMIN libgminlib.a modules porfuncs.f90
CMakeFiles libamber12dummylib.a libmyblas.a n
</pre>
Plain [[GMIN]] is also built at the same time should you need it. You can move this into your ~/bin directory if you like, or anywhere else in your ''$PATH'' to make running it simple.

'''Note: If you want to use OPTIM with the new C++ implementation of the NEB routine, you will need to obtain the source code for that separately. See [https://wikis.ch.cam.ac.uk/wales/wiki/index.php/OPTIM here] for instructions.'''

==Compiling by setting options on the command line==
If you know the options you'd like to set already (you can see them all in ccmake), you can save some time by passing them directly to ''cmake'' on the command line, bypassing the need for ''ccmake''. For example, to compile A9GMIN (GMIN with the AMBER9 interface) using the Intel ifort compiler, you would run the following commands:
<pre>
FC=ifort cmake -DWITH_AMBER=1 ../../source
make -j8
</pre>
where '../../source' is the relative location of the GMIN source directory. You can find some more examples of compiling from the command line below.

'''Note: Sometimes you may get error''' (for example, Fatal Error: Can't open module file 'someModule.mod' for reading at (1): No such file or directory) when following this procedure. In that case there are three things you could try: make sure you are building in a new directory, if that does not help run `make VERBOSE=1` instead of `make -j8` or simply switch to using ccmake.

==Compiling with MPI==
To compile with MPI support add the following flags when running cmake on the command line:
<pre>
FC=mpif90 CC=mpicc cmake ../source -DCOMPILER_SWITCH=pgi -DWITH_MPI=yes
</pre>
Here -DCOMPILER_SWITCH=pgi assumes you're using the Portland ''pgi'' compiler. Make sure you have the correct modules loaded (in this case ''pgi'' and ''mpi-pgi''), and that the particular mpi you want (in this case ''mpi-pgi'') is listed before any other mpi's loaded (so that it has the highest priority). The modules can be loaded by typing:
<pre>
module load pgi/64/
module load mpi/openmpi/pgi/64/1.6.3
</pre>

and you can check which modules are loaded and in which order/priority by the ''module list'' command. You may need to ''module unload <name>'' any other mpi's that are higher up in the list than the one you want. You can of course set the COMPILER_SWITCH and WITH_MPI flags in ''ccmake'' if you prefer.

Note: It has been observed that pgi/64/15.1 leads to compilation errors, and for now, it is best to use pgi/64/14.9

==Advanced mode - changing compiler flags with ccmake==
[[Image:Ccmakeadvanced.png|thumb|ccmake advanced mode|200px|right]]

Although initially the ''ccmake'' GUI looks very simple, there is a lot going on under the hood. By pressing 't' you can enter 'Advanced mode' which will show you all of the hidden options, for example the compiler flags that are being passed to ''make'' when you compile the code. You can also make changes to the flags here, for example if you would like to add '-p' to do profiling.

As with changing the build type, you simply select the field you'd like to change using the arrow keys, press Return, make your changes and press Return again to save them. When you subsequently configure and generate as above, those altered flag will be used for the subsequent compilation.

Note that these changes only apply in the build directory in which you make them.

==Debugging runtime problems using gdb or valgrind==
If you are getting a segmentation fault, crash or other unexpected behaviour, you might want to run your job through a debugger like [http://www.gnu.org/software/gdb/ gdb] or [http://valgrind.org/ valgrind]. In order to maximise your chances of getting useful output, you should build a 'Debug' version of the program you are having trouble. To do this, you can either change the ''CMAKE_BUILD_TYPE'' in ''ccmake'' to 'Debug' (press Return, change 'Release' to 'Debug' and press Return again), or on the command line like so for GMIN with AMBER 9 using the Intel ifort compiler:
<pre>
FC=ifort cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_AMBER=1 ../../source
make -j8
</pre>

You can then run the binary ''through'' gdb or valgrind as follows:
<pre>
gdb A9GMIN
</pre>
or
<pre>
valgrind A9GMIN
</pre>

I won't cover debugging with these tools here as it's a science in itself! Do some Googling and ask for help as needed :)

==Debugging compilation problems==
There are many ways to try and track down why your code is not compiling. Before you start changing compilers, building a 'Debug' version or changing machines, you might want to try running make again with the ''VERBOSE'' option enabled. This will dump a lot of potentially useful output:
<pre>
VERBOSE=1 make
</pre>

One possible gotcha: all .f and .f90 files in the relevant source directories will be compiled and added to a library. This is quite different from the old Makefile way of doing things, where source files were explicitly specified for compilation (via their corresponding .o file). So, if you are testing something by for instance copying code.f90 to code.myhack.f90 and code.orig.f90, then slightly editing a line or two of code.myhack.f90 and copying it back to code.f90 for use, this will probably cause linking problems due to multiply-defined subroutines (from all three files). The solution, if you must have alternative versions of the same file hanging round, is to differentiate the filenames by a suffix AFTER the .f[90] .

Another occasional issue is the unexplained compiler bug - a problem with the version of the compiler you happen to be using. You can can an idea for which compiler versions we expect to work by checking the Jenkins build-bot output, as described in the 'Seeing console output' section of the [[Jenkins CI]] page. If you are using a different version of the compiler in question, consider swapping to the version Jenkins is using with 'module swap'.

If the error message you are getting doesn't make sense to you after some Googling, go and ask someone - we all have these problems. Things you can try first include trying a different compiler version, or an entirely different compiler e.g. pgi rather than ifort for example. You should bear in mind that as mentioned above, not all versions of each code will compile with every compiler. Make sure you're not trying to build something that isn't expected to work.

==Extra command line build examples==
The below commands are absolutely not an exhaustive list, but should give you an idea of what is possible. You can use ''ccmake'' as described above to discover which variables (e.g. WITH_AMBER) can be manipulated on the command line like this. All of these examples assume your svn repository is set up in ''/home/CRSID/svn'' - make the appropriate modifications if you have it elsewhere.

===GMIN===
'''A12GMIN''' (GMIN with AMBER12) using ifort:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/ifort_amber12
cd !$
FC=ifort cmake -DWITH_AMBER12=1 ../../source
make -j8
</pre>

'''C35GMIN''' (GMIN with CHARMM 35) using pgi:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/pgi_charmm35
cd !$
FC=pgf90 cmake -DWITH_CHARMM35=1 ../../source
make -j8
</pre>

'''CUDAGMIN''' (GMIN leveraging GPU minimisation via the AMBER 12 interface) using ifort:
<pre>
module load cuda/5.5
mkdir -p ~/softwarewales/GMIN/builds/ifort_cuda
cd !$
FC=ifort cmake -DWITH_CUDA=1 ../../source
make -j8
</pre>
This will only work on machines with specific NVIDIA GPUs, for example when submitting jobs on the pat cluster. There is some additional information on the [[Using GMIN with GPUs]] page.

'''DFTBGMIN''' (GMIN with DFTBP) using Intel on nest:
<pre>
module load mkl/64/2022/0/0 cmake/3.23.2 ifort/64/2020/4/304
mkdir -p ~/softwarewales/GMIN/builds/ifort_dftbp
cd !$
FC=ifort CC=icc CXX=icc cmake ../../source -DWITH_DFTBP=yes
make -j8
</pre>

or using GCC on nest:
<pre>
module load cmake/3.23.2 gcc/12.2.0
mkdir -p ~/softwarewales/GMIN/builds/gfortran_dftbp
cd !$
cmake ../../source -DWITH_DFTBP=yes -DWITH_MYBLAS=no -DBLAS_LIBRARIES=/usr/lib64/libopenblas.so.0
make -j8
</pre>

===OPTIM===
'''A9OPTIM''' (OPTIM with AMBER9) using ifort:
<pre>
mkdir -p ~/softwarewales/OPTIM/builds/ifort_amber
cd !$
FC=ifort cmake -DWITH_AMBER9=1 ../../source
make -j8
</pre>

'''C35OPTIM''' (OPTIM with CHARMM 35) using pgi:
<pre>
mkdir -p ~/softwarewales/OPTIM/builds/pgi_charmm35
cd !$
FC=pgf90 cmake -DWITH_CHARMM35=1 ../../source
make -j8
</pre>

'''CUDAOPTIM''' (OPTIM leveraging GPU via the AMBER 12 interface) using ifort:
<pre>
module load cuda/5.5
mkdir -p ~/softwarewales/OPTIM/builds/ifort_cuda5.5
cd !$
FC=ifort CC=icc cmake -DWITH_CUDA=1 ../../source
make -j8
</pre>
This will only work on machines with specific NVIDIA GPUs, for example when submitting jobs on the pat cluster. There is some additional information on the [[Using GMIN and OPTIM with GPUs]] page.

'''DFTBOPTIM''' (OPTIM with DFTBP) using Intel on nest:
<pre>
module load mkl/64/2022/0/0 cmake/3.23.2 ifort/64/2020/4/304
mkdir -p ~/softwarewales/OPTIM/builds/ifort_dftbp
cd !$
FC=ifort CC=icc CXX=icc cmake ../../source -DWITH_DFTBP=yes
make -j8
</pre>

or using GCC on nest:
<pre>
module load cmake/3.23.2 gcc/12.2.0
mkdir -p ~/softwarewales/OPTIM/builds/gfortran_dftbp
cd !$
cmake ../../source -DWITH_DFTBP=yes -DWITH_MYBLAS=no -DBLAS_LIBRARIES=/usr/lib64/libopenblas.so.0
make -j8
</pre>

===PATHSAMPLE===
There are very few options for [[PATHSAMPLE]] as we don't need to worry about interfacing with a particular potential. As a result, every binary is simply called ''PATHSAMPLE''.

Using nagfor (the NAG fortran compiler - check you have the module loaded - very strict!):
<pre>
mkdir -p ~/softwarewales/PATHSAMPLE/builds/nagfor
cd !$
FC=nagfor cmake ../../source
make -j8
</pre>

Using pgi (much more generous with coding slips/non-standard uses):
<pre>
mkdir -p ~/softwarewales/PATHSAMPLE/builds/pgi
cd !$
FC=pgf90 cmake ../../source
make -j8
</pre>

==Configuring defaults - for developers==

Fortran compilers and their corresponding default settings are all controlled by the file $SVN/CMakeModules/FindFORTRANCOMPILER.cmake ($SVN is your svn root directory). In particular, we may wish to edit the flags used for each set of compilers and build type. These are contained in the following block:

<pre>
if(NOT COMPILER_FLAGS_WERE_SET)
message("Setting initial values for compiler flags")
if(COMPILER_SWITCH MATCHES "pgi")
set (CMAKE_Fortran_FLAGS "-Mextend" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-O3 -Munroll -Mnoframe" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG "-Mextend -C -g -gopt -Mbounds -Mchkfpstk -Mchkptr -Mchkstk -Mcoff -Mdwarf1 -Mdwarf2 -Melf -Mpgicoff -traceback" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-Mextend -C -g -gopt -Mbounds -Mchkfpstk -Mchkptr -Mchkstk -Mcoff -Mdwarf1 -Mdwarf2 -Mdwarf3 -Melf -Mpgicoff -traceback" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-Mfree" CACHE TYPE STRING)
elseif(COMPILER_SWITCH MATCHES "gfortran")
set (CMAKE_Fortran_FLAGS "-ffixed-line-length-200 -ffree-line-length-0" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-O3" CACHE TYPE STRING FORCE)
# set (CMAKE_Fortran_FLAGS_DEBUG "-g -fbounds-check -Wuninitialized -O -ftrapv -fimplicit-none -fno-automatic" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG "-g -fbounds-check -Wuninitialized -O -ftrapv -fno-automatic" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "${CMAKE_Fortran_FLAGS_DEBUG} -fimplicit-none" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-ffree-form" CACHE TYPE STRING)
elseif(COMPILER_SWITCH MATCHES "nag")
set (CMAKE_Fortran_FLAGS "-132 -kind=byte -maxcontin=3000" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-mismatch_all -O4" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG "-g -mismatch_all -ieee=stop" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-C=all -mtrace=all -gline -g -mismatch_all -ieee=stop" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-free" CACHE TYPE STRING) # js850> is this ever used?
elseif(COMPILER_SWITCH MATCHES "ifort")
set (CMAKE_Fortran_FLAGS "-132 -heap-arrays -assume byterecl" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-O3" CACHE TYPE STRING FORCE)
# Warnings about temporary argument creation and edit descriptor widths are disabled with the final flags.
set (CMAKE_Fortran_FLAGS_DEBUG "-C -g -traceback -debug full -check all,noarg_temp_created -diag-disable 8290,8291" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-debug all -check all -implicitnone -warn unused -fp-stack-check -ftrapuv -check pointers -check bounds" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-free" CACHE TYPE STRING)
else()
message(FATAL_ERROR "unknown comiler switch: ${COMPILER_SWITCH}")
endif()
SET(COMPILER_FLAGS_WERE_SET yes CACHE TYPE INTERNAL)
endif(NOT COMPILER_FLAGS_WERE_SET)
</pre>

The main if/elseif blocks correspond to compiler switches. Inside these, there are the default flags for each of our build types (release, debug and debug_slow), which are configured using ccmake. These can be edited, if we wish to change the default behaviour (e.g. a recent addition of -check all,noarg_temp_created -diag-disable 8290,8291 to disable annoying warning messages for ifort).

Compiling Wales Group codes using cmake

2022-09-09T14:01:51Z

Jwrm2: /* GMIN */ Added DFTBP examples.

[http://www.cmake.org/ CMake] (Cross-platform Make) provides a simple, platform independent way for us to compile and test the group codebase. Dependencies are handled automatically, compilation can proceed in parallel to avoid long waits while testing changes and builds are done entirely outside of the source directory. It also enables us to use the [[Jenkins CI]] 'build bot' system to automatically compile and test the code on a nightly basis - helping us catch troublesome commits before they affect other users.

Although everything below refers to compiling [[GMIN]] with the Intel ''ifort'' compiler and AMBER9 - the exact same procedure works for [[OPTIM]] and [[PATHSAMPLE]].

Note that not every option for our codes is expected to actually compile with every compiler, for example, anything using CHARMM35/36 will not compile with ''nagfor'' or ''gfortran''. This is nothing to do with our code - it's a CHARMM issue. You can get an idea for what should work by looking at the automated [[Jenkins CI]] builds.

==Preparing to compile==
Before you get started, you need to ensure that the machine you are planning to compile on has cmake 2.8 or higher installed. You can check the current version like so:
<pre>
cmake --version
</pre>

The clusters have a module for cmake 3.0, which you can load using the following command:
<pre>
module load cmake/3.0.0
</pre>

You also need to create a directory to build the code in. We suggest that you create a directory for the compiler you are using within the program directory, under a subdirectory called 'builds' - for example for compiling GMIN with ifort, you would make a directory here:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/ifort
cd ~/softwarewales/GMIN/builds/ifort
</pre>
You can call these directories whatever you like - but make sure it is clear to you what they contain! You might also want to check which version of the compiler you have loaded. This is important as the different clusters and workstations may have different default versions loaded, some of which might not work properly. You can check the compiler version currently loaded using the same '--version' flag we used for ''cmake'' above:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ifort --version
ifort (IFORT) 12.1.3 20120212
Copyright (C) 1985-2012 Intel Corporation. All rights reserved.
</pre>

To load a different compiler, you can use the ''module load'' or ''module swap'' commands. A list of all available modules can be accessed using:
<pre>
module av
</pre>
If you are having problems compiling, one of the first things to check is whether it works with a different version of the compiler!

'''Note: When compiling GMIN, if you are getting the error that there is no implicit type for ERFC in ewald.f90, try using a newer version of your compiler. This should be the built-in complementary error function.'''

==Compiling using the ccmake GUI interface to set options==
[[Image:Ccmake.png|thumb|ccmake set up to compile A9GMIN|200px|right]]

One advantage using cmake has over make is that we can use the simple ccmake GUI. This interface lets us set options like compiling with AMBER9, or CHARMM35, toggle between 'Release' and 'Debug' builds (see below) - and examine and alter the flags being uses for the compilation if we wish. Before we can run ccmake, we need to specify the compiler and run cmake in our build directory (e.g. softwarewales/GMIN/builds/ifort). We specify the '''F'''ortran '''C'''ompiler by setting the '''$FC''' environment variable (in this case the Intel Fortran compiler, ifort), and then run ''cmake'' (on the command line), passing it the relative location of the [[GMIN]] source directory:
<pre>
FC=ifort cmake ../../source
</pre>

If you run ''ls'', you will see some cmake files have been generated:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ls
CMakeCache.txt CMakeFiles cmake_install.cmake Makefile modules
</pre>

You can now run ''ccmake'' to open the GUI:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ccmake .
</pre>

To navigate between options, use the arrow keys. Options can be toggled by pressing Return. To compile [[GMIN]] with AMBER9 (A9GMIN), we need to toggle the ''WITH_AMBER'' option ''ON''. Once you have done this, you need to configure and generate appropriate cmake info. This is done by pressing 'c' to configure, 'e' to exit and then 'g' to generate.

'''Note: for some builds (CHARMM with DFTB and CUDAGMIN), you might need to configure, exit and generate twice to set all necessary options'''

You can now compile A9GMIN in parallel as follows:
<pre>
make -j8
</pre>

The '-j8' flag here tells make to use up to 8 'threads' when building. For optimal performance, you should keep this slightly greater than the number of cores (CPUs) the node you are working on has. If all goes well, you should now have an A9GMIN binary in your build directory - congratulations!
<pre>
Linking Fortran executable A9GMIN
[100%] Built target A9GMIN

--------------------------------------------------------------------------------------------- 15:23:45

csw34@sinister:~/softwarewales/GMIN/builds/ifort> ls
A9GMIN cmake_install.cmake libcudadummylib.a libmylapack.a NAB
AMBER display_version.f90 libdummylib.a Makefile nab_binaries_built
CMakeCache.txt GMIN libgminlib.a modules porfuncs.f90
CMakeFiles libamber12dummylib.a libmyblas.a n
</pre>
Plain [[GMIN]] is also built at the same time should you need it. You can move this into your ~/bin directory if you like, or anywhere else in your ''$PATH'' to make running it simple.

'''Note: If you want to use OPTIM with the new C++ implementation of the NEB routine, you will need to obtain the source code for that separately. See [https://wikis.ch.cam.ac.uk/wales/wiki/index.php/OPTIM here] for instructions.'''

==Compiling by setting options on the command line==
If you know the options you'd like to set already (you can see them all in ccmake), you can save some time by passing them directly to ''cmake'' on the command line, bypassing the need for ''ccmake''. For example, to compile A9GMIN (GMIN with the AMBER9 interface) using the Intel ifort compiler, you would run the following commands:
<pre>
FC=ifort cmake -DWITH_AMBER=1 ../../source
make -j8
</pre>
where '../../source' is the relative location of the GMIN source directory. You can find some more examples of compiling from the command line below.

'''Note: Sometimes you may get error''' (for example, Fatal Error: Can't open module file 'someModule.mod' for reading at (1): No such file or directory) when following this procedure. In that case there are three things you could try: make sure you are building in a new directory, if that does not help run `make VERBOSE=1` instead of `make -j8` or simply switch to using ccmake.

==Compiling with MPI==
To compile with MPI support add the following flags when running cmake on the command line:
<pre>
FC=mpif90 CC=mpicc cmake ../source -DCOMPILER_SWITCH=pgi -DWITH_MPI=yes
</pre>
Here -DCOMPILER_SWITCH=pgi assumes you're using the Portland ''pgi'' compiler. Make sure you have the correct modules loaded (in this case ''pgi'' and ''mpi-pgi''), and that the particular mpi you want (in this case ''mpi-pgi'') is listed before any other mpi's loaded (so that it has the highest priority). The modules can be loaded by typing:
<pre>
module load pgi/64/
module load mpi/openmpi/pgi/64/1.6.3
</pre>

and you can check which modules are loaded and in which order/priority by the ''module list'' command. You may need to ''module unload <name>'' any other mpi's that are higher up in the list than the one you want. You can of course set the COMPILER_SWITCH and WITH_MPI flags in ''ccmake'' if you prefer.

Note: It has been observed that pgi/64/15.1 leads to compilation errors, and for now, it is best to use pgi/64/14.9

==Advanced mode - changing compiler flags with ccmake==
[[Image:Ccmakeadvanced.png|thumb|ccmake advanced mode|200px|right]]

Although initially the ''ccmake'' GUI looks very simple, there is a lot going on under the hood. By pressing 't' you can enter 'Advanced mode' which will show you all of the hidden options, for example the compiler flags that are being passed to ''make'' when you compile the code. You can also make changes to the flags here, for example if you would like to add '-p' to do profiling.

As with changing the build type, you simply select the field you'd like to change using the arrow keys, press Return, make your changes and press Return again to save them. When you subsequently configure and generate as above, those altered flag will be used for the subsequent compilation.

Note that these changes only apply in the build directory in which you make them.

==Debugging runtime problems using gdb or valgrind==
If you are getting a segmentation fault, crash or other unexpected behaviour, you might want to run your job through a debugger like [http://www.gnu.org/software/gdb/ gdb] or [http://valgrind.org/ valgrind]. In order to maximise your chances of getting useful output, you should build a 'Debug' version of the program you are having trouble. To do this, you can either change the ''CMAKE_BUILD_TYPE'' in ''ccmake'' to 'Debug' (press Return, change 'Release' to 'Debug' and press Return again), or on the command line like so for GMIN with AMBER 9 using the Intel ifort compiler:
<pre>
FC=ifort cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_AMBER=1 ../../source
make -j8
</pre>

You can then run the binary ''through'' gdb or valgrind as follows:
<pre>
gdb A9GMIN
</pre>
or
<pre>
valgrind A9GMIN
</pre>

I won't cover debugging with these tools here as it's a science in itself! Do some Googling and ask for help as needed :)

==Debugging compilation problems==
There are many ways to try and track down why your code is not compiling. Before you start changing compilers, building a 'Debug' version or changing machines, you might want to try running make again with the ''VERBOSE'' option enabled. This will dump a lot of potentially useful output:
<pre>
VERBOSE=1 make
</pre>

One possible gotcha: all .f and .f90 files in the relevant source directories will be compiled and added to a library. This is quite different from the old Makefile way of doing things, where source files were explicitly specified for compilation (via their corresponding .o file). So, if you are testing something by for instance copying code.f90 to code.myhack.f90 and code.orig.f90, then slightly editing a line or two of code.myhack.f90 and copying it back to code.f90 for use, this will probably cause linking problems due to multiply-defined subroutines (from all three files). The solution, if you must have alternative versions of the same file hanging round, is to differentiate the filenames by a suffix AFTER the .f[90] .

Another occasional issue is the unexplained compiler bug - a problem with the version of the compiler you happen to be using. You can can an idea for which compiler versions we expect to work by checking the Jenkins build-bot output, as described in the 'Seeing console output' section of the [[Jenkins CI]] page. If you are using a different version of the compiler in question, consider swapping to the version Jenkins is using with 'module swap'.

If the error message you are getting doesn't make sense to you after some Googling, go and ask someone - we all have these problems. Things you can try first include trying a different compiler version, or an entirely different compiler e.g. pgi rather than ifort for example. You should bear in mind that as mentioned above, not all versions of each code will compile with every compiler. Make sure you're not trying to build something that isn't expected to work.

==Extra command line build examples==
The below commands are absolutely not an exhaustive list, but should give you an idea of what is possible. You can use ''ccmake'' as described above to discover which variables (e.g. WITH_AMBER) can be manipulated on the command line like this. All of these examples assume your svn repository is set up in ''/home/CRSID/svn'' - make the appropriate modifications if you have it elsewhere.

===GMIN===
'''A12GMIN''' (GMIN with AMBER12) using ifort:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/ifort_amber12
cd !$
FC=ifort cmake -DWITH_AMBER12=1 ../../source
make -j8
</pre>

'''C35GMIN''' (GMIN with CHARMM 35) using pgi:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/pgi_charmm35
cd !$
FC=pgf90 cmake -DWITH_CHARMM35=1 ../../source
make -j8
</pre>

'''CUDAGMIN''' (GMIN leveraging GPU minimisation via the AMBER 12 interface) using ifort:
<pre>
module load cuda/5.5
mkdir -p ~/softwarewales/GMIN/builds/ifort_cuda
cd !$
FC=ifort cmake -DWITH_CUDA=1 ../../source
make -j8
</pre>
This will only work on machines with specific NVIDIA GPUs, for example when submitting jobs on the pat cluster. There is some additional information on the [[Using GMIN with GPUs]] page.

'''DFTBGMIN''' (GMIN with DFTBP) using Intel on nest:
<pre>
module load mkl/64/2022/0/0 cmake/3.23.2 ifort/64/2020/4/304
mkdir -p ~/softwarewales/GMIN/builds/ifort_dftbp
cd !$
FC=ifort CC=icc CXX=icc cmake ../../source -DWITH_DFTBP=yes
make -j8
</pre>

or using GCC on nest:
<pre>
module load cmake/3.23.2 gcc/12.2.0
mkdir -p ~/softwarewales/GMIN/builds/gfortran_dftbp
cd !$
cmake ../../source -DWITH_DFTBP=yes -DWITH_MYBLAS=no -DBLAS_LIBRARIES=/usr/lib64/libopenblas.so.0
make -j8
</pre>

===OPTIM===
'''A9OPTIM''' (OPTIM with AMBER9) using ifort:
<pre>
mkdir -p ~/softwarewales/OPTIM/builds/ifort_amber
cd !$
FC=ifort cmake -DWITH_AMBER9=1 ../../source
make -j8
</pre>

'''C35OPTIM''' (OPTIM with CHARMM 35) using pgi:
<pre>
mkdir -p ~/softwarewales/OPTIM/builds/pgi_charmm35
cd !$
FC=pgf90 cmake -DWITH_CHARMM35=1 ../../source
make -j8
</pre>

'''CUDAOPTIM''' (OPTIM leveraging GPU via the AMBER 12 interface) using ifort:
<pre>
module load cuda/5.5
mkdir -p ~/softwarewales/OPTIM/builds/ifort_cuda5.5
cd !$
FC=ifort CC=icc cmake -DWITH_CUDA=1 ../../source
make -j8
</pre>
This will only work on machines with specific NVIDIA GPUs, for example when submitting jobs on the pat cluster. There is some additional information on the [[Using GMIN and OPTIM with GPUs]] page.

===PATHSAMPLE===
There are very few options for [[PATHSAMPLE]] as we don't need to worry about interfacing with a particular potential. As a result, every binary is simply called ''PATHSAMPLE''.

Using nagfor (the NAG fortran compiler - check you have the module loaded - very strict!):
<pre>
mkdir -p ~/softwarewales/PATHSAMPLE/builds/nagfor
cd !$
FC=nagfor cmake ../../source
make -j8
</pre>

Using pgi (much more generous with coding slips/non-standard uses):
<pre>
mkdir -p ~/softwarewales/PATHSAMPLE/builds/pgi
cd !$
FC=pgf90 cmake ../../source
make -j8
</pre>

==Configuring defaults - for developers==

Fortran compilers and their corresponding default settings are all controlled by the file $SVN/CMakeModules/FindFORTRANCOMPILER.cmake ($SVN is your svn root directory). In particular, we may wish to edit the flags used for each set of compilers and build type. These are contained in the following block:

<pre>
if(NOT COMPILER_FLAGS_WERE_SET)
message("Setting initial values for compiler flags")
if(COMPILER_SWITCH MATCHES "pgi")
set (CMAKE_Fortran_FLAGS "-Mextend" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-O3 -Munroll -Mnoframe" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG "-Mextend -C -g -gopt -Mbounds -Mchkfpstk -Mchkptr -Mchkstk -Mcoff -Mdwarf1 -Mdwarf2 -Melf -Mpgicoff -traceback" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-Mextend -C -g -gopt -Mbounds -Mchkfpstk -Mchkptr -Mchkstk -Mcoff -Mdwarf1 -Mdwarf2 -Mdwarf3 -Melf -Mpgicoff -traceback" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-Mfree" CACHE TYPE STRING)
elseif(COMPILER_SWITCH MATCHES "gfortran")
set (CMAKE_Fortran_FLAGS "-ffixed-line-length-200 -ffree-line-length-0" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-O3" CACHE TYPE STRING FORCE)
# set (CMAKE_Fortran_FLAGS_DEBUG "-g -fbounds-check -Wuninitialized -O -ftrapv -fimplicit-none -fno-automatic" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG "-g -fbounds-check -Wuninitialized -O -ftrapv -fno-automatic" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "${CMAKE_Fortran_FLAGS_DEBUG} -fimplicit-none" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-ffree-form" CACHE TYPE STRING)
elseif(COMPILER_SWITCH MATCHES "nag")
set (CMAKE_Fortran_FLAGS "-132 -kind=byte -maxcontin=3000" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-mismatch_all -O4" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG "-g -mismatch_all -ieee=stop" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-C=all -mtrace=all -gline -g -mismatch_all -ieee=stop" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-free" CACHE TYPE STRING) # js850> is this ever used?
elseif(COMPILER_SWITCH MATCHES "ifort")
set (CMAKE_Fortran_FLAGS "-132 -heap-arrays -assume byterecl" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_RELEASE "-O3" CACHE TYPE STRING FORCE)
# Warnings about temporary argument creation and edit descriptor widths are disabled with the final flags.
set (CMAKE_Fortran_FLAGS_DEBUG "-C -g -traceback -debug full -check all,noarg_temp_created -diag-disable 8290,8291" CACHE TYPE STRING FORCE)
set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-debug all -check all -implicitnone -warn unused -fp-stack-check -ftrapuv -check pointers -check bounds" CACHE TYPE STRING FORCE)
set (FORTRAN_FREEFORM_FLAG "-free" CACHE TYPE STRING)
else()
message(FATAL_ERROR "unknown comiler switch: ${COMPILER_SWITCH}")
endif()
SET(COMPILER_FLAGS_WERE_SET yes CACHE TYPE INTERNAL)
endif(NOT COMPILER_FLAGS_WERE_SET)
</pre>

The main if/elseif blocks correspond to compiler switches. Inside these, there are the default flags for each of our build types (release, debug and debug_slow), which are configured using ccmake. These can be edited, if we wish to change the default behaviour (e.g. a recent addition of -check all,noarg_temp_created -diag-disable 8290,8291 to disable annoying warning messages for ifort).

Git Workflow

2022-07-05T11:32:24Z

Jwrm2: /* Useful commands to know */ Added local branch cleaning.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. If you get the error message

Error: failed to call git rev-parse --git-dir: exit status 128 : fatal: .git

then try

$ git lfs install --skip-repo

instead. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/paperswales papers]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/paperswales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add. You will need to add them individually however, with
$ git add -f filename
and cannot add all files at the same time (i.e. using git add -A).

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You can also choose who to assign it to. Make sure you tick the boxes to delete the feature branch after the request is accepted, and to squash your branch into one commit. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you assign to will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

$ git fetch -p && for branch in $(git branch -vv | grep ': gone]' | awk '{print $1}'); do git branch -D $branch; done

Delete all local branches that do not exist on GitLab. This command is useful to periodically clean up local branches after they have been merged and deleted on GitLab. Warning: do not run this command if you've created a new local branch and not yet pushed it to GitLab.

Git Workflow

2022-03-22T13:24:23Z

Jwrm2: Tidied up LFS installation information.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. If you get the error message

Error: failed to call git rev-parse --git-dir: exit status 128 : fatal: .git

then try

$ git lfs install --skip-repo

instead. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/paperswales papers]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/paperswales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add. You will need to add them individually however, with
$ git add -f filename
and cannot add all files at the same time (i.e. using git add -A).

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You can also choose who to assign it to. Make sure you tick the boxes to delete the feature branch after the request is accepted, and to squash your branch into one commit. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you assign to will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

Git Workflow

2021-04-30T10:40:11Z

Jwrm2: /* Basic Workflow */ Added information on git stash.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/paperswales papers]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/paperswales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every commit you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If other people have changed files you are working on, the pull will fail with an informative message. In this case, run

$ git stash

which sets aside your local changes. Try the pull again, which should now succeed. Then run

$ git stash pop

to reapply your local changes to the updated repository. The merge will usually happen automatically, but sometimes you will need to resolve conflicts yourself.

Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add. You will need to add them individually however, with
$ git add -f filename
and cannot add all files at the same time (i.e. using git add -A).

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You can also choose who to assign it to. Make sure you tick the boxes to delete the feature branch after the request is accepted, and to squash your branch into one commit. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you assign to will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

Git Workflow

2021-04-30T10:37:25Z

Jwrm2: /* Writing a Paper */ Slightly adjusted example workflow.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/paperswales papers]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/paperswales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every push you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If you need to merge with files you are working on, do a commit (but not a push) first, then a pull. Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git pull
# git add all the edited files
# git commit with a helpful message
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add. You will need to add them individually however, with
$ git add -f filename
and cannot add all files at the same time (i.e. using git add -A).

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You can also choose who to assign it to. Make sure you tick the boxes to delete the feature branch after the request is accepted, and to squash your branch into one commit. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you assign to will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

Git Workflow

2021-03-25T16:23:45Z

Jwrm2: /* Useful commands to know */ Corrected git clean

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/paperswales papers]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/paperswales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every push you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If you need to merge with files you are working on, do a commit (but not a push) first, then a pull. Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git add all the edited files
# git commit with a helpful message
# git pull
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add.

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You can also choose who to assign it to. Make sure you tick the boxes to delete the feature branch after the request is accepted, and to squash your branch into one commit. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you assign to will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean -f

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

Git Workflow

2021-03-25T16:23:28Z

Jwrm2: /* Useful commands to know */ Added git clean.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/paperswales papers]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/paperswales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every push you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If you need to merge with files you are working on, do a commit (but not a push) first, then a pull. Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git add all the edited files
# git commit with a helpful message
# git pull
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add.

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You can also choose who to assign it to. Make sure you tick the boxes to delete the feature branch after the request is accepted, and to squash your branch into one commit. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you assign to will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

$ git clean

Throw away all untracked files. They will be deleted. Run with -n rather than -f to see which files would be deleted, but without actually doing anything.

Git Workflow

2021-03-23T16:05:53Z

Jwrm2: /* Writing a Paper */ Leave out LaTeX intermediate files.

Group software and papers in the process of being written are stored on the University's GitLab repositories [https://gitlab.developers.cam.ac.uk/]. You should be able to log in via Raven, but someone with privileges will need to add you to the two projects. This page describes a typical workflow for retrieving, modifying and updating the repositories. It is not, however, a comprehensive guide to Git. For that, consult your favourite web search engine.

==Setting up SSH access==

To smoothly access Gitlab without having to type your user name and password the whole time, set up an SSH key. In a terminal on your desktop, type

$ ssh-keygen -t ed25519 -C "GitLab"

If it complains about overwriting, then you have already done this step and you probably don't want to overwrite. It will ask you where to save the key: the default location should be fine. Now you are prompted for a passphrase. You can press ENTER to leave it blank, although that does mean that anyone who breaks into your computer can get access to GitLab with no further effort. You now have a file (default location is ~/.ssh/id_ed25519.pub) that contains your public key. Copy the entire contents of this file.

On the GitLab website, click on your user icon in the top right and select 'Settings' and then 'SSH Keys' from the left menu. Paste the contents of your public key file into the box. Put something useful in the title, like the name of your desktop (yes, you should probably do this for each machine you want to access GitLab from, rather than copying keys between machines). Optionally, you can insert an expiry date, such as the date your funding runs out. Click the 'Add key' button.

To test that you now have access, in a terminal type

$ ssh -T git@gitlab.developers.cam.ac.uk

After accepting the RSA identity, you should see a welcome message and then the connection will close.

Some users have reported problems with using ed25519. RSA is available as an alternative. Generate an RSA key with

$ ssh-keygen -t rsa -C "GitLab"

which will be saved by default at ~/.ssh/id_rsa.pub. Follow the same steps as above to add your public key to Gitlab.

==Installing git LFS==

Our software repository (git@gitlab.developers.cam.ac.uk:ch/wales/softwarewales.git) uses git Large File Storage (LFS) to manage some of the larger files. You must have the git LFS addon installed and initialised before cloning the software repository. If you do not, the clone will appear to succeed, but you will be missing some files. If you are using a department managed workstation, or any cluster other than rogue or nest, you will need to load a newer version of git:

$ module load git/2.0.0

Replace 2.0.0 with whatever the newest available version of git is. Or, on your personal Ubuntu machine, run

$ sudo apt-get install git-lfs

to install the necessary package. Whatever computer you are using, then run

$ git lfs install

to inform git about the new LFS addon. This command only needs to be run once on each machine you intend to clone the software repository on. After you have cloned the repository, you should inspect the LFS files to make sure the clone worked correctly. You can see a list of the files in .gitattributes in the repository root directory. A good file to check might be THESES/PHD/ChrisWhittlestonPhD.pdf. If this file is a PDF of several megabytes, then the LFS succeeded. If it is a small plaintext file containing a URL, the LFS clone did not succeed.

If you have already cloned the repository before installing the LFS addon, you will need to clone again (a pull will not suffice).

==Initial Checkout==

You need to fetch the repository. In Git terms, this is called cloning. You should only need to carry out this step once for each repository. In your favourite web browser, navigate to the project page on Gitlab, eg. [https://gitlab.developers.cam.ac.uk/ch/wales/paperswales papers]. Spot the blue button on the right labelled 'Clone' and click on it. Copy the link under 'Clone with SSH' to the clipboard (don't use the 'clone with HTTPS' link, or you will have to type your username and password every time). In a terminal, choose a suitable location, like your home directory and change to there. Now type

$ git clone git@gitlab.developers.cam.ac.uk:ch/wales/paperswales.git

replacing the address with what you just copied. Git will download the repository. Once it has finished, check that you now have lots of new directories with the contents of the repository.

You should also tell Git your name and email address. Git will record these in the commit logs so other users will know who to complain to when a commit breaks everything. Run

$ git config --global user.name "An Other"
$ git config --global user.email "ao123@cam.ac.uk"

replacing the name and email address in quotes as appropriate.

==Basic Workflow==

Details for specific cases are below, but first, we mention the most important commands that you'll be running all the time. Imagine you've just arrived in the morning and it's time to start working on myfile.f90. The first command to run is

$ git pull

This command contacts the remote repository on GitLab and fetches any commits that people may have made. Run this command frequently, and at least before every push you make. One notable difference from updating in svn is that git will not merge other people's changes with files you have changed since your last commit. If you need to merge with files you are working on, do a commit (but not a push) first, then a pull. Now you edit myfile.90 and want to commit your changes. Run

$ git add myfile.f90

Now the file is, in Git terminology, staged for commit. You haven't committed anything yet. You can add other files to the staging area too. Once your commit is ready, run

$ git commit -m "Informative message."

Replace 'Informative message' with a brief message describing what changes are in your commit. At this point, you have updated your local repository and entered a commit in the permanent record. However, the commit hasn't gone to GitLab yet. To send it to GitLab (called the remote by Git), run

$ git push

You can send multiple commits at once. This workflow should encourage you to commit often. Maybe you write a new function. Put in a commit. Then you add some stuff to keywords.f90 for the new functionality. Do another commit. Next you find a bug and fix it. Do another commit.

Note that the remote can function as the backup of your work. Therefore you should probably push any new commits at least as often as the end of each day.

==Writing a Paper==

Writing a paper is slightly simpler than editing the group code (Discuss...) because we aren't worrying about multiple branches. Create a new directory for your paper and use git add to add it. Start writing the paper in the new directory. Each session of editing should involve

# git pull
# make some edits
# git add all the edited files
# git commit with a helpful message
# git pull
# git push

Simples. All authors will be editing the same branch (the master branch), so you'll see other authors' updates straight away. This approach keeps things easy, but if two authors are working at exactly the same time, there may be some merging to do. Reduce the amount of merging by committing, pulling and pushing often.

Do not add intermediate LaTeX files (.aux, .log, etc.) to the repository. Do not add your .dvi/.ps/.pdf document either. These extensions are all specified in the .gitignore file for the papers repository, so they will not show up with git status. If you are using .pdf or .ps for your images, you should still be able to add them with git add.

==Working on the group code==

You've just been talking to David and you've come up with an exciting new feature to add to GMIN. It's going to take several days of coding, during which you'll want to back up your work on the remote, but you don't want to interfere with other people using GMIN. The solution is to create a new branch. A branch is your own version controlled copy of the code that you can edit at will without messing GMIN up for anyone else. All development should occur on branches. To create a new branch, run

$ git checkout -b exciting_feature

This command both creates the branch and switches your working copy to it. Initially, your new branch is the same as the master branch you cut it from. However, the branch does not yet exist on GitLab. To create it, run

$ git push --set-upstream origin exciting_feature

Now go ahead and edit files, making commits and pushing them to GitLab frequently.

When your feature is complete and you have checked it works and that you haven't broken anything else, it's time to get it into the master branch. Several steps are required. Firstly, it's quite likely that other people have changed master since you cut off your branch. You need to test that your changes function with the new changes to master, so first you need to merge in master:

$ git checkout master
$ git pull
$ git checkout exciting_feature
$ git merge master
$ git push

The pull commands makes sure that your local copy of the repository is up to date. The merge command merges changes that have been made to the master branch to your branch. It creates new commits, that you then push to the remote of your branch.

Most users do not have permission to edit the master branch. To get your new feature in, you have to create a merge request. Go to the project page on GitLab. From the drop down menu of branches, select exciting_feature. Click the blue 'Create merge request' button in the top right. You now have a page in which you can give your merge request a title and description. You can also choose who to assign it to. Make sure you tick the boxes to delete the feature branch after the request is accepted, and to squash your branch into one commit. These options help keep the remote repository and history clean. You can edit the commit message for the one commit that will be created: by default it will be the name of your branch. The person you assign to will get a notification and a copy of your changes. They will look through your changes to make sure they follow the group coding standards ([https://wikis.ch.cam.ac.uk/wales/wiki/index.php/Wales_Group_Fortran_conventions_for_group_software here]) and that you haven't broken anything. If there are any issues, they may request that you make some changes, which you can then commit to the exciting_feature branch. The pull request will be updated and the reviewer will get a notification. Once your code has passed the review, the reviewer will click the Merge button and your branch will be merged into master. Just 'Approve' from the reviewer won't usually be enough because you probably don't have permission to write to Master and hence cannot merge your new branch into master yourself. Once it has been merged, you can clean up your repository with the following commands

$ git checkout master
$ git pull
$ git branch -d exciting_feature

These commands switch your working copy back to master, update your local copy of the repository, and delete the branch you made. You might like to check that your new feature is in the master branch files before deleting your branch. If something has gone wrong and you delete your branch before the changes are in master, it is possible to recover, as the commits won't actually be deleted from the remote for a few weeks, but the recovery is an advanced topic that is best avoided.

===Large Files===

If any new file you are adding is large (>10MB), it should be stored on git LFS, rather than as a normal file. Fortunately, this is easy to do for new files. If you have already committed a large file and would now like to change it, you have a very complicated process ahead. The best instructions the author could find when doing this in the initial repository migration were here https://stackoverflow.com/questions/60995429/gradually-converting-a-repo-to-use-git-lfs-for-certain-files-one-file-at-a-time in the question, with the caveat that the bfg utility does not work and it was necessary to use git filter-branch as described here https://dalibornasevic.com/posts/2-permanently-remove-files-and-folders-from-a-git-repository instead. Note this process will delete the history of your existing file and it will only appear from the most recent commit. It may be possible to adjust this with a git rebase, but the author has not investigated.

Anyway, if you haven't yet committed your large file, you simply need to run

$ git lfs track "<path-to-file>"

which informs git that this file is to be uploaded using LFS. This command edits the file .gitattributes, which will also need to be added to your commit. If you make a mistake, you can edit .gitattributes manually. You can see some examples as well as the current list of files that are uploaded with LFS in .gitattributes. As you can see from inspecting the file, it is also possible to use wildcards ('*') to specify multiple files at once. Be careful with your rules though: the rule is applied over all files, so if you add a rule for 'myfile.f90' and somewhere else in the repository there is another file with the same name, it will now also be uploaded with LFS. This can be useful for specifying, for example, all .mp4 files in the whole repository, with "*.mp4". However, if you really want just a specific file, specify the path from the repository root, for example "OPTIM/source/myfile.f90".

==Useful commands to know==

$ git status

At any point, this command will show you what branch you are on, what files you have modified and staged and your local position compared to the remote. Use it often.

$ git branch

Display a list of all the current branches.

$ git diff

Show the differences between your working copy and the last commit, for all files. Add a file name to show only the differences for a specific file.

$ git log

Display the commit history. Add a file or directory name afterwards to only show the commits that affected that file, or any file in the directory.

$ git reset HEAD myfile.f90

Unstage myfile.f90 that you accidentally staged for the next commit, but actually don't want to commit just yet. The working copy of the file is not altered.

$ git checkout -- myfile.f90

Revert myfile.f90 that you've completely messed up to what it was at the last commit. Changes to your working copy are lost.

$ git reset --hard

Throw away all working and staged changes, reverting the current state to the last commit.

$ git reset --hard 909a3cac63ae8782b258ebb8c27af361b555bff6

Throw away all working and staged changes, reverting the current state to that of the commit specified. The long hex number is a commit hash. It is not human readable, but you can copy the relevant one from the commit log.

Git Workflow

2021-02-12T12:56:44Z

Jwrm2: /* Initial Checkout */ Added information on setting name and email address.

Git Workflow

2021-02-03T14:29:26Z

Jwrm2: /* Working on the group code */ Updates to merging workflow.

Git Workflow

2021-01-19T11:06:11Z

Jwrm2: /* Installing git LFS */ Updated how to check whether the LFS clone succeeded.

Git Workflow

2021-01-04T13:32:26Z

Jwrm2: /* Installing git LFS */ Added module load information.

Git Workflow

2020-12-23T16:13:31Z

Jwrm2: /* Working on the group code */ Added instructions for large files.

Git Workflow

2020-12-23T16:02:43Z

Jwrm2: Added information on git LFS

Git Workflow

2020-12-16T10:15:33Z

Jwrm2: /* Writing a Paper */ Corrected pulling order

Git Workflow

2020-12-16T10:14:07Z

Jwrm2: /* Basic Workflow */ More details on git pull.

Git Workflow

2020-12-10T17:47:00Z

Jwrm2: /* Initial Checkout */ Added warning against using https

Git Workflow

2020-12-09T22:08:29Z

Jwrm2: /* Initial Checkout */ Typo fixed

Git Workflow

2020-12-09T21:54:02Z

Jwrm2: /* Setting up SSH access */ Added information on using rsa rather than ed25519

Git Workflow

2020-12-09T18:19:33Z

Jwrm2: Fixed broken coding standards link

Comprehensive Contents Page

2020-09-28T10:09:54Z

Jwrm2: /* Acquiring and compiling the group software */ Link to Git page

This page is designed to organise all of the pages on this wiki, as well as provide other useful links. Note that some pages may appear under more than one heading.

== Getting Started ==
[[Wales Group]] provides good step-by-step instructions. Relevant pages are:

=== Acquiring and compiling the group software ===
* [[SVN setup]]
* [[Git Workflow]]
* [[Wales Group Version control]] - to keep the code standardised.
* Theory Sector [http://wwmm.ch.cam.ac.uk/wikis/cuc3/index.php/SVN_Page SVN Page] - some useful general information on SVN commands.
* [[Compiling Wales Group codes using cmake]] - CMake (Cross-platform Make) allows us to compile and test the group codebase regardless of platform. This page provides crucial information how to compile using cmake.
* [[ElaborateDiff]]

=== Maintaining code health ===
* [[Jenkins CI]] - explains Jenkins, which we use to download our code and compile each of our targets with each of the compilers every night.
* https://wales-jenkins.ch.cam.ac.uk/ - log for our Jenkins tests.
* [[Branching and Merging]]
* [[Cmake interface building]]
* [[Installing python modules]]
* [[Revamping the modules system]]

=== Collaborators without access to the SVN repository ===
For licensing reasons, some code cannot be included in the Wales Group public tarball.
* http://www-wales.ch.cam.ac.uk/svn.tar.bz2 - Wales group public tarball. Includes [[GMIN]], [[OPTIM]] and [[PATHSAMPLE]].
If a collaborator has a [[CHARMM]] or [[AMBER]] licence, we do maintain separate tarballs which include the [[CHARMM]], [[AMBER]] and [[CHARMM]]+[[AMBER]] source and interfaces. These are not linked anywhere on the website and require a username ('''wales''') and password ('''group''') to download:

* [http://www-wales.ch.cam.ac.uk/CHARMM/svn.CHARMM.tar.bz2 CHARMM]
* [http://www-wales.ch.cam.ac.uk/AMBER/svn.AMBER.tar.bz2 AMBER]
* [http://www-wales.ch.cam.ac.uk/both/svn.both.tar.bz2 AMBER+CHARMM]

=== Running on Windows ===
Not particularly recommended.
* [[Running Wales Group software on Windows 7]]

== Wales Group Programs ==

=== Programs ===
* [[GMIN]]: A program for finding global minima and calculating thermodynamic properties from basin-sampling.
* [[OPTIM]]: A program for optimizing geometries and calculating reaction pathways.
* [[PATHSAMPLE]]: A driver for OPTIM to create stationary point databases using discrete path sampling and perform kinetic analysis.
* [[Pele]]: Python energy landscape explorer. A pythonic rewrite of some core functionality of GMIN, OPTIM, and PATHSAMPLE. Can be very useful for visualizing your system and for rapidly implementing and testing new ideas.

=== Curated Examples ===
* https://github.com/wales-group/examples - set of tutorials detailing how to use GMIN, OPTIM and PATHSAMPLE. Essential for beginners.
* http://www-wales.ch.cam.ac.uk/VM/Wales_Group_VM.ova - Pre-prepared teaching virtual machine. This contains the code and examples.
* https://www.virtualbox.org/wiki/Downloads - This is required if using the VM above.
* https://github.com/wales-group/examples.git - Alternatively, you can run the examples on your own machine. To get hold of the relevant files:
<pre>
git clone https://github.com/wales-group/examples.git
</pre>

== Useful Notes on Wales Group Programs and Subroutines ==
=== [[GMIN]] ===
* [[Adding a model to GMIN]] - rough outline of the subroutines that need to be changed to add a new model to GMIN
* [[Compiling Wales Group codes using cmake | Compiling GMIN using cmake ]]
* [[Selecting search parameters for GMIN]]
* [[Global optimization of biomolecules using CHARMM]]
* [[Global optimization of biomolecules using AMBER9]]
* [[Global optimization of biomolecules using AMBER9 with Structural Restraints]]
* [[Calculating binding free energy using the FSA method]]
* [[Restarting a GMIN run from a dump file]]
* [[Using the implicit membrane model IMM1]]
* [[Running a Go model with the AMHGMIN]]
* [[Running a G\=o model with the AMHGMIN]]
* [[Ligand binding-mode searches with HBONDMATRIX]]
* [[Compiling and using GMIN with QUIP]]
* [[Using GMIN and OPTIM with GPUs]]
* [[Using GMIN to generate endpoints]]
* [[Using GMIN to generate endpoints (CHARMM)]]
* [[Generating a GMIN Eclipse project]]
* [[Mutational BH steps]]
* [[Biomolecules in the energy landscape framework]]
* [[DMAGMIN setup]]
* [[Keywords]]
* [[PYGMIN & DMACRYS]]
* [[Rotamer moves in AMBER]]
* [[Python interface for GMIN/OPTIM]]

==== Scripts ====
* [[makerestart]]: A bash script to automatically set up a GMIN restart run
* [[progress]] A bash script to tell you the % completion of a GMIN job and give an estimated time remaining

==== Useful info for coding GMIN ====
* [[Program flow]] - contains information about what the various files in GMIN do and what order they're called.
* [[amberinterface]]

==== Projects ====
* [[GMIN MOVES module]]
* [[GMIN SANITY module]]
* [[GMIN TESTS module]]
* [[CAMSHIFT]]

=== [[OPTIM]] ===
* [[Adding a model to OPTIM]] - rough outline of the subrounties that need to be changed to add a new model to OPTIM
* [[Adding partially finished OPTIM stationary points to a PATHSAMPLE database]]
* [[perm-pdb.py]]: A python program that creates a ''perm.allow'' file for use with [[OPTIM]] and [[PATHSAMPLE]].
* [[visualising normal modes using VMD and OPTIM]]
* [[Compiling Wales Group codes using cmake | Compiling OPTIM using cmake ]]
* [[OPTIM/Q-Chem Tutorial]]
* [[OPTIM and PY ellipsoids tutorial]]
* [[OPTIM output files]]
* [[Minimizing a structure using OPTIM and AMBER9]]
* [[Minimizing a structure using OPTIM and CHARMM]]
* [[Creating movies (.mpg) of paths using OPTIM]]
* [[Performing a normal mode analysis of a biomolecule using OPTIM (AMBER and CHARMM)]]
* [[Debugging odd transition states in OPTIM]]
* [[Connecting two minima with a pathway]] - step by step
* [[Compiling and using OPTIM with QUIP]]
* [[Running an Gaussian03 interfaced OPTIM job]]
* [[The effect of calculating less than the maximum number of eigenvalues using ENDHESS n]]
* [[Biomolecules in the energy landscape framework]]
* [[BLJ60 example setup]]
* [[Finding an initial path with OPTIM and starting up PATHSAMPLE]]
* [[Finding an initial path with OPTIM and starting up PATHSAMPLE (CHARMM)]]
* [[Python interface for GMIN/OPTIM]]
* [[Thomson problem in OPTIM]]
* [[Instanton tunneling and classical rate calculations with OPTIM]]
* [[Loading OPTIM's min.data.info files into PATHSAMPLE]]
* [[common setup problem : No Frequency Warning]]

=== [[PATHSAMPLE]] ===
* [[Adding a model to PATHSAMPLE]] - rough outline of the subrounties that need to be changed to add a new model to PATHSAMPLE
* [[Alternatively, making the initial path with PATHSAMPLE itself]]
* [[Alternatively, making the initial path with PATHSAMPLE itself (CHARMM)]]
* [[perm-pdb.py]]: A python program that creates a ''perm.allow'' file for use with [[OPTIM]] and [[PATHSAMPLE]].
* [[dijkstra_test.py]]: A python script to test whether the information in pairlist and ts.data connects the A and B set. (If not, PATHSAMPLE will not work without actually exiting.)
* [[Compiling Wales Group codes using cmake | Compiling PATHSAMPLE using cmake ]]
* [[IMPORTANT: Using PATHSAMPLE safely on sinister]]
* [[Adding a model for PATHSAMPLE]]
* [[List of output files for PATHSAMPLE]]
* [[Using BHINTERP to find minima between two end points]]
* [[Finding an initial path between two end points (minima)]]
* [[Adding partially finished OPTIM stationary points to a PATHSAMPLE database]]
* [[Optimising a path]]
* [[Fine tuning UNTRAP]] - ensuring that it picks sensible minima
* [[Calculating rate constants (GT and fastest path)]]
* [[Calculating rate constants (SGT, DGT, and SDGT)]]
* [[Identifying the k fastest paths between endpoints using KSHORTESTPATHS]]
* [[Removing minima and transition states from the database]]
* [[Relaxing existing minima with new potential and creating new database]]
* [[Relaxing existing transition states with new potential and creating new database]]
* [[If things go wrong...]]
* [[If you lost file min.data, but still you have points.min]]
* [[path.info file is not read, causes PATHSAMPLE to die]]
* [[BLJ60 example setup]]
* [[When PATHSAMPLE finds a connected path, but using DIJKSTRA 0 fails to find the connected path]]
* [[Biomolecules in PATHSAMPLE]]
* [[Biomolecules in the energy landscape framework]]
* [[Expanding the kinetic transition network with PATHSAMPLE]]
* [[Expanding the kinetic transition network with PATHSAMPLE (CHARMM)]]
* [[Finding an initial path with OPTIM and starting up PATHSAMPLE]]
* [[Finding an initial path with OPTIM and starting up PATHSAMPLE (CHARMM)]]
* [[Pathsampling short paths]]
* [[Pathsampling short paths (CHARMM)]]
* [[Loading OPTIM's min.data.info files into PATHSAMPLE]]
* [[Connecting Sub-databases]]
* [[CHECKSPMUTATE]]: An extension of CHECKSPODATA which allows for a protein to be mutated or transformed into a homologue.
* [[Pathway Gap Filling Post-CHECKSPMUTATE]]: Post-processing following CHECKSPMUTATE
* [[STARTING INITIAL PATH JOBS WITH PATHSAMPLE]]: How to start initial path jobs with PATHSAMPLE if no min.data or path.info files are present.

=== [[Notes on MINPERMDIST | MINPERMDIST]] ===

=== [[Quasi-continuous interpolation for biomolecules | QCI]] ===

== Non-Group Software ==

=== [[AMBER]] ===
Molecular dynamics simulation program and associated force fields.
* [http://ambermd.org/ AMBER]
* [http://ambermd.org/tutorials/ AMBER tutorials] - recommended reading for '''ANYONE''' using AMBER!
* [[Notes on AMBER 12 interface]]
* [[Using AMBER 14 on the GPU and compute clusters]]
* [[Generating parameters using AMBER's built in General Forcefield (gaff)]]
* [[Generating parameters using RESP charges from GAMESS-US]]
* [[Simple scripts for LEaP to create topology and coordinate files]]
* [[Preparing an AMBER topology file for a protein system]] - step by step guide
* [[Setting up]] - step by step guide to prepare and then symmetrise a simple (protein-only) system
* [[Using Molfacture to edit molecules and add hydrogens]]
* [[Preparing an AMBER topology file for a protein plus ligand system]] - step by step guide
* [[Symmetrising AMBER topology files]] - step by step guide for symmetrising a complex protein+ligand system
* [[Producing a PDB from a coordinates and topology file]] - using ''amdpdb''
* [[Running GMIN with MD move steps AMBER]]
* [[Performing a normal mode analysis of a biomolecule using OPTIM (AMBER and CHARMM)]]
* [[Evaluating different components of AMBER energy function with SANDER]]
* [[Mutational BH steps]]
* [[CHECKSPMUTATE]]: An extension of CHECKSPODATA which allows for a protein to be mutated or transformed into a homologue.
* [[Pathway Gap Filling Post-CHECKSPMUTATE]]: Post-processing following CHECKSPMUTATE
* [[REMD with AMBER]]
* [[Performing a hydrogen-bond analysis]]
* [[Alternatively, making the initial path with PATHSAMPLE itself]]
* [[Biomolecules in the energy landscape framework]]
* [[perm-prmtop.py]] - A python program that converts an AMBER9 topology file into one with a symmetrised potential with respect to exchange (updated for AMBER12 and ff14SB).
* [[Rotamer moves in AMBER]]
* [[Creating mismatched DNA duplex using NAB]]

=== [[aux2bib]] ===
To generate a bib file containing only the entries cited in a given .tex file from a larger bib or multiple bib files.
* [https://ctan.org/pkg/bibtools Get script here]

=== [[CamCasp]] ===
Cambridge package for Calculation of Anisotropic Site Properties
From Anthony Stone's website: 'CamCASP is a collection of scripts and programs written by Dr Alston Misquitta and myself for the calculation ab initio of distributed multipoles, polarizabilities, dispersion coefficients and repulsion parameters for individual molecules, and interaction energies between pairs of molecules using SAPT(DFT).'
* [http://www-stone.ch.cam.ac.uk/programs.html CamCASP home]
* [[CamCASP/Programming]]
* [[CamCASP/Programming/5/example1]]
* [[CamCASP/Notes]]
* [[CamCASP/Bugs]]
* [[CamCASP/ToDo/diskIO]]
* [[CamCASP/ToDo/Memory]]
* [[CamCASP/CodeExamples/DirectAccess]]

=== [[CPMD]] ===
Implementation of DFT for ''ab-initio'' molecular dynamics.
* [http://www.cpmd.org/ Home Page]
* [[CPMDInput]]

=== [[CHARMM]] ===
Molecular dynamics simulation program and associated force fields.
* [https://www.charmm.org/charmm/?CFID=65f7b3aa-8037-452a-bcd1-7583dd83a087&CFTOKEN=0 CHARMM]
* [[Generating pdb, crd and psf for a peptide sequence]]
* [[Converting between '.crd' and '.pdb']]
* [[Calculating energy of a conformation]]
* [[Calculating molecular properties]]
* [[Calculating order parameters]]
* [[CAMSHIFT]]
* [[Setting up (CHARMM)]] - step by step guide to prepare and then symmetrise a simple (protein-only) system
* [[If you need to change the number of atoms (e.g. making a united-atom charmm19 .crd file, or if atoms are missing)]]
* [[Performing a normal mode analysis of a biomolecule using OPTIM (AMBER and CHARMM)]]
* [[Minimizing a structure using OPTIM and CHARMM]]
* [[Alternatively, making the initial path with PATHSAMPLE itself (CHARMM)]]
* [[Expanding the kinetic transition network with PATHSAMPLE (CHARMM)]]
* [[Finding an initial path with OPTIM and starting up PATHSAMPLE (CHARMM)]]
* [[Pathsampling short paths (CHARMM)]]

=== [[disconnectionDPS]] ===
Produces disconnectivity graphs from min.data and ts.data files. This is included in the Wales group public tarball.
* [[Constructing Free Energy Disconnectivity Graphs]]

=== [[DMACRYS]] ===
Package which models crystals of rigid molecules.
* [http://www.chem.ucl.ac.uk/cposs/dmacrys/index.html Home Page]
* [[DMACRYS interface]]
* [[DMAGMIN setup]]
* [[PYGMIN & DMACRYS]]

=== [[GAMESS]] ===
General ''ab initio'' quantum chemistry package.
* [https://www.msg.chem.iastate.edu/gamess/ GAMESS]

=== [[Gaussian]] ===
General purpose package for computational chemistry calculations.
* [[Running an Gaussian03 interfaced OPTIM job]]

=== [[gnuplot]] ===
Open source graphing program.
* [http://www.gnuplot.info/ gnuplot]
* [[Plotting a quick histogram in gnuplot using the raw data]]
* [[Plotting data in real time]]
* [[Linear and non-linear regression in gnuplot]]

=== [[GROMACS]] ===
Molecular dynamics package.
* [[Installing GROMACS on Clust]]
* [http://www.mdtutorials.com/gmx/ External tutorials]
* [http://www.gromacs.org/Documentation/Tutorials More external tutorials]

=== [[HiRE-RNA]] ===
High-res course-grained energy model for RNA.
* [https://pubs.acs.org/doi/10.1021/jp102497y Explanatory Paper]

=== [[latex2html]] ===
Script which converts latex documents into HTML pages.
* [https://www.latex2html.org/ Get script here]

=== [[MMTSB-toolset]] ===
Group of perl scripts which can be used to setup and run energy minimization, structural analysis and MD with CHARMM or AMBER.
* [http://feig.bch.msu.edu/mmtsb/Main_Page Documentation]
* [http://www.mmtsb.org/workshops/mmtsb-ctbp_2006/Tutorials/WorkshopTutorials_2006.html External tutorials]
* [[Installing and setting up the MMTSB toolset]]
* [[REX (Replica EXchange MD) with the MMTSB-toolset]]

=== [[Simulations using OPEP | OPEP]] ===
OPEP is a coarse-grained force field providing a potential for proteins and RNA.
* [http://opep.galaxy.ibpc.fr/ OPEP file generator here]
* [[Biomolecules in the energy landscape framework]]

=== [[pgprof]] ===
Profiler for portland-compiled codes
* [[Portland compiler fails trying to allocate an unexpectedly large amount of memory: issue with large arrays]]

=== [[Pymol]] ===
Molecular visualisation program.
* [https://pymol.org/2/ PyMOL]
* [https://pymolwiki.org/index.php/Main_Page PyMOL Community Wiki]
* [[loading AMBER prmtop and inpcrd files into Pymol]]
* [[producing sexy ray-traced images]]
* [[advanced colouring]]
* [[Installing python modules]]
* [[PYGMIN & DMACRYS]]
* [[path2pdb.py]] - A python program to convert ''path.info'' to ''path_all.pdb'' - you can easy visualize your path in VMD :)
* [[extractedmin2pdb.py]]: A python program to convert ''exctractedmin'' to PDB format
=== [[VASP]] ===
OPTIM has an interface to VASP, which is installed on CSD3. In collaboration with Bora Karasulu the interface has been updated to use VASP format POSCAR input files for both single- and double-ended optimisations and path searches. The OPTIM odata file requires a line like

VASP 'mpirun -ppn 16 -np 16 /home/bk393/APPS/vasp.5.4.4/with-VTST/bin/vasp_std > vasp.out'

POSCAR files can be visualised using ase, the Atomic Simulation Environment, which can be accessed on volkhan via

module load anaconda/python3/5.3.0

pip install ase --user

ase-gui POSCAR1.vasp &

which assumes that ~/.input/bin is in your $PATH environment variable.

=== [[VMD]] ===
Molecular visualisation program.
* [http://www.ks.uiuc.edu/Research/vmd/current/ug/ug.html Documentation]
* [http://www.ks.uiuc.edu/Training/Tutorials/vmd/tutorial-html/index.html External tutorials]
* [[using VMD to display and manipulate '.pdb' files]]
* [[loading coordinate files into VMD with the help of an AMBER topology file]] e.g. to visualise the results of a GMIN run using AMBER9
* [[visualising normal modes using VMD and OPTIM]]
* [[path2pdb.py]]: A python program to convert ''path.info'' to ''path_all.pdb'' - you can easy visualize your path in VMD :)
* [[path2xyz.py]]: A python program to convert ''path.info'' to ''path_all.xyz''
* [[extractedmin2pdb.py]]: A python program to convert ''exctractedmin'' to PDB format
* [[Useful .vmdrc file]]
* [[plotGMINms.tcl]]: a tcl script for plotting ellipsoids in VMD.
* [[VMD script to annotate each frame of a trajectory]]

=== [[xfig]] ===
Open source vector graphics editor
* [https://ctan.org/tex-archive/support/epstopdf/ Convert eps to pdf]

=== [[Xmakemol]] ===
Program for visualising atomic and molecular systems.
* [https://www.nongnu.org/xmakemol/ XMakemol]

=== [[xmgrace]] ===
2D plotting tool.
* [http://exciting-code.org/xmgrace-quickstart Xmgrace]

== Theoretical/Mathematical Notes ==

* [[Density of states and thermodynamics from energy distributions at different temperatures]]
* [[Ellipsoid.model]]
* [[Ellipsoid.model.xyz]]
* [[Ellipsoid.xyz]]
* [[Gencoords]]
* [[GenCoords]]
* [[GenCoords Models]]
* [[Rotamer moves in AMBER]]
* [[Thomson problem in OPTIM]]

=== Angle-axis notes ===

* [[Angle-axis framework]]
* [[Computing normal modes in angle-axis]]

=== Rigid Bodies ===

* [[Automatic Rigid Body Grouping]]
* [[Rigid body input files for proteins using genrigid-input.py]]
* [[Local Rigid Body Framework]]
* [[Local rigid body in OPTIM]]

== Useful Scripts ==
* [[perm-prmtop.py]]: A python program that converts an AMBER9 topology file into one with a symmetrised potential with respect to exchange (updated for AMBER12 and ff14SB).
* [[perm-pdb.py]]: A python program that creates a ''perm.allow'' file for use with [[OPTIM]] and [[PATHSAMPLE]].
* [[path2pdb.py]]: A python program to convert ''path.info'' to ''path_all.pdb'' - you can easy visualize your path in VMD :)
* [[path2xyz.py]]: A python program to convert ''path.info'' to ''path_all.xyz''
* [[dijkstra_test.py]]: A python script to test whether the information in pairlist and ts.data connects the A and B set. (If not, PATHSAMPLE will not work without actually exiting.)
* [[extractedmin2pdb.py]]: A python program to convert ''exctractedmin'' to PDB format
* [[colourdiscon.py]]: A python program for sorting input for disconnectivity graphs
* [[pdb_to_movie.py]]: A python program to create an AMH movieseg file from a PDB file
* [[makerestart]]: A bash script to automatically set up a GMIN restart run
* [[progress]] A bash script to tell you the % completion of a GMIN job and give an estimated time remaining
* [[recommended bash aliases]]
* [[David's .inputrc file]]
* [[Useful .vmdrc file]]
* [[Density of states and thermodynamics from energy distributions at different temperatures]]
* [[GenCoords]]: A fortran program to generate coarse grain building blocks and initial coords using a set of geometric models.
* [[plotGMINms.tcl]]: a tcl script for plotting ellipsoids in VMD.
See also the SCRIPTS/ directory in the SVN repository!
* [[Computing CHARMM FF energy using GMIN, MMTSB and CHARMM]] - Computes the Charmm FF energy of the same structure. Useful for cross-validating force field settings in GMIN data file, CHARMM input file and MMTSB options.
* [[Automatic Rigid Body Grouping]]
* [[ElaborateDiff]]
* [[Parameter-scanning script]]
* [[Pdb to movie.py]]
* [[VMD script to annotate each frame of a trajectory]]

== Useful links ==
* [http://www.ch.cam.ac.uk/computing/theory-compute-clusters The Theory Compute Clusters support page]. Contains useful cluster specific information, including example job submission scripts.

* A useful website which contains AMBER (GAFF) and OPLS parameters for small molecules. http://virtualchemistry.org/gmld.php . This could save us lot of time while trying to derive parameters on our own. If you are lucky, the molecule of your interest may already be there in the existing database. The topology files are in GROMACS format but possibly can be converted into AMBER parameter files. (script anyone ?)

* The moving-domain QM/MM method developed by Victor Batista's group http://gascon.chem.uconn.edu/software. This approach can be used in the derivation of charges for large proteins and nucleic acids, where a full-fledged ONIOM based calculation is comptutationally prohibitive. It has been applied to systems like the Gramicidin ion channel and Photosystem II.

== Miscellaneous ==
* [[Animated GIF on the group website]]
* [[Backup strategy]]
* [[Chain crossing]]
* [[Computer Office services]]
* [[Computing values only once]]
* [[Decoding heat capacity curves]]
* [[Differences from Clust]]
* [[Fixing thunderbird links]]
* [[If you need to change the number of atoms (e.g. making a united-atom charmm19 .crd file, or if atoms are missing)]]
* [[Intel Trace Analyzer and Collector]]
* [[LDAP plans]]
* [[Lapack compilation]]
* [[Mek-quake Queueing system]]
* [[Mek-quake initial setup notes]]
* [[New mek-quake]]
* [[Maui compilation]]
* [[Torque and Maui]]
* [[Mercurial]]
* [[Migrating to the new SVN server]]
* [[NECI Parallelization]]
* [[Optimization tricks]]
* [[Other IT stuff]]
* [[Porfuncs Documentation]]
* [[Progress]]
* [[Proposed changes to backup and archiving]]
* [[Rama upgrade]]
* [[Remastering Knoppix]]
* [[See unpacked nodes]]
* [[Tardis scheduling policy]]
* [[Zippo Sicortex machine]]
* [[Beginner's guide to working in Wales group]]

== Useful linux stuff ==

===Basics===
* [[basic linux commands everyone should know!]]
* [[piping and redirecting output from one command or file to another]] - how to save yourself hours!
* [[bash loop tricks]]
* [[bash history searching]]

===Remote access===
* [[setting up aliases to quickly log you in to a different machine]]
* [[transfering files to and from your workstation]] -using ''scp'' or ''rsync''
* [[using 'ssh-keygen' to automatically log you into clusters from your workstation]] (no more typing in your password!)
* [[mounting sharedscratch locally]]

===Find and replace===
* [[short 'sed' examples]]
* [[quick guide to awk]]
* [[short 'awk' examples]]

===File manipulation===
* [[sorting a file by multiple columns]]
* [[using tar and gzip to compress/uncompress files | using tar and bzip2 to compress/uncompress files]]
* [[conversion between different data file formats]] -'almost one-line' scripts
* [[conversion between different image file formats]] - the ''convert'' command
* [[removing an excessive number of files from a directory - when 'rm' just isn't enough]]

===Cluster queues===
* [[submitting jobs, interactively or to a cluster queue system]]
* [[identifying job on a node]] - if you need to kill only one of few running jobs
* [[getting started with SLURM]]
* [[a guide to using SLURM to run PATHSAMPLE]]
* [[a guide to using SLURM to run GPU jobs on pat]]
* [[managing interactive jobs on cluster]]

===Miscellaneous/uncategorised===
* [[installing packages on your managed CUC3 workstation]]
* [[running programs in the background]] - so you can use your shell for other things at the same time
* [[finding bugs in latex documents that will not compile]]
* [[printing files from the command line using 'lpr']]
* [[uploading non image files to the wiki]]

== Compiler Flags ==

* [[Compiler Flags]]
* [[Blacklisting Compilers]]
* [[Lapack compilation]]
* [[Pdb to movie.py]]
* [[Portland compiler fails trying to allocate an unexpectedly large amount of memory: issue with large arrays]]

== SuSE ==

* [[Upgrading destiny]]
* [[Upgrading sword]]
* [[SuSE 10.1 workstation image]]
* [[SuSE 10.2 workstation image]]
* [[SuSE 10.3 workstation image]]
* [[SuSE 11.1]]

--[[User:adk44|adk44]] 17.00, 9 May 2019 (BST)

Git Workflow

2020-09-28T10:09:24Z

Jwrm2: Moved from private wiki

Getting started with SLURM

2020-05-27T13:58:43Z

Jwrm2: /* Theory */ Fixed some typos.

For running computationally expensive tasks for long periods of time, you will want to create a job and run it on a cluster. This page is intended to teach you everything required to run your first job on a SLURM CPU cluster (sinister, volkhan). For convenience, we will refer primarily to sinister, but the information is transferrable to other SLURM clusters, with a small note about SLURM versions [[A guide to using SLURM to run PATHSAMPLE | here]]. This page does not contain information about PBS (dexter), but there is some information [[Submitting jobs, interactively or to a cluster queue system | here]]. If you are running GPU (pat) jobs, you will need additional information from [[A guide to using SLURM to run GPU jobs on pat | here]].

==Basic terminology==

A cluster is a large computer that has many processors. The processors are grouped into nodes, which operate semi-independently of each other. Each node has it's own disk (accessed at /scratch/) and memory, but is capable of communicating with other nodes through a network. When you login to sinister with SSH, you are logged into the head node. This is a special node designed for interacting with users. The other nodes are compute nodes, designed for running long and intensive tasks. SLURM (Simple Linux Utility for Resource Management), now officially called the Slurm Workload Manager, is a programme that manages the compute nodes, allocating resources, starting and ending jobs, and managing the queue. To get the compute nodes to do anything, you need to ask through SLURM.

==How to not be anti-social==

When you are using sinister, the most important thing to be aware of is that other people are using it too. You need to be careful to not act in a way that impedes other users from getting on with their work. Primarily this means not excessively using the head node, as if you do other users will get a very slow response time on their SSH and will not be able to operate. There are some specific things to avoid.

*Do not run long expensive tasks on the head node: that is what the compute nodes are for. Anything longer than a few minutes should not be run on the head node. For a short expensive task (like compiling GMIN), prefix your command with 'nice', which tells the operating system to give your process a lower priority, meaning any simple commands other people might be running are not slowed down. You are not likely to notice a significant impact on the speed of your process by doing this.

$ nice make

*Do not rapidly copy a lot of data over the network. Although the nodes are networked together and moving data between them is a simple process (using NFS, the Network File System), it is relatively slow and is computationally expensive. If you constantly move data over the NFS, for example by writing a verbose log file, other users will notice. This is most likely to be an issue when writing to the directory /sharedscratch/, which is a large working space partition accessible over the network from any partition. Each node has its own space, accessed at /scratch/ on the node, or through /nodescratch/ from the head node. Best practices to avoid this problem are detailed below, but it is mentioned here due to its great importance.

*Be realistic about the requirements for your job. The cluster is a shared resource that is not infinite. To make the most efficient use of the resource, do not do things like request 32 cores for a job that can only make use of 4, or request a time limit of 1 week for a job that will only take 1 hour. The better the information you give to the queuing system, the better it can serve you.

==Submitting a job==

To submit a job, you create a job submission script, here we'll call it submit.sh, and then run

$ sbatch submit.sh

which tells SLURM to execute the contents of your submission script. When you do this, SLURM will first look through any information at the top of the job script for configuration instructions like the job time limit. Then it will place your job into a queue. When the necessary resources become available, any commands in your submission script will be executed in sequence on a compute node. Let's have a look at a simple submission script for running GMIN.

#!/bin/bash
#SBATCH -J GMIN_LJ38
#SBATCH --time=1:00:00
#SBATCH -n1

echo Starting job $SLURM_JOB_ID
hostname

TMP=/scratch/jwrm2/$SLURM_JOB_ID
mkdir -p $TMP
cp $SLURM_SUBMIT_DIR/data $SLURM_SUBMIT_DIR/coords $TMP/
cd $TMP

/home/jwrm2/bin/GMIN > logfile

cp -p lowest logfile $SLURM_SUBMIT_DIR && rm -rf $TMP
cd $SLURM_SUBMIT_DIR

We'll go through this line by line. The first line (affectionately known as the 'shebang') tells SLURM what shell to use to interpret the commands. You don't have to use Bash, but it's what most people are used to, so it's recommended if you want anyone else to be able to help you.

Next we have a series of configuration commands for SLURM. Each of these starts with #SBATCH and they are ignored by Bash. Some of the most useful are:

* -J <name> The name of the job, that will show up in diagnostic information. Make your life easier by choosing a suitably descriptive one.
* --time=hh:mm:ss The maximum amount of time your job will run for. If this time limit is reached and the job hasn't finished, SLURM will kill it. It is in yours and everyone else's interest to make this a fairly accurate estimate of how long the job will actually take (plus a little bit).
* --ntasks=<number> The number of processors your job requires.
* --nodes=<number> The number of nodes your job requires.
* --mem=<value> The amount of memory your job needs. The default is often fine.

Consult the SLURM documentation [https://slurm.schedmd.com/sbatch.html] for a full list. Here, we request one processor, for a maximum time of 1 hour, and we set the name to 'GMIN_LJ38'.

The remaining commands are executed by Bash once the job has begun. First we write out some diagnostic information: the ID of the job and which compute node it is running on. This is written to a file slurm-<job_ID>.out in the directory we run sbatch from, for example

$ cat slurm-214233.out
Starting job 214233
compute-0-17.local

The first line is a useful check that the job actually began. Knowing which node your job ran on can be useful if it terminates early/unexpectedly.

The next section of the submission script is about reducing the NFS load. Instead of running the job on /sharedscratch/, which the node accesses over the NFS, we create a directory in the node's own /scratch/ space. Change 'jwrm2' to your own username. We copy only the files GMIN needs in order to run, then change to the new directory. $SLURM_SUBMIT_DIR is a variable holding the directory from whcih sbatch was run.

Next we actually run the programme. We redirect the GMIN output to a log file. If you do not do this redirection, output will instead go to the slurm-<job_ID>.out file, which most likely resides on /sharedscratch/.

Finally, we have clean up actions to be run after GMIN has finished. We copy back only the data we are interested in to the directory from which sbatch was run, then delete the temporary directory we created on the nodescratch. Note the '&&' syntax, which means 'run the command after the && only if the command before the && was successful'. In this case, it means that the temporary directory will not be deleted if we couldn't copy those files back.

==The queuing system==

When you submit your job with sbatch, it may not be run instantly. It is quite possible that the the resources your job needs are currently being used by someone else. Your job enters the queue. You can see the queue by typing

$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
214233 CLUSTER GMIN_LJ38 jwrm2 PD 0:00 1 (Resources)
214207 CLUSTER GLP-1 kr366 R 29:31 2 compute-0-[22-23]
214112 CLUSTER pentamer_1_1292 ld506 R 1-02:03:44 1 compute-0-12

There is a lot of information here. The first column gives the ID of the job. The next column is not usually relevant. Then we see the name of the job (assigned with -J in the submission script) and the user who submitted the job. The ST column tells you the current status of the job: 'PD' means 'pending', ie. the job is waiting to run, but is not yet running; 'R' means 'running'. We also have the time the job has been running for, the number of nodes the job is using and which they are. If the job is not running, the final column may display a reason. If it shows (Resources), then your job will run as soon as the necessary resources are available. If it instead shows (Priority), then there is another waiting job in the queue that will be run before yours. If you only want to see information about your jobs (or any other user's), you can use the -u option:

$ squeue -u jwrm2

will only show the jobs submitted by the user jwrm2.

How does SLURM decide which jobs to run? Principally, it works out a user's priority by a fairshare system: the more compute time you have recently used, the lower your priority will be. You can see everyone's fairshare values by looking at the final column of the output of

$ sshare -a

Higher numbers mean a higher priority for that user. However, SLURM will also try to fit smaller jobs into gaps. If there is a 12 node job waiting from a user with a very high priority, but only one node is currently available, a short one node job from a user with low priority may be able to fit in. Therefore it is in your interest to make realistic estimates of the resources your job needs: it may end up running sooner.

==Oops, I just submitted the wrong job==

You can abort a job when it is waiting to run or when it is running. Find the job ID, either by looking at squeue or the output file in the submission directory. Let's say it was 214233:

$ scancel 214233

will immediately kill the job.

==Running PATHSAMPLE with SLURM==

PATHSAMPLE runs a little differently from standard GMIN: it launches new OPTIM jobs and sends them to the other processors available. This is pretty much taken care of if you include 'SLURM' in the pathdata file, but there are a couple of extra considerations. Firstly, PATHSAMPLE needs a nodes.info file with information about the available processors. Generate this file at the start of you submission script with:

echo $SLURM_NTASKS > nodes.info
srun hostname >> nodes.info
echo $USER >> nodes.info
pwd >> nodes.info

Note the use of srun here. It means: run this command on every one of the available processors.

Secondly, we need to decide whether PATHSAMPLE itself will be run from /sharedscratch/ (not the OPTIM jobs, PATHSAMPLE will always use /nodescratch/ for those). Running PATHSAMPLE from /sharedscratch/ is simpler and it means that if the job terminates before PATHSAMPLE completes all the requested cycles, the database on /sharedscratch/ will be in the most current state. Here is an example PATHSAMPLE script that does this

#!/bin/bash
#SBATCH --time=20:0:0
#SBATCH --job-name=PATHSAMPLE_LJ38
#SBATCH --ntasks=16

echo $SLURM_NTASKS > nodes.info
srun hostname >> nodes.info
echo $USER >> nodes.info
pwd >> nodes.info

/home/jwrm/bin/PATHSAMPLE > logfile

We run PATHSAMPLE for 20 hours on 16 cores. SLURM is free to assign those cores over multiple nodes as it sees fit. Note there is no creation of a temporary directory on /scratch/, PATHSAMPLE access the database stored in /sharedscratch/. Whenever an OPTIM job is created, PATHSAMPLE creates a temporary directory on /scratch/, copies the files required for OPTIM to run, launches OPTIM with srun, waits for OPTIM to finish, copies the files back and adds new stationary points to the database, then deletes the temporary directory. That is all taken care of for you. Because this approach involves writing to /sharedscratch/, it is only appropriate if each OPTIM job takes a while to run (more than a few minutes) and if the amount of data required for each OPTIM job is small. If either of those conditions is violated, users may notice a slowdown on the head node and be annoyed with you. In that case a different approach is required.

PATHSAMPLE can be run from /nodescratch/. This method reduces the NFS load, but is slightly more complicated and if the job terminates early, you will need to get your expanded database manually (see [[Getting started with SLURM#My job got cancelled, where are my files? | here]]). Here is an example PATHSAMPLE script that does this

#!/bin/bash
#SBATCH -J PATHSAMPLE_LJ38
#SBATCH --time=20:0:0
#SBATCH --ntask=4
#SBATCH --nodes=1

echo Starting job $SLURM_JOB_ID
hostname

TMP=/scratch/jwrm2/$SLURM_JOB_ID
mkdir -p $TMP
cp $SLURM_SUBMIT_DIR/pathdata $SLURM_SUBMIT_DIR/min.data $SLURM_SUBMIT_DIR/ts.data $SLURM_SUBMIT_DIR/points.min $SLURM_SUBMIT_DIR/points.ts $SLURM_SUBMIT_DIR/min.A $SLURM_SUBMIT_DIR/min.B $SLURM_SUBMIT_DIR/odata.connect $TMP/
cd $TMP

echo $SLURM_NTASKS > nodes.info
srun hostname >> nodes.info
echo $USER >> nodes.info
pwd >> nodes.info

/home/jwrm2/svn/PATHSAMPLE/builds/gfortran/PATHSAMPLE > logfile

cp min.data ts.data points.min points.ts min.A min.B logfile $SLURM_SUBMIT_DIR/ && rm -rf $TMP
cd $SLURM_SUBMIT_DIR/

We run for 20 hours on 4 cores. Since we are worried about the amount of data we're copying around, it makes sense to request these on the same node. We create a temporary directory on the node's /scratch/ space and copy the whole PATHSAMPLE database to it. Then we generate the nodes.info file and we're ready to run PATHSAMPLE itself. At the end, we copy the whole database back to /sharedscratch/ and delete the temporary directory if successful. This solution is not perfect, as we still have to copy the whole database over the NFS at the start and the end, but if each OPTIM job needs a lot of writing, or writes data very rapdily, it may be preferable.

==My job got cancelled, where are my files?==

If your job has finished, it will no longer appear in the output of squeue. It may have successfully completed, or it might have run out of time (or memory, etc.). Look at the end of the slurm-<job_ID>.out file. If it shows

slurmstepd: *** JOB 214233 ON compute-0-36 CANCELLED AT 2020-05-18T10:45:16 DUE TO TIME LIMIT ***

then your job ran out of time and was killed. Maybe this was a week long GMIN run that ran out of time 10 steps from the end. How annoying. Time to run it again from the beginning with a longer time limit, because there is no GMIN.dump file in the submission directory? Hardly... In our job submission script, we only deleted the temporary directory after the successful completion of the job. Therefore if the job did not successfully complete, the directory and its contents are still there. The information at the top of the slurm-<job_ID>.out file will help you find them. We created the temporary directory at

/scratch/jwrm2/214233

To access this from the head node, we need to know what node it was created on. That's why we ran 'hostname' at the start of the job. Let's say the output of that was

compute-0-17.local

We can access our files at

/nodescratch/compute-0-17/jwrm2/214233

Note that normal Bash tab completion of the directory name may not work here. That doesn't necessarily mean the directory doesn't exist, it's just a peculiarity of NFS automount.

It would be polite to delete the temporary directory after recovering any files you need.

==A smarter way to get your files==

Although recovering files from /nodescratch/ is fine, if you've every had 52 GMIN jobs terminate and needed to recover their GMIN.dump files for restarting, it gets a little tedious. Fortunately there is a better way, leveraging the signal mechanism of Linux. Here is an example:

#!/bin/bash
#SBATCH -J GMIN_LJ38
#SBATCH --time=5:00
#SBATCH -n1
#SBATCH --signal=B:USR1@120

signal_handler()
{
# Timeout commands go here
echo "Caught USR1 signal"
cp GMIN.dump logfile $SLURM_SUBMIT_DIR
exit
}
trap 'signal_handler' USR1

echo Starting job $SLURM_JOB_ID
hostname

TMP=/scratch/jwrm2/$SLURM_JOB_ID
mkdir -p $TMP
cp $SLURM_SUBMIT_DIR/data $SLURM_SUBMIT_DIR/coords $TMP/
cd $TMP

/home/jwrm2/bin/GMIN > logfile &
wait

cp -p lowest logfile $SLURM_SUBMIT_DIR && rm -rf $TMP
cd $SLURM_SUBMIT_DIR

This script will run any commands enclosed in braces at '# Timeout commands go here', 120 seconds before the job times out. It will terminate itself immediately after completing those commands, so you will not get a 'CANCELLED' message in the slurm-<job_ID>.out file (but only because of the 'exit', otherwise it would continue execution after the 'wait', which is probably not what you want). You can use it to copy back recovery files like GMIN.dump, or the updated PATHSAMPLE database, before the job terminates. If 120 seconds is not enough time for your timeout commands, change the 120 in the line '#SBATCH --signal=B:USR1@120'. The temporary directory will not be deleted if the timeout is reached, although you could put 'rm -rf $TMP' in the timeout section if you were really confident that you were grabbing everything you needed. You should occasionally go through /nodescratch/ and delete any leftover temporary directories that may be hanging around.

===Theory===

You don't need to understand how the above example works to use it, but we're all curious scientists, aren't we? The Linux kernel can communicate with a running process by sending it 'signals'. Common ones include 'SIGTERM' and 'SIGKILL'. In fact, when your job reaches the time limit, it gets sent a 'SIGTERM' signal, telling it to terminate. If the process has not set up anything specific to do on receiving a particular signal, the default action is to exit. However, most signals can have custom responses set up. An exception is 'SIGKILL', for which immediate termination cannot be overridden.

The line '#SBATCH --signal=B:USR1@120' tells SLURM to please send the signal 'USR1' (user specified signal number 1) to the job 120 seconds before reaching the time limit. 'USR1' is not sent by any system processes, so we're free to use it as we wish. Next we create a Bash function, 'signal_handler()', with our sequence of commands. We tell the Bash script to jump to our function on receiving the 'USR1' signal, overriding the default exit behaviour (trap 'signal_handler' USR1).

The final finicky bit is how we run our main programme. If we ran 'GMIN', the signal would get sent to GMIN, as the active process, rather than Bash. GMIN wouldn't know what to do with it, so would immediately exit. Not good. By running 'GMIN &' instead, the signal goes to Bash. However, if we did just that, the job would quickly reach the end after launching GMIN, which would terminate the job and all it's children, including GMIN. Oops. By writing 'wait', we tell Bash to go into an 'interruptible sleep', meaning it will sit there not doing anything until it receives a signal. Therefore it is able to excecute the commands in the 'signal_handler()' function on receiving the 'USR1' signal. As a matter of interest, execution of the main script then resumes where it left off before the signal, but the wait has now been completed (it only waits for the first signal) and execution proceeds to the 'cp' line. Putting 'exit' at the end of the function means we don't jump back to after wait. We could have put another 'wait' instead of 'exit', but that would only be useful if GMIN finished in the short period of time between completing the function and reaching the job time limit. Better to just save 2 minutes of CPU time by exiting immediately.

How does our job finish normally, if we've put it to sleep then? Well, when GMIN exits it sends, you guessed it... a signal, in this case 'SIGCHLD'. That will wake up Bash from its wait and the script will proceed to completion.

Getting started with SLURM

2020-05-26T13:22:52Z

Jwrm2: Added links to other pages with queuing information.

For running computationally expensive tasks for long periods of time, you will want to create a job and run it on a cluster. This page is intended to teach you everything required to run your first job on a SLURM CPU cluster (sinister, volkhan). For convenience, we will refer primarily to sinister, but the information is transferrable to other SLURM clusters, with a small note about SLURM versions [[A guide to using SLURM to run PATHSAMPLE | here]]. This page does not contain information about PBS (dexter), but there is some information [[Submitting jobs, interactively or to a cluster queue system | here]]. If you are running GPU (pat) jobs, you will need additional information from [[A guide to using SLURM to run GPU jobs on pat | here]].

==Basic terminology==

A cluster is a large computer that has many processors. The processors are grouped into nodes, which operate semi-independently of each other. Each node has it's own disk (accessed at /scratch/) and memory, but is capable of communicating with other nodes through a network. When you login to sinister with SSH, you are logged into the head node. This is a special node designed for interacting with users. The other nodes are compute nodes, designed for running long and intensive tasks. SLURM (Simple Linux Utility for Resource Management), now officially called the Slurm Workload Manager, is a programme that manages the compute nodes, allocating resources, starting and ending jobs, and managing the queue. To get the compute nodes to do anything, you need to ask through SLURM.

==How to not be anti-social==

When you are using sinister, the most important thing to be aware of is that other people are using it too. You need to be careful to not act in a way that impedes other users from getting on with their work. Primarily this means not excessively using the head node, as if you do other users will get a very slow response time on their SSH and will not be able to operate. There are some specific things to avoid.

*Do not run long expensive tasks on the head node: that is what the compute nodes are for. Anything longer than a few minutes should not be run on the head node. For a short expensive task (like compiling GMIN), prefix your command with 'nice', which tells the operating system to give your process a lower priority, meaning any simple commands other people might be running are not slowed down. You are not likely to notice a significant impact on the speed of your process by doing this.

$ nice make

*Do not rapidly copy a lot of data over the network. Although the nodes are networked together and moving data between them is a simple process (using NFS, the Network File System), it is relatively slow and is computationally expensive. If you constantly move data over the NFS, for example by writing a verbose log file, other users will notice. This is most likely to be an issue when writing to the directory /sharedscratch/, which is a large working space partition accessible over the network from any partition. Each node has its own space, accessed at /scratch/ on the node, or through /nodescratch/ from the head node. Best practices to avoid this problem are detailed below, but it is mentioned here due to its great importance.

*Be realistic about the requirements for your job. The cluster is a shared resource that is not infinite. To make the most efficient use of the resource, do not do things like request 32 cores for a job that can only make use of 4, or request a time limit of 1 week for a job that will only take 1 hour. The better the information you give to the queuing system, the better it can serve you.

==Submitting a job==

To submit a job, you create a job submission script, here we'll call it submit.sh, and then run

$ sbatch submit.sh

which tells SLURM to execute the contents of your submission script. When you do this, SLURM will first look through any information at the top of the job script for configuration instructions like the job time limit. Then it will place your job into a queue. When the necessary resources become available, any commands in your submission script will be executed in sequence on a compute node. Let's have a look at a simple submission script for running GMIN.

#!/bin/bash
#SBATCH -J GMIN_LJ38
#SBATCH --time=1:00:00
#SBATCH -n1

echo Starting job $SLURM_JOB_ID
hostname

TMP=/scratch/jwrm2/$SLURM_JOB_ID
mkdir -p $TMP
cp $SLURM_SUBMIT_DIR/data $SLURM_SUBMIT_DIR/coords $TMP/
cd $TMP

/home/jwrm2/bin/GMIN > logfile

cp -p lowest logfile $SLURM_SUBMIT_DIR && rm -rf $TMP
cd $SLURM_SUBMIT_DIR

We'll go through this line by line. The first line (affectionately known as the 'shebang') tells SLURM what shell to use to interpret the commands. You don't have to use Bash, but it's what most people are used to, so it's recommended if you want anyone else to be able to help you.

Next we have a series of configuration commands for SLURM. Each of these starts with #SBATCH and they are ignored by Bash. Some of the most useful are:

* -J <name> The name of the job, that will show up in diagnostic information. Make your life easier by choosing a suitably descriptive one.
* --time=hh:mm:ss The maximum amount of time your job will run for. If this time limit is reached and the job hasn't finished, SLURM will kill it. It is in yours and everyone else's interest to make this a fairly accurate estimate of how long the job will actually take (plus a little bit).
* --ntasks=<number> The number of processors your job requires.
* --nodes=<number> The number of nodes your job requires.
* --mem=<value> The amount of memory your job needs. The default is often fine.

Consult the SLURM documentation [https://slurm.schedmd.com/sbatch.html] for a full list. Here, we request one processor, for a maximum time of 1 hour, and we set the name to 'GMIN_LJ38'.

The remaining commands are executed by Bash once the job has begun. First we write out some diagnostic information: the ID of the job and which compute node it is running on. This is written to a file slurm-<job_ID>.out in the directory we run sbatch from, for example

$ cat slurm-214233.out
Starting job 214233
compute-0-17.local

The first line is a useful check that the job actually began. Knowing which node your job ran on can be useful if it terminates early/unexpectedly.

The next section of the submission script is about reducing the NFS load. Instead of running the job on /sharedscratch/, which the node accesses over the NFS, we create a directory in the node's own /scratch/ space. Change 'jwrm2' to your own username. We copy only the files GMIN needs in order to run, then change to the new directory. $SLURM_SUBMIT_DIR is a variable holding the directory from whcih sbatch was run.

Next we actually run the programme. We redirect the GMIN output to a log file. If you do not do this redirection, output will instead go to the slurm-<job_ID>.out file, which most likely resides on /sharedscratch/.

Finally, we have clean up actions to be run after GMIN has finished. We copy back only the data we are interested in to the directory from which sbatch was run, then delete the temporary directory we created on the nodescratch. Note the '&&' syntax, which means 'run the command after the && only if the command before the && was successful'. In this case, it means that the temporary directory will not be deleted if we couldn't copy those files back.

==The queuing system==

When you submit your job with sbatch, it may not be run instantly. It is quite possible that the the resources your job needs are currently being used by someone else. Your job enters the queue. You can see the queue by typing

$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
214233 CLUSTER GMIN_LJ38 jwrm2 PD 0:00 1 (Resources)
214207 CLUSTER GLP-1 kr366 R 29:31 2 compute-0-[22-23]
214112 CLUSTER pentamer_1_1292 ld506 R 1-02:03:44 1 compute-0-12

There is a lot of information here. The first column gives the ID of the job. The next column is not usually relevant. Then we see the name of the job (assigned with -J in the submission script) and the user who submitted the job. The ST column tells you the current status of the job: 'PD' means 'pending', ie. the job is waiting to run, but is not yet running; 'R' means 'running'. We also have the time the job has been running for, the number of nodes the job is using and which they are. If the job is not running, the final column may display a reason. If it shows (Resources), then your job will run as soon as the necessary resources are available. If it instead shows (Priority), then there is another waiting job in the queue that will be run before yours. If you only want to see information about your jobs (or any other user's), you can use the -u option:

$ squeue -u jwrm2

will only show the jobs submitted by the user jwrm2.

How does SLURM decide which jobs to run? Principally, it works out a user's priority by a fairshare system: the more compute time you have recently used, the lower your priority will be. You can see everyone's fairshare values by looking at the final column of the output of

$ sshare -a

Higher numbers mean a higher priority for that user. However, SLURM will also try to fit smaller jobs into gaps. If there is a 12 node job waiting from a user with a very high priority, but only one node is currently available, a short one node job from a user with low priority may be able to fit in. Therefore it is in your interest to make realistic estimates of the resources your job needs: it may end up running sooner.

==Oops, I just submitted the wrong job==

You can abort a job when it is waiting to run or when it is running. Find the job ID, either by looking at squeue or the output file in the submission directory. Let's say it was 214233:

$ scancel 214233

will immediately kill the job.

==Running PATHSAMPLE with SLURM==

PATHSAMPLE runs a little differently from standard GMIN: it launches new OPTIM jobs and sends them to the other processors available. This is pretty much taken care of if you include 'SLURM' in the pathdata file, but there are a couple of extra considerations. Firstly, PATHSAMPLE needs a nodes.info file with information about the available processors. Generate this file at the start of you submission script with:

echo $SLURM_NTASKS > nodes.info
srun hostname >> nodes.info
echo $USER >> nodes.info
pwd >> nodes.info

Note the use of srun here. It means: run this command on every one of the available processors.

Secondly, we need to decide whether PATHSAMPLE itself will be run from /sharedscratch/ (not the OPTIM jobs, PATHSAMPLE will always use /nodescratch/ for those). Running PATHSAMPLE from /sharedscratch/ is simpler and it means that if the job terminates before PATHSAMPLE completes all the requested cycles, the database on /sharedscratch/ will be in the most current state. Here is an example PATHSAMPLE script that does this

#!/bin/bash
#SBATCH --time=20:0:0
#SBATCH --job-name=PATHSAMPLE_LJ38
#SBATCH --ntasks=16

echo $SLURM_NTASKS > nodes.info
srun hostname >> nodes.info
echo $USER >> nodes.info
pwd >> nodes.info

/home/jwrm/bin/PATHSAMPLE > logfile

We run PATHSAMPLE for 20 hours on 16 cores. SLURM is free to assign those cores over multiple nodes as it sees fit. Note there is no creation of a temporary directory on /scratch/, PATHSAMPLE access the database stored in /sharedscratch/. Whenever an OPTIM job is created, PATHSAMPLE creates a temporary directory on /scratch/, copies the files required for OPTIM to run, launches OPTIM with srun, waits for OPTIM to finish, copies the files back and adds new stationary points to the database, then deletes the temporary directory. That is all taken care of for you. Because this approach involves writing to /sharedscratch/, it is only appropriate if each OPTIM job takes a while to run (more than a few minutes) and if the amount of data required for each OPTIM job is small. If either of those conditions is violated, users may notice a slowdown on the head node and be annoyed with you. In that case a different approach is required.

PATHSAMPLE can be run from /nodescratch/. This method reduces the NFS load, but is slightly more complicated and if the job terminates early, you will need to get your expanded database manually (see [[Getting started with SLURM#My job got cancelled, where are my files? | here]]). Here is an example PATHSAMPLE script that does this

#!/bin/bash
#SBATCH -J PATHSAMPLE_LJ38
#SBATCH --time=20:0:0
#SBATCH --ntask=4
#SBATCH --nodes=1

echo Starting job $SLURM_JOB_ID
hostname

TMP=/scratch/jwrm2/$SLURM_JOB_ID
mkdir -p $TMP
cp $SLURM_SUBMIT_DIR/pathdata $SLURM_SUBMIT_DIR/min.data $SLURM_SUBMIT_DIR/ts.data $SLURM_SUBMIT_DIR/points.min $SLURM_SUBMIT_DIR/points.ts $SLURM_SUBMIT_DIR/min.A $SLURM_SUBMIT_DIR/min.B $SLURM_SUBMIT_DIR/odata.connect $TMP/
cd $TMP

echo $SLURM_NTASKS > nodes.info
srun hostname >> nodes.info
echo $USER >> nodes.info
pwd >> nodes.info

/home/jwrm2/svn/PATHSAMPLE/builds/gfortran/PATHSAMPLE > logfile

cp min.data ts.data points.min points.ts min.A min.B logfile $SLURM_SUBMIT_DIR/ && rm -rf $TMP
cd $SLURM_SUBMIT_DIR/

We run for 20 hours on 4 cores. Since we are worried about the amount of data we're copying around, it makes sense to request these on the same node. We create a temporary directory on the node's /scratch/ space and copy the whole PATHSAMPLE database to it. Then we generate the nodes.info file and we're ready to run PATHSAMPLE itself. At the end, we copy the whole database back to /sharedscratch/ and delete the temporary directory if successful. This solution is not perfect, as we still have to copy the whole database over the NFS at the start and the end, but if each OPTIM job needs a lot of writing, or writes data very rapdily, it may be preferable.

==My job got cancelled, where are my files?==

If your job has finished, it will no longer appear in the output of squeue. It may have successfully completed, or it might have run out of time (or memory, etc.). Look at the end of the slurm-<job_ID>.out file. If it shows

slurmstepd: *** JOB 214233 ON compute-0-36 CANCELLED AT 2020-05-18T10:45:16 DUE TO TIME LIMIT ***

then your job ran out of time and was killed. Maybe this was a week long GMIN run that ran out of time 10 steps from the end. How annoying. Time to run it again from the beginning with a longer time limit, because there is no GMIN.dump file in the submission directory? Hardly... In our job submission script, we only deleted the temporary directory after the successful completion of the job. Therefore if the job did not successfully complete, the directory and its contents are still there. The information at the top of the slurm-<job_ID>.out file will help you find them. We created the temporary directory at

/scratch/jwrm2/214233

To access this from the head node, we need to know what node it was created on. That's why we ran 'hostname' at the start of the job. Let's say the output of that was

compute-0-17.local

We can access our files at

/nodescratch/compute-0-17/jwrm2/214233

Note that normal Bash tab completion of the directory name may not work here. That doesn't necessarily mean the directory doesn't exist, it's just a peculiarity of NFS automount.

It would be polite to delete the temporary directory after recovering any files you need.

==A smarter way to get your files==

Although recovering files from /nodescratch/ is fine, if you've every had 52 GMIN jobs terminate and needed to recover their GMIN.dump files for restarting, it gets a little tedious. Fortunately there is a better way, leveraging the signal mechanism of Linux. Here is an example:

#!/bin/bash
#SBATCH -J GMIN_LJ38
#SBATCH --time=5:00
#SBATCH -n1
#SBATCH --signal=B:USR1@120

signal_handler()
{
# Timeout commands go here
echo "Caught USR1 signal"
cp GMIN.dump logfile $SLURM_SUBMIT_DIR
exit
}
trap 'signal_handler' USR1

echo Starting job $SLURM_JOB_ID
hostname

TMP=/scratch/jwrm2/$SLURM_JOB_ID
mkdir -p $TMP
cp $SLURM_SUBMIT_DIR/data $SLURM_SUBMIT_DIR/coords $TMP/
cd $TMP

/home/jwrm2/bin/GMIN > logfile &
wait

cp -p lowest logfile $SLURM_SUBMIT_DIR && rm -rf $TMP
cd $SLURM_SUBMIT_DIR

This script will run any commands enclosed in braces at '# Timeout commands go here', 120 seconds before the job times out. It will terminate itself immediately after completing those commands, so you will not get a 'CANCELLED' message in the slurm-<job_ID>.out file (but only because of the 'exit', otherwise it would continue execution after the 'wait', which is probably not what you want). You can use it to copy back recovery files like GMIN.dump, or the updated PATHSAMPLE database, before the job terminates. If 120 seconds is not enough time for your timeout commands, change the 120 in the line '#SBATCH --signal=B:USR1@120'. The temporary directory will not be deleted if the timeout is reached, although you could put 'rm -rf $TMP' in the timeout section if you were really confident that you were grabbing everything you needed. You should occasionally go through /nodescratch/ and delete any leftover temporary directories that may be hanging around.

===Theory===

You don't need to understand how the above example works to use it, but we're all curious scientists, aren't we? The Linux kernel can communicate with a running process by sending it 'signals'. Common ones include 'SIGTERM' and 'SIGKILL'. In fact, when your job reaches the time limit, it gets sent a 'SIGTERM' signal, telling it to terminate. If the process has not set up anything specific to do on receiving a particular signal, the default action is to exit. However, most signals can have custom responses set up. An exception is 'SIGKILL', for which immediate termination cannot be overridden.

The line '#SBATCH --signal=B:USR1@120' tells SLURM to please send the signal 'USR1' (user specified signal number 1) to the job 120 seconds before reaching the time limit. 'USR1' is not sent by any system processes, so we're free to use it as we wish. Next we create a Bash function, 'signal_handler()', with our sequence of commands. We tell the Bash script to jump to our function on receiving the 'USR1' signal, overriding the default exit behaviour (trap 'signal_handler' USR1).

The final finicky bit is how we run our main programme. If we ran 'GMIN', the signal would get sent to GMIN, as the active process, rather than Bash. GMIN wouldn't know what to do with it, so would immediately exit. Not good. By running 'GMIN &' instead, the signal goes to Bash. However, if we did just that, the job would quickly reach the end after launching GMIN, whcih would terminate the job and all it's children, including GMIN. Oops. By writing 'wait', we tell Bash to go into an 'interruptible sleep', meaning it will sit there not doing anything until it receives a signal. Therefore it is able to exceute the commands in the 'signal_handler()' function on receiving the 'USR1' signal. As a matter of interest, execution of the main script then resumes where it left off before the signal, but the wait has now been completed (it only waits for the first signal) and execution proceeds to the 'cp' line. Putting 'exit' at the end of the function means we don't junp back to after wait. We could have put another 'wait' instead of 'exit', but that would only be useful if GMIN finished in the short period of time between completing the function and reaching the job time limit. Better to just save 2 minutes of CPU time by exiting immediately.

How does our job finish normally, if we've put it to sleep then? Well, when GMIN exits it sends, you guessed it... a signal, in this case 'SIGCHLD'. That will wake up Bash from its wait and the script will proceed to completion.

Comprehensive Contents Page

2020-05-26T13:17:24Z

Jwrm2: /* Cluster queues */ Added introductory SLURM page.

This page is designed to organise all of the pages on this wiki, as well as provide other useful links. Note that some pages may appear under more than one heading.

== Getting Started ==
[[Wales Group]] provides good step-by-step instructions. Relevant pages are:

=== Acquiring and compiling the group software ===
* [[SVN setup]]
* [[Wales Group Version control]] - to keep the code standardised.
* Theory Sector [http://wwmm.ch.cam.ac.uk/wikis/cuc3/index.php/SVN_Page SVN Page] - some useful general information on SVN commands.
* [[Compiling Wales Group codes using cmake]] - CMake (Cross-platform Make) allows us to compile and test the group codebase regardless of platform. This page provides crucial information how to compile using cmake.
* [[ElaborateDiff]]

=== Maintaining code health ===
* [[Jenkins CI]] - explains Jenkins, which we use to download our code and compile each of our targets with each of the compilers every night.
* https://wales-jenkins.ch.cam.ac.uk/ - log for our Jenkins tests.
* [[Branching and Merging]]
* [[Cmake interface building]]
* [[Installing python modules]]
* [[Revamping the modules system]]

=== Collaborators without access to the SVN repository ===
For licensing reasons, some code cannot be included in the Wales Group public tarball.
* http://www-wales.ch.cam.ac.uk/svn.tar.bz2 - Wales group public tarball. Includes [[GMIN]], [[OPTIM]] and [[PATHSAMPLE]].
If a collaborator has a [[CHARMM]] or [[AMBER]] licence, we do maintain separate tarballs which include the [[CHARMM]], [[AMBER]] and [[CHARMM]]+[[AMBER]] source and interfaces. These are not linked anywhere on the website and require a username ('''wales''') and password ('''group''') to download:

* [http://www-wales.ch.cam.ac.uk/CHARMM/svn.CHARMM.tar.bz2 CHARMM]
* [http://www-wales.ch.cam.ac.uk/AMBER/svn.AMBER.tar.bz2 AMBER]
* [http://www-wales.ch.cam.ac.uk/both/svn.both.tar.bz2 AMBER+CHARMM]

=== Running on Windows ===
Not particularly recommended.
* [[Running Wales Group software on Windows 7]]

== Wales Group Programs ==

=== Programs ===
* [[GMIN]]: A program for finding global minima and calculating thermodynamic properties from basin-sampling.
* [[OPTIM]]: A program for optimizing geometries and calculating reaction pathways.
* [[PATHSAMPLE]]: A driver for OPTIM to create stationary point databases using discrete path sampling and perform kinetic analysis.
* [[Pele]]: Python energy landscape explorer. A pythonic rewrite of some core functionality of GMIN, OPTIM, and PATHSAMPLE. Can be very useful for visualizing your system and for rapidly implementing and testing new ideas.

=== Curated Examples ===
* https://github.com/wales-group/examples - set of tutorials detailing how to use GMIN, OPTIM and PATHSAMPLE. Essential for beginners.
* http://www-wales.ch.cam.ac.uk/VM/Wales_Group_VM.ova - Pre-prepared teaching virtual machine. This contains the code and examples.
* https://www.virtualbox.org/wiki/Downloads - This is required if using the VM above.
* https://github.com/wales-group/examples.git - Alternatively, you can run the examples on your own machine. To get hold of the relevant files:
<pre>
git clone https://github.com/wales-group/examples.git
</pre>

== Useful Notes on Wales Group Programs and Subroutines ==
=== [[GMIN]] ===
* [[Adding a model to GMIN]] - rough outline of the subroutines that need to be changed to add a new model to GMIN
* [[Compiling Wales Group codes using cmake | Compiling GMIN using cmake ]]
* [[Selecting search parameters for GMIN]]
* [[Global optimization of biomolecules using CHARMM]]
* [[Global optimization of biomolecules using AMBER9]]
* [[Global optimization of biomolecules using AMBER9 with Structural Restraints]]
* [[Calculating binding free energy using the FSA method]]
* [[Restarting a GMIN run from a dump file]]
* [[Using the implicit membrane model IMM1]]
* [[Running a Go model with the AMHGMIN]]
* [[Running a G\=o model with the AMHGMIN]]
* [[Ligand binding-mode searches with HBONDMATRIX]]
* [[Compiling and using GMIN with QUIP]]
* [[Using GMIN and OPTIM with GPUs]]
* [[Using GMIN to generate endpoints]]
* [[Using GMIN to generate endpoints (CHARMM)]]
* [[Generating a GMIN Eclipse project]]
* [[Mutational BH steps]]
* [[Biomolecules in the energy landscape framework]]
* [[DMAGMIN setup]]
* [[Keywords]]
* [[PYGMIN & DMACRYS]]
* [[Rotamer moves in AMBER]]
* [[Python interface for GMIN/OPTIM]]

==== Scripts ====
* [[makerestart]]: A bash script to automatically set up a GMIN restart run
* [[progress]] A bash script to tell you the % completion of a GMIN job and give an estimated time remaining

==== Useful info for coding GMIN ====
* [[Program flow]] - contains information about what the various files in GMIN do and what order they're called.
* [[amberinterface]]

==== Projects ====
* [[GMIN MOVES module]]
* [[GMIN SANITY module]]
* [[GMIN TESTS module]]
* [[CAMSHIFT]]

=== [[OPTIM]] ===
* [[Adding a model to OPTIM]] - rough outline of the subrounties that need to be changed to add a new model to OPTIM
* [[Adding partially finished OPTIM stationary points to a PATHSAMPLE database]]
* [[perm-pdb.py]]: A python program that creates a ''perm.allow'' file for use with [[OPTIM]] and [[PATHSAMPLE]].
* [[visualising normal modes using VMD and OPTIM]]
* [[Compiling Wales Group codes using cmake | Compiling OPTIM using cmake ]]
* [[OPTIM/Q-Chem Tutorial]]
* [[OPTIM and PY ellipsoids tutorial]]
* [[OPTIM output files]]
* [[Minimizing a structure using OPTIM and AMBER9]]
* [[Minimizing a structure using OPTIM and CHARMM]]
* [[Creating movies (.mpg) of paths using OPTIM]]
* [[Performing a normal mode analysis of a biomolecule using OPTIM (AMBER and CHARMM)]]
* [[Debugging odd transition states in OPTIM]]
* [[Connecting two minima with a pathway]] - step by step
* [[Compiling and using OPTIM with QUIP]]
* [[Running an Gaussian03 interfaced OPTIM job]]
* [[The effect of calculating less than the maximum number of eigenvalues using ENDHESS n]]
* [[Biomolecules in the energy landscape framework]]
* [[BLJ60 example setup]]
* [[Finding an initial path with OPTIM and starting up PATHSAMPLE]]
* [[Finding an initial path with OPTIM and starting up PATHSAMPLE (CHARMM)]]
* [[Python interface for GMIN/OPTIM]]
* [[Thomson problem in OPTIM]]
* [[Instanton tunneling and classical rate calculations with OPTIM]]
* [[Loading OPTIM's min.data.info files into PATHSAMPLE]]
* [[common setup problem : No Frequency Warning]]

=== [[PATHSAMPLE]] ===
* [[Adding a model to PATHSAMPLE]] - rough outline of the subrounties that need to be changed to add a new model to PATHSAMPLE
* [[Alternatively, making the initial path with PATHSAMPLE itself]]
* [[Alternatively, making the initial path with PATHSAMPLE itself (CHARMM)]]
* [[perm-pdb.py]]: A python program that creates a ''perm.allow'' file for use with [[OPTIM]] and [[PATHSAMPLE]].
* [[dijkstra_test.py]]: A python script to test whether the information in pairlist and ts.data connects the A and B set. (If not, PATHSAMPLE will not work without actually exiting.)
* [[Compiling Wales Group codes using cmake | Compiling PATHSAMPLE using cmake ]]
* [[IMPORTANT: Using PATHSAMPLE safely on sinister]]
* [[Adding a model for PATHSAMPLE]]
* [[List of output files for PATHSAMPLE]]
* [[Using BHINTERP to find minima between two end points]]
* [[Finding an initial path between two end points (minima)]]
* [[Adding partially finished OPTIM stationary points to a PATHSAMPLE database]]
* [[Optimising a path]]
* [[Fine tuning UNTRAP]] - ensuring that it picks sensible minima
* [[Calculating rate constants (GT and fastest path)]]
* [[Calculating rate constants (SGT, DGT, and SDGT)]]
* [[Identifying the k fastest paths between endpoints using KSHORTESTPATHS]]
* [[Removing minima and transition states from the database]]
* [[Relaxing existing minima with new potential and creating new database]]
* [[Relaxing existing transition states with new potential and creating new database]]
* [[If things go wrong...]]
* [[If you lost file min.data, but still you have points.min]]
* [[path.info file is not read, causes PATHSAMPLE to die]]
* [[BLJ60 example setup]]
* [[When PATHSAMPLE finds a connected path, but using DIJKSTRA 0 fails to find the connected path]]
* [[Biomolecules in PATHSAMPLE]]
* [[Biomolecules in the energy landscape framework]]
* [[Expanding the kinetic transition network with PATHSAMPLE]]
* [[Expanding the kinetic transition network with PATHSAMPLE (CHARMM)]]
* [[Finding an initial path with OPTIM and starting up PATHSAMPLE]]
* [[Finding an initial path with OPTIM and starting up PATHSAMPLE (CHARMM)]]
* [[Pathsampling short paths]]
* [[Pathsampling short paths (CHARMM)]]
* [[Loading OPTIM's min.data.info files into PATHSAMPLE]]
* [[Connecting Sub-databases]]

=== [[Notes on MINPERMDIST | MINPERMDIST]] ===

=== [[Quasi-continuous interpolation for biomolecules | QCI]] ===

== Non-Group Software ==

=== [[AMBER]] ===
Molecular dynamics simulation program and associated force fields.
* [http://ambermd.org/ AMBER]
* [http://ambermd.org/tutorials/ AMBER tutorials] - recommended reading for '''ANYONE''' using AMBER!
* [[Notes on AMBER 12 interface]]
* [[Using AMBER 14 on the GPU and compute clusters]]
* [[Generating parameters using AMBER's built in General Forcefield (gaff)]]
* [[Generating parameters using RESP charges from GAMESS-US]]
* [[Simple scripts for LEaP to create topology and coordinate files]]
* [[Preparing an AMBER topology file for a protein system]] - step by step guide
* [[Setting up]] - step by step guide to prepare and then symmetrise a simple (protein-only) system
* [[Using Molfacture to edit molecules and add hydrogens]]
* [[Preparing an AMBER topology file for a protein plus ligand system]] - step by step guide
* [[Symmetrising AMBER topology files]] - step by step guide for symmetrising a complex protein+ligand system
* [[Producing a PDB from a coordinates and topology file]] - using ''amdpdb''
* [[Running GMIN with MD move steps AMBER]]
* [[Performing a normal mode analysis of a biomolecule using OPTIM (AMBER and CHARMM)]]
* [[Evaluating different components of AMBER energy function with SANDER]]
* [[Mutational BH steps]]
* [[REMD with AMBER]]
* [[Performing a hydrogen-bond analysis]]
* [[Alternatively, making the initial path with PATHSAMPLE itself]]
* [[Biomolecules in the energy landscape framework]]
* [[perm-prmtop.py]] - A python program that converts an AMBER9 topology file into one with a symmetrised potential with respect to exchange (updated for AMBER12 and ff14SB).
* [[Rotamer moves in AMBER]]
* [[Creating mismatched DNA duplex using NAB]]

=== [[aux2bib]] ===
To generate a bib file containing only the entries cited in a given .tex file from a larger bib or multiple bib files.
* [https://ctan.org/pkg/bibtools Get script here]

=== [[CamCasp]] ===
Cambridge package for Calculation of Anisotropic Site Properties
From Anthony Stone's website: 'CamCASP is a collection of scripts and programs written by Dr Alston Misquitta and myself for the calculation ab initio of distributed multipoles, polarizabilities, dispersion coefficients and repulsion parameters for individual molecules, and interaction energies between pairs of molecules using SAPT(DFT).'
* [http://www-stone.ch.cam.ac.uk/programs.html CamCASP home]
* [[CamCASP/Programming]]
* [[CamCASP/Programming/5/example1]]
* [[CamCASP/Notes]]
* [[CamCASP/Bugs]]
* [[CamCASP/ToDo/diskIO]]
* [[CamCASP/ToDo/Memory]]
* [[CamCASP/CodeExamples/DirectAccess]]

=== [[CPMD]] ===
Implementation of DFT for ''ab-initio'' molecular dynamics.
* [http://www.cpmd.org/ Home Page]
* [[CPMDInput]]

=== [[CHARMM]] ===
Molecular dynamics simulation program and associated force fields.
* [https://www.charmm.org/charmm/?CFID=65f7b3aa-8037-452a-bcd1-7583dd83a087&CFTOKEN=0 CHARMM]
* [[Generating pdb, crd and psf for a peptide sequence]]
* [[Converting between '.crd' and '.pdb']]
* [[Calculating energy of a conformation]]
* [[Calculating molecular properties]]
* [[Calculating order parameters]]
* [[CAMSHIFT]]
* [[Setting up (CHARMM)]] - step by step guide to prepare and then symmetrise a simple (protein-only) system
* [[If you need to change the number of atoms (e.g. making a united-atom charmm19 .crd file, or if atoms are missing)]]
* [[Performing a normal mode analysis of a biomolecule using OPTIM (AMBER and CHARMM)]]
* [[Minimizing a structure using OPTIM and CHARMM]]
* [[Alternatively, making the initial path with PATHSAMPLE itself (CHARMM)]]
* [[Expanding the kinetic transition network with PATHSAMPLE (CHARMM)]]
* [[Finding an initial path with OPTIM and starting up PATHSAMPLE (CHARMM)]]
* [[Pathsampling short paths (CHARMM)]]

=== [[disconnectionDPS]] ===
Produces disconnectivity graphs from min.data and ts.data files. This is included in the Wales group public tarball.
* [[Constructing Free Energy Disconnectivity Graphs]]

=== [[DMACRYS]] ===
Package which models crystals of rigid molecules.
* [http://www.chem.ucl.ac.uk/cposs/dmacrys/index.html Home Page]
* [[DMACRYS interface]]
* [[DMAGMIN setup]]
* [[PYGMIN & DMACRYS]]

=== [[GAMESS]] ===
General ''ab initio'' quantum chemistry package.
* [https://www.msg.chem.iastate.edu/gamess/ GAMESS]

=== [[Gaussian]] ===
General purpose package for computational chemistry calculations.
* [[Running an Gaussian03 interfaced OPTIM job]]

=== [[gnuplot]] ===
Open source graphing program.
* [http://www.gnuplot.info/ gnuplot]
* [[Plotting a quick histogram in gnuplot using the raw data]]
* [[Plotting data in real time]]
* [[Linear and non-linear regression in gnuplot]]

=== [[GROMACS]] ===
Molecular dynamics package.
* [[Installing GROMACS on Clust]]
* [http://www.mdtutorials.com/gmx/ External tutorials]
* [http://www.gromacs.org/Documentation/Tutorials More external tutorials]

=== [[HiRE-RNA]] ===
High-res course-grained energy model for RNA.
* [https://pubs.acs.org/doi/10.1021/jp102497y Explanatory Paper]

=== [[latex2html]] ===
Script which converts latex documents into HTML pages.
* [https://www.latex2html.org/ Get script here]

=== [[MMTSB-toolset]] ===
Group of perl scripts which can be used to setup and run energy minimization, structural analysis and MD with CHARMM or AMBER.
* [http://feig.bch.msu.edu/mmtsb/Main_Page Documentation]
* [http://www.mmtsb.org/workshops/mmtsb-ctbp_2006/Tutorials/WorkshopTutorials_2006.html External tutorials]
* [[Installing and setting up the MMTSB toolset]]
* [[REX (Replica EXchange MD) with the MMTSB-toolset]]

=== [[Simulations using OPEP | OPEP]] ===
OPEP is a coarse-grained force field providing a potential for proteins and RNA.
* [http://opep.galaxy.ibpc.fr/ OPEP file generator here]
* [[Biomolecules in the energy landscape framework]]

=== [[pgprof]] ===
Profiler for portland-compiled codes
* [[Portland compiler fails trying to allocate an unexpectedly large amount of memory: issue with large arrays]]

=== [[Pymol]] ===
Molecular visualisation program.
* [https://pymol.org/2/ PyMOL]
* [https://pymolwiki.org/index.php/Main_Page PyMOL Community Wiki]
* [[loading AMBER prmtop and inpcrd files into Pymol]]
* [[producing sexy ray-traced images]]
* [[advanced colouring]]
* [[Installing python modules]]
* [[PYGMIN & DMACRYS]]
* [[path2pdb.py]] - A python program to convert ''path.info'' to ''path_all.pdb'' - you can easy visualize your path in VMD :)
* [[extractedmin2pdb.py]]: A python program to convert ''exctractedmin'' to PDB format
=== [[VASP]] ===
OPTIM has an interface to VASP, which is installed on CSD3. In collaboration with Bora Karasulu the interface has been updated to use VASP format POSCAR input files for both single- and double-ended optimisations and path searches. The OPTIM odata file requires a line like

VASP 'mpirun -ppn 16 -np 16 /home/bk393/APPS/vasp.5.4.4/with-VTST/bin/vasp_std > vasp.out'

POSCAR files can be visualised using ase, the Atomic Simulation Environment, which can be accessed on volkhan via

module load anaconda/python3/5.3.0

pip install ase --user

ase-gui POSCAR1.vasp &

which assumes that ~/.input/bin is in your $PATH environment variable.

=== [[VMD]] ===
Molecular visualisation program.
* [http://www.ks.uiuc.edu/Research/vmd/current/ug/ug.html Documentation]
* [http://www.ks.uiuc.edu/Training/Tutorials/vmd/tutorial-html/index.html External tutorials]
* [[using VMD to display and manipulate '.pdb' files]]
* [[loading coordinate files into VMD with the help of an AMBER topology file]] e.g. to visualise the results of a GMIN run using AMBER9
* [[visualising normal modes using VMD and OPTIM]]
* [[path2pdb.py]]: A python program to convert ''path.info'' to ''path_all.pdb'' - you can easy visualize your path in VMD :)
* [[path2xyz.py]]: A python program to convert ''path.info'' to ''path_all.xyz''
* [[extractedmin2pdb.py]]: A python program to convert ''exctractedmin'' to PDB format
* [[Useful .vmdrc file]]
* [[plotGMINms.tcl]]: a tcl script for plotting ellipsoids in VMD.
* [[VMD script to annotate each frame of a trajectory]]

=== [[xfig]] ===
Open source vector graphics editor
* [https://ctan.org/tex-archive/support/epstopdf/ Convert eps to pdf]

=== [[Xmakemol]] ===
Program for visualising atomic and molecular systems.
* [https://www.nongnu.org/xmakemol/ XMakemol]

=== [[xmgrace]] ===
2D plotting tool.
* [http://exciting-code.org/xmgrace-quickstart Xmgrace]

== Theoretical/Mathematical Notes ==

* [[Density of states and thermodynamics from energy distributions at different temperatures]]
* [[Ellipsoid.model]]
* [[Ellipsoid.model.xyz]]
* [[Ellipsoid.xyz]]
* [[Gencoords]]
* [[GenCoords]]
* [[GenCoords Models]]
* [[Rotamer moves in AMBER]]
* [[Thomson problem in OPTIM]]

=== Angle-axis notes ===

* [[Angle-axis framework]]
* [[Computing normal modes in angle-axis]]

=== Rigid Bodies ===

* [[Automatic Rigid Body Grouping]]
* [[Rigid body input files for proteins using genrigid-input.py]]
* [[Local Rigid Body Framework]]
* [[Local rigid body in OPTIM]]

== Useful Scripts ==
* [[perm-prmtop.py]]: A python program that converts an AMBER9 topology file into one with a symmetrised potential with respect to exchange (updated for AMBER12 and ff14SB).
* [[perm-pdb.py]]: A python program that creates a ''perm.allow'' file for use with [[OPTIM]] and [[PATHSAMPLE]].
* [[path2pdb.py]]: A python program to convert ''path.info'' to ''path_all.pdb'' - you can easy visualize your path in VMD :)
* [[path2xyz.py]]: A python program to convert ''path.info'' to ''path_all.xyz''
* [[dijkstra_test.py]]: A python script to test whether the information in pairlist and ts.data connects the A and B set. (If not, PATHSAMPLE will not work without actually exiting.)
* [[extractedmin2pdb.py]]: A python program to convert ''exctractedmin'' to PDB format
* [[colourdiscon.py]]: A python program for sorting input for disconnectivity graphs
* [[pdb_to_movie.py]]: A python program to create an AMH movieseg file from a PDB file
* [[makerestart]]: A bash script to automatically set up a GMIN restart run
* [[progress]] A bash script to tell you the % completion of a GMIN job and give an estimated time remaining
* [[recommended bash aliases]]
* [[David's .inputrc file]]
* [[Useful .vmdrc file]]
* [[Density of states and thermodynamics from energy distributions at different temperatures]]
* [[GenCoords]]: A fortran program to generate coarse grain building blocks and initial coords using a set of geometric models.
* [[plotGMINms.tcl]]: a tcl script for plotting ellipsoids in VMD.
See also the SCRIPTS/ directory in the SVN repository!
* [[Computing CHARMM FF energy using GMIN, MMTSB and CHARMM]] - Computes the Charmm FF energy of the same structure. Useful for cross-validating force field settings in GMIN data file, CHARMM input file and MMTSB options.
* [[Automatic Rigid Body Grouping]]
* [[ElaborateDiff]]
* [[Parameter-scanning script]]
* [[Pdb to movie.py]]
* [[VMD script to annotate each frame of a trajectory]]

== Useful links ==
* [http://www.ch.cam.ac.uk/computing/theory-compute-clusters The Theory Compute Clusters support page]. Contains useful cluster specific information, including example job submission scripts.

* A useful website which contains AMBER (GAFF) and OPLS parameters for small molecules. http://virtualchemistry.org/gmld.php . This could save us lot of time while trying to derive parameters on our own. If you are lucky, the molecule of your interest may already be there in the existing database. The topology files are in GROMACS format but possibly can be converted into AMBER parameter files. (script anyone ?)

* The moving-domain QM/MM method developed by Victor Batista's group http://gascon.chem.uconn.edu/software. This approach can be used in the derivation of charges for large proteins and nucleic acids, where a full-fledged ONIOM based calculation is comptutationally prohibitive. It has been applied to systems like the Gramicidin ion channel and Photosystem II.

== Miscellaneous ==
* [[Animated GIF on the group website]]
* [[Backup strategy]]
* [[Chain crossing]]
* [[Computer Office services]]
* [[Computing values only once]]
* [[Decoding heat capacity curves]]
* [[Differences from Clust]]
* [[Fixing thunderbird links]]
* [[If you need to change the number of atoms (e.g. making a united-atom charmm19 .crd file, or if atoms are missing)]]
* [[Intel Trace Analyzer and Collector]]
* [[LDAP plans]]
* [[Lapack compilation]]
* [[Mek-quake Queueing system]]
* [[Mek-quake initial setup notes]]
* [[New mek-quake]]
* [[Maui compilation]]
* [[Torque and Maui]]
* [[Mercurial]]
* [[Migrating to the new SVN server]]
* [[NECI Parallelization]]
* [[Optimization tricks]]
* [[Other IT stuff]]
* [[Porfuncs Documentation]]
* [[Progress]]
* [[Proposed changes to backup and archiving]]
* [[Rama upgrade]]
* [[Remastering Knoppix]]
* [[See unpacked nodes]]
* [[Tardis scheduling policy]]
* [[Zippo Sicortex machine]]
* [[Beginner's guide to working in Wales group]]

== Useful linux stuff ==

===Basics===
* [[basic linux commands everyone should know!]]
* [[piping and redirecting output from one command or file to another]] - how to save yourself hours!
* [[bash loop tricks]]
* [[bash history searching]]

===Remote access===
* [[setting up aliases to quickly log you in to a different machine]]
* [[transfering files to and from your workstation]] -using ''scp'' or ''rsync''
* [[using 'ssh-keygen' to automatically log you into clusters from your workstation]] (no more typing in your password!)
* [[mounting sharedscratch locally]]

===Find and replace===
* [[short 'sed' examples]]
* [[quick guide to awk]]
* [[short 'awk' examples]]

===File manipulation===
* [[sorting a file by multiple columns]]
* [[using tar and gzip to compress/uncompress files | using tar and bzip2 to compress/uncompress files]]
* [[conversion between different data file formats]] -'almost one-line' scripts
* [[conversion between different image file formats]] - the ''convert'' command
* [[removing an excessive number of files from a directory - when 'rm' just isn't enough]]

===Cluster queues===
* [[submitting jobs, interactively or to a cluster queue system]]
* [[identifying job on a node]] - if you need to kill only one of few running jobs
* [[getting started with SLURM]]
* [[a guide to using SLURM to run PATHSAMPLE]]
* [[a guide to using SLURM to run GPU jobs on pat]]
* [[managing interactive jobs on cluster]]

===Miscellaneous/uncategorised===
* [[installing packages on your managed CUC3 workstation]]
* [[running programs in the background]] - so you can use your shell for other things at the same time
* [[finding bugs in latex documents that will not compile]]
* [[printing files from the command line using 'lpr']]
* [[uploading non image files to the wiki]]

== Compiler Flags ==

* [[Compiler Flags]]
* [[Blacklisting Compilers]]
* [[Lapack compilation]]
* [[Pdb to movie.py]]
* [[Portland compiler fails trying to allocate an unexpectedly large amount of memory: issue with large arrays]]

== SuSE ==

* [[Upgrading destiny]]
* [[Upgrading sword]]
* [[SuSE 10.1 workstation image]]
* [[SuSE 10.2 workstation image]]
* [[SuSE 10.3 workstation image]]
* [[SuSE 11.1]]

--[[User:adk44|adk44]] 17.00, 9 May 2019 (BST)

Getting started with SLURM

2020-05-26T13:11:43Z

Jwrm2: Created page with basic information about SLURM.

For running computationally expensive tasks for long periods of time, you will want to create a job and run it on a cluster. This page is intended to teach you everything required to run your first job on a SLURM CPU cluster (sinister, volkhan). For convenience, we will refer primarily to sinister, but the information is transferrable to other SLURM clusters. This page does not contain information about PBS (dexter). If you are running GPU (pat) jobs, you will need additional information from [[A guide to using SLURM to run GPU jobs on pat | here]].

==Basic terminology==

A cluster is a large computer that has many processors. The processors are grouped into nodes, which operate semi-independently of each other. Each node has it's own disk (accessed at /scratch/) and memory, but is capable of communicating with other nodes through a network. When you login to sinister with SSH, you are logged into the head node. This is a special node designed for interacting with users. The other nodes are compute nodes, designed for running long and intensive tasks. SLURM (Simple Linux Utility for Resource Management), now officially called the Slurm Workload Manager, is a programme that manages the compute nodes, allocating resources, starting and ending jobs, and managing the queue. To get the compute nodes to do anything, you need to ask through SLURM.

==How to not be anti-social==

When you are using sinister, the most important thing to be aware of is that other people are using it too. You need to be careful to not act in a way that impedes other users from getting on with their work. Primarily this means not excessively using the head node, as if you do other users will get a very slow response time on their SSH and will not be able to operate. There are some specific things to avoid.

*Do not run long expensive tasks on the head node: that is what the compute nodes are for. Anything longer than a few minutes should not be run on the head node. For a short expensive task (like compiling GMIN), prefix your command with 'nice', which tells the operating system to give your process a lower priority, meaning any simple commands other people might be running are not slowed down. You are not likely to notice a significant impact on the speed of your process by doing this.

$ nice make

*Do not rapidly copy a lot of data over the network. Although the nodes are networked together and moving data between them is a simple process (using NFS, the Network File System), it is relatively slow and is computationally expensive. If you constantly move data over the NFS, for example by writing a verbose log file, other users will notice. This is most likely to be an issue when writing to the directory /sharedscratch/, which is a large working space partition accessible over the network from any partition. Each node has its own space, accessed at /scratch/ on the node, or through /nodescratch/ from the head node. Best practices to avoid this problem are detailed below, but it is mentioned here due to its great importance.

*Be realistic about the requirements for your job. The cluster is a shared resource that is not infinite. To make the most efficient use of the resource, do not do things like request 32 cores for a job that can only make use of 4, or request a time limit of 1 week for a job that will only take 1 hour. The better the information you give to the queuing system, the better it can serve you.

==Submitting a job==

To submit a job, you create a job submission script, here we'll call it submit.sh, and then run

$ sbatch submit.sh

which tells SLURM to execute the contents of your submission script. When you do this, SLURM will first look through any information at the top of the job script for configuration instructions like the job time limit. Then it will place your job into a queue. When the necessary resources become available, any commands in your submission script will be executed in sequence on a compute node. Let's have a look at a simple submission script for running GMIN.

#!/bin/bash
#SBATCH -J GMIN_LJ38
#SBATCH --time=1:00:00
#SBATCH -n1

echo Starting job $SLURM_JOB_ID
hostname

TMP=/scratch/jwrm2/$SLURM_JOB_ID
mkdir -p $TMP
cp $SLURM_SUBMIT_DIR/data $SLURM_SUBMIT_DIR/coords $TMP/
cd $TMP

/home/jwrm2/bin/GMIN > logfile

cp -p lowest logfile $SLURM_SUBMIT_DIR && rm -rf $TMP
cd $SLURM_SUBMIT_DIR

We'll go through this line by line. The first line (affectionately known as the 'shebang') tells SLURM what shell to use to interpret the commands. You don't have to use Bash, but it's what most people are used to, so it's recommended if you want anyone else to be able to help you.

Next we have a series of configuration commands for SLURM. Each of these starts with #SBATCH and they are ignored by Bash. Some of the most useful are:

* -J <name> The name of the job, that will show up in diagnostic information. Make your life easier by choosing a suitably descriptive one.
* --time=hh:mm:ss The maximum amount of time your job will run for. If this time limit is reached and the job hasn't finished, SLURM will kill it. It is in yours and everyone else's interest to make this a fairly accurate estimate of how long the job will actually take (plus a little bit).
* --ntasks=<number> The number of processors your job requires.
* --nodes=<number> The number of nodes your job requires.
* --mem=<value> The amount of memory your job needs. The default is often fine.

Consult the SLURM documentation [https://slurm.schedmd.com/sbatch.html] for a full list. Here, we request one processor, for a maximum time of 1 hour, and we set the name to 'GMIN_LJ38'.

The remaining commands are executed by Bash once the job has begun. First we write out some diagnostic information: the ID of the job and which compute node it is running on. This is written to a file slurm-<job_ID>.out in the directory we run sbatch from, for example

$ cat slurm-214233.out
Starting job 214233
compute-0-17.local

The first line is a useful check that the job actually began. Knowing which node your job ran on can be useful if it terminates early/unexpectedly.

The next section of the submission script is about reducing the NFS load. Instead of running the job on /sharedscratch/, which the node accesses over the NFS, we create a directory in the node's own /scratch/ space. Change 'jwrm2' to your own username. We copy only the files GMIN needs in order to run, then change to the new directory. $SLURM_SUBMIT_DIR is a variable holding the directory from whcih sbatch was run.

Next we actually run the programme. We redirect the GMIN output to a log file. If you do not do this redirection, output will instead go to the slurm-<job_ID>.out file, which most likely resides on /sharedscratch/.

Finally, we have clean up actions to be run after GMIN has finished. We copy back only the data we are interested in to the directory from which sbatch was run, then delete the temporary directory we created on the nodescratch. Note the '&&' syntax, which means 'run the command after the && only if the command before the && was successful'. In this case, it means that the temporary directory will not be deleted if we couldn't copy those files back.

==The queuing system==

When you submit your job with sbatch, it may not be run instantly. It is quite possible that the the resources your job needs are currently being used by someone else. Your job enters the queue. You can see the queue by typing

$ squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
214233 CLUSTER GMIN_LJ38 jwrm2 PD 0:00 1 (Resources)
214207 CLUSTER GLP-1 kr366 R 29:31 2 compute-0-[22-23]
214112 CLUSTER pentamer_1_1292 ld506 R 1-02:03:44 1 compute-0-12

There is a lot of information here. The first column gives the ID of the job. The next column is not usually relevant. Then we see the name of the job (assigned with -J in the submission script) and the user who submitted the job. The ST column tells you the current status of the job: 'PD' means 'pending', ie. the job is waiting to run, but is not yet running; 'R' means 'running'. We also have the time the job has been running for, the number of nodes the job is using and which they are. If the job is not running, the final column may display a reason. If it shows (Resources), then your job will run as soon as the necessary resources are available. If it instead shows (Priority), then there is another waiting job in the queue that will be run before yours. If you only want to see information about your jobs (or any other user's), you can use the -u option:

$ squeue -u jwrm2

will only show the jobs submitted by the user jwrm2.

How does SLURM decide which jobs to run? Principally, it works out a user's priority by a fairshare system: the more compute time you have recently used, the lower your priority will be. You can see everyone's fairshare values by looking at the final column of the output of

$ sshare -a

Higher numbers mean a higher priority for that user. However, SLURM will also try to fit smaller jobs into gaps. If there is a 12 node job waiting from a user with a very high priority, but only one node is currently available, a short one node job from a user with low priority may be able to fit in. Therefore it is in your interest to make realistic estimates of the resources your job needs: it may end up running sooner.

==Oops, I just submitted the wrong job==

You can abort a job when it is waiting to run or when it is running. Find the job ID, either by looking at squeue or the output file in the submission directory. Let's say it was 214233:

$ scancel 214233

will immediately kill the job.

==Running PATHSAMPLE with SLURM==

PATHSAMPLE runs a little differently from standard GMIN: it launches new OPTIM jobs and sends them to the other processors available. This is pretty much taken care of if you include 'SLURM' in the pathdata file, but there are a couple of extra considerations. Firstly, PATHSAMPLE needs a nodes.info file with information about the available processors. Generate this file at the start of you submission script with:

echo $SLURM_NTASKS > nodes.info
srun hostname >> nodes.info
echo $USER >> nodes.info
pwd >> nodes.info

Note the use of srun here. It means: run this command on every one of the available processors.

Secondly, we need to decide whether PATHSAMPLE itself will be run from /sharedscratch/ (not the OPTIM jobs, PATHSAMPLE will always use /nodescratch/ for those). Running PATHSAMPLE from /sharedscratch/ is simpler and it means that if the job terminates before PATHSAMPLE completes all the requested cycles, the database on /sharedscratch/ will be in the most current state. Here is an example PATHSAMPLE script that does this

#!/bin/bash
#SBATCH --time=20:0:0
#SBATCH --job-name=PATHSAMPLE_LJ38
#SBATCH --ntasks=16

echo $SLURM_NTASKS > nodes.info
srun hostname >> nodes.info
echo $USER >> nodes.info
pwd >> nodes.info

/home/jwrm/bin/PATHSAMPLE > logfile

We run PATHSAMPLE for 20 hours on 16 cores. SLURM is free to assign those cores over multiple nodes as it sees fit. Note there is no creation of a temporary directory on /scratch/, PATHSAMPLE access the database stored in /sharedscratch/. Whenever an OPTIM job is created, PATHSAMPLE creates a temporary directory on /scratch/, copies the files required for OPTIM to run, launches OPTIM with srun, waits for OPTIM to finish, copies the files back and adds new stationary points to the database, then deletes the temporary directory. That is all taken care of for you. Because this approach involves writing to /sharedscratch/, it is only appropriate if each OPTIM job takes a while to run (more than a few minutes) and if the amount of data required for each OPTIM job is small. If either of those conditions is violated, users may notice a slowdown on the head node and be annoyed with you. In that case a different approach is required.

PATHSAMPLE can be run from /nodescratch/. This method reduces the NFS load, but is slightly more complicated and if the job terminates early, you will need to get your expanded database manually (see [[Getting started with SLURM#My job got cancelled, where are my files? | here]]). Here is an example PATHSAMPLE script that does this

#!/bin/bash
#SBATCH -J PATHSAMPLE_LJ38
#SBATCH --time=20:0:0
#SBATCH --ntask=4
#SBATCH --nodes=1

echo Starting job $SLURM_JOB_ID
hostname

TMP=/scratch/jwrm2/$SLURM_JOB_ID
mkdir -p $TMP
cp $SLURM_SUBMIT_DIR/pathdata $SLURM_SUBMIT_DIR/min.data $SLURM_SUBMIT_DIR/ts.data $SLURM_SUBMIT_DIR/points.min $SLURM_SUBMIT_DIR/points.ts $SLURM_SUBMIT_DIR/min.A $SLURM_SUBMIT_DIR/min.B $SLURM_SUBMIT_DIR/odata.connect $TMP/
cd $TMP

echo $SLURM_NTASKS > nodes.info
srun hostname >> nodes.info
echo $USER >> nodes.info
pwd >> nodes.info

/home/jwrm2/svn/PATHSAMPLE/builds/gfortran/PATHSAMPLE > logfile

cp min.data ts.data points.min points.ts min.A min.B logfile $SLURM_SUBMIT_DIR/ && rm -rf $TMP
cd $SLURM_SUBMIT_DIR/

We run for 20 hours on 4 cores. Since we are worried about the amount of data we're copying around, it makes sense to request these on the same node. We create a temporary directory on the node's /scratch/ space and copy the whole PATHSAMPLE database to it. Then we generate the nodes.info file and we're ready to run PATHSAMPLE itself. At the end, we copy the whole database back to /sharedscratch/ and delete the temporary directory if successful. This solution is not perfect, as we still have to copy the whole database over the NFS at the start and the end, but if each OPTIM job needs a lot of writing, or writes data very rapdily, it may be preferable.

==My job got cancelled, where are my files?==

If your job has finished, it will no longer appear in the output of squeue. It may have successfully completed, or it might have run out of time (or memory, etc.). Look at the end of the slurm-<job_ID>.out file. If it shows

slurmstepd: *** JOB 214233 ON compute-0-36 CANCELLED AT 2020-05-18T10:45:16 DUE TO TIME LIMIT ***

then your job ran out of time and was killed. Maybe this was a week long GMIN run that ran out of time 10 steps from the end. How annoying. Time to run it again from the beginning with a longer time limit, because there is no GMIN.dump file in the submission directory? Hardly... In our job submission script, we only deleted the temporary directory after the successful completion of the job. Therefore if the job did not successfully complete, the directory and its contents are still there. The information at the top of the slurm-<job_ID>.out file will help you find them. We created the temporary directory at

/scratch/jwrm2/214233

To access this from the head node, we need to know what node it was created on. That's why we ran 'hostname' at the start of the job. Let's say the output of that was

compute-0-17.local

We can access our files at

/nodescratch/compute-0-17/jwrm2/214233

Note that normal Bash tab completion of the directory name may not work here. That doesn't necessarily mean the directory doesn't exist, it's just a peculiarity of NFS automount.

It would be polite to delete the temporary directory after recovering any files you need.

==A smarter way to get your files==

Although recovering files from /nodescratch/ is fine, if you've every had 52 GMIN jobs terminate and needed to recover their GMIN.dump files for restarting, it gets a little tedious. Fortunately there is a better way, leveraging the signal mechanism of Linux. Here is an example:

#!/bin/bash
#SBATCH -J GMIN_LJ38
#SBATCH --time=5:00
#SBATCH -n1
#SBATCH --signal=B:USR1@120

signal_handler()
{
# Timeout commands go here
echo "Caught USR1 signal"
cp GMIN.dump logfile $SLURM_SUBMIT_DIR
exit
}
trap 'signal_handler' USR1

echo Starting job $SLURM_JOB_ID
hostname

TMP=/scratch/jwrm2/$SLURM_JOB_ID
mkdir -p $TMP
cp $SLURM_SUBMIT_DIR/data $SLURM_SUBMIT_DIR/coords $TMP/
cd $TMP

/home/jwrm2/bin/GMIN > logfile &
wait

cp -p lowest logfile $SLURM_SUBMIT_DIR && rm -rf $TMP
cd $SLURM_SUBMIT_DIR

This script will run any commands enclosed in braces at '# Timeout commands go here', 120 seconds before the job times out. It will terminate itself immediately after completing those commands, so you will not get a 'CANCELLED' message in the slurm-<job_ID>.out file (but only because of the 'exit', otherwise it would continue execution after the 'wait', which is probably not what you want). You can use it to copy back recovery files like GMIN.dump, or the updated PATHSAMPLE database, before the job terminates. If 120 seconds is not enough time for your timeout commands, change the 120 in the line '#SBATCH --signal=B:USR1@120'. The temporary directory will not be deleted if the timeout is reached, although you could put 'rm -rf $TMP' in the timeout section if you were really confident that you were grabbing everything you needed. You should occasionally go through /nodescratch/ and delete any leftover temporary directories that may be hanging around.

===Theory===

You don't need to understand how the above example works to use it, but we're all curious scientists, aren't we? The Linux kernel can communicate with a running process by sending it 'signals'. Common ones include 'SIGTERM' and 'SIGKILL'. In fact, when your job reaches the time limit, it gets sent a 'SIGTERM' signal, telling it to terminate. If the process has not set up anything specific to do on receiving a particular signal, the default action is to exit. However, most signals can have custom responses set up. An exception is 'SIGKILL', for which immediate termination cannot be overridden.

The line '#SBATCH --signal=B:USR1@120' tells SLURM to please send the signal 'USR1' (user specified signal number 1) to the job 120 seconds before reaching the time limit. 'USR1' is not sent by any system processes, so we're free to use it as we wish. Next we create a Bash function, 'signal_handler()', with our sequence of commands. We tell the Bash script to jump to our function on receiving the 'USR1' signal, overriding the default exit behaviour (trap 'signal_handler' USR1).

The final finicky bit is how we run our main programme. If we ran 'GMIN', the signal would get sent to GMIN, as the active process, rather than Bash. GMIN wouldn't know what to do with it, so would immediately exit. Not good. By running 'GMIN &' instead, the signal goes to Bash. However, if we did just that, the job would quickly reach the end after launching GMIN, whcih would terminate the job and all it's children, including GMIN. Oops. By writing 'wait', we tell Bash to go into an 'interruptible sleep', meaning it will sit there not doing anything until it receives a signal. Therefore it is able to exceute the commands in the 'signal_handler()' function on receiving the 'USR1' signal. As a matter of interest, execution of the main script then resumes where it left off before the signal, but the wait has now been completed (it only waits for the first signal) and execution proceeds to the 'cp' line. Putting 'exit' at the end of the function means we don't junp back to after wait. We could have put another 'wait' instead of 'exit', but that would only be useful if GMIN finished in the short period of time between completing the function and reaching the job time limit. Better to just save 2 minutes of CPU time by exiting immediately.

How does our job finish normally, if we've put it to sleep then? Well, when GMIN exits it sends, you guessed it... a signal, in this case 'SIGCHLD'. That will wake up Bash from its wait and the script will proceed to completion.