View source for Compiling Wales Group codes using cmake

[http://www.cmake.org/ CMake] (Cross-platform Make) provides a simple, platform independent way for us to compile and test the group codebase. Dependencies are handled automatically, compilation can proceed in parallel to avoid long waits while testing changes and builds are done entirely outside of the source directory. It also enables us to use the [[Jenkins CI]] 'build bot' system to automatically compile and test the code on a nightly basis - helping us catch troublesome commits before they affect other users. 

Although everything below refers to compiling [[GMIN]] with the Intel ''ifort'' compiler and AMBER9 - the exact same procedure works for [[OPTIM]] and [[PATHSAMPLE]].

Note that not every option for our codes is expected to actually compile with every compiler, for example, anything using CHARMM35/36 will not compile with ''nagfor'' or ''gfortran''. This is nothing to do with our code - it's a CHARMM issue. You can get an idea for what should work by looking at the automated [[Jenkins CI]] builds.

==Preparing to compile==
Before you get started, you need to ensure that the machine you are planning to compile on has cmake 2.8 or higher installed. You can check the current version like so:
<pre>
cmake --version
</pre>

The clusters have a module for cmake 3.0 (cmake 3.6.2 on Nest), which you can load using the following command:
<pre>
module load cmake/3.0.0
</pre>

You also need to create a directory to build the code in. We suggest that you create a directory for the compiler you are using within the program directory, under a subdirectory called 'builds' - for example for compiling GMIN with ifort, you would make a directory here:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/ifort
cd ~/softwarewales/GMIN/builds/ifort
</pre>
You can call these directories whatever you like - but make sure it is clear to you what they contain! You might also want to check which version of the compiler you have loaded. This is important as the different clusters and workstations may have different default versions loaded, some of which might not work properly. You can check the compiler version currently loaded using the same '--version' flag we used for ''cmake'' above:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ifort --version
ifort (IFORT) 12.1.3 20120212
Copyright (C) 1985-2012 Intel Corporation.  All rights reserved.
</pre>

To load a different compiler, you can use the ''module load'' or ''module swap'' commands. A list of all available modules can be accessed using:
<pre>
module av
</pre>
If you are having problems compiling, one of the first things to check is whether it works with a different version of the compiler!

'''Note: When compiling GMIN, if you are getting the error that there is no implicit type for ERFC in ewald.f90, try using a newer version of your compiler. This should be the built-in complementary error function.'''

==Compiling using the ccmake GUI interface to set options==
[[Image:Ccmake.png|thumb|ccmake set up to compile A9GMIN|200px|right]]

One advantage using cmake has over make is that we can use the simple ccmake GUI. This interface lets us set options like compiling with AMBER9, or CHARMM35, toggle between 'Release' and 'Debug' builds (see below) - and examine and alter the flags being uses for the compilation if we wish. Before we can run ccmake, we need to specify the compiler and run cmake in our build directory (e.g. softwarewales/GMIN/builds/ifort). We specify the '''F'''ortran '''C'''ompiler by setting the '''$FC''' environment variable (in this case the Intel Fortran compiler, ifort), and then run ''cmake'' (on the command line), passing it the relative location of the [[GMIN]] source directory:
<pre>
FC=ifort cmake ../../source
</pre>

If you run ''ls'', you will see some cmake files have been generated:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ls
CMakeCache.txt  CMakeFiles  cmake_install.cmake  Makefile  modules
</pre>

You can now run ''ccmake'' to open the GUI:
<pre>
csw34@sinister:~/softwarewales/GMIN/builds/ifort> ccmake .
</pre>

To navigate between options, use the arrow keys. Options can be toggled by pressing Return. To compile [[GMIN]] with AMBER9 (A9GMIN), we need to toggle the ''WITH_AMBER'' option ''ON''. Once you have done this, you need to configure and generate appropriate cmake info. This is done by pressing 'c' to configure, 'e' to exit and then 'g' to generate.

'''Note: for some builds (CHARMM with DFTB and CUDAGMIN), you might need to configure, exit and generate twice to set all necessary options'''

You can now compile A9GMIN in parallel as follows:
<pre>
make -j8
</pre>

The '-j8' flag here tells make to use up to 8 'threads' when building. For optimal performance, you should keep this slightly greater than the number of cores (CPUs) the node you are working on has. If all goes well, you should now have an A9GMIN binary in your build directory - congratulations! 
<pre>
Linking Fortran executable A9GMIN
[100%] Built target A9GMIN

--------------------------------------------------------------------------------------------- 15:23:45

csw34@sinister:~/softwarewales/GMIN/builds/ifort> ls
A9GMIN          cmake_install.cmake   libcudadummylib.a  libmylapack.a  NAB
AMBER           display_version.f90   libdummylib.a      Makefile       nab_binaries_built
CMakeCache.txt  GMIN                  libgminlib.a       modules        porfuncs.f90
CMakeFiles      libamber12dummylib.a  libmyblas.a        n
</pre>
Plain [[GMIN]] is also built at the same time should you need it. You can move this into your ~/bin directory if you like, or anywhere else in your ''$PATH'' to make running it simple.


'''Note: If you want to use OPTIM with the new C++ implementation of the NEB routine, you will need to obtain the source code for that separately. See [https://wikis.ch.cam.ac.uk/wales/wiki/index.php/OPTIM here] for instructions.'''

==Compiling by setting options on the command line==
If you know the options you'd like to set already (you can see them all in ccmake), you can save some time by passing them directly to ''cmake'' on the command line, bypassing the need for ''ccmake''. For example, to compile A9GMIN (GMIN with the AMBER9 interface) using the Intel ifort compiler, you would run the following commands:
<pre>
FC=ifort cmake -DWITH_AMBER=1 ../../source
make -j8
</pre>
where '../../source' is the relative location of the GMIN source directory. You can find some more examples of compiling from the command line below.

'''Note: Sometimes you may get error''' (for example, Fatal Error: Can't open module file 'someModule.mod' for reading at (1): No such file or directory) when following this procedure. In that case there are three things you could try: make sure you are building in a new directory, if that does not help run `make VERBOSE=1` instead of `make -j8` or simply switch to using ccmake.

==Compiling with MPI==
To compile with MPI support add the following flags when running cmake on the command line:
<pre>
FC=mpif90 CC=mpicc cmake ../source -DCOMPILER_SWITCH=pgi -DWITH_MPI=yes
</pre>
Here -DCOMPILER_SWITCH=pgi assumes you're using the Portland ''pgi'' compiler. Make sure you have the correct modules loaded (in this case ''pgi'' and ''mpi-pgi''), and that the particular mpi you want (in this case ''mpi-pgi'') is listed before any other mpi's loaded (so that it has the highest priority). The modules can be loaded by typing:
<pre>
module load pgi/64/
module load mpi/openmpi/pgi/64/1.6.3
</pre>

and you can check which modules are loaded and in which order/priority by the ''module list'' command. You may need to ''module unload <name>'' any other mpi's that are higher up in the list than the one you want. You can of course set the COMPILER_SWITCH and WITH_MPI flags in ''ccmake'' if you prefer.

Note: It has been observed that pgi/64/15.1 leads to compilation errors, and for now, it is best to use pgi/64/14.9

==Advanced mode - changing compiler flags with ccmake==
[[Image:Ccmakeadvanced.png|thumb|ccmake advanced mode|200px|right]]

Although initially the ''ccmake'' GUI looks very simple, there is a lot going on under the hood. By pressing 't' you can enter 'Advanced mode' which will show you all of the hidden options, for example the compiler flags that are being passed to ''make'' when you compile the code. You can also make changes to the flags here, for example if you would like to add '-p' to do  profiling. 

As with changing the build type, you simply select the field you'd like to change using the arrow keys, press Return, make your changes and press Return again to save them. When you subsequently configure and generate as above, those altered flag will be used for the subsequent compilation.

Note that these changes only apply in the build directory in which you make them.

==Debugging runtime problems using gdb or valgrind==
If you are getting a segmentation fault, crash or other unexpected behaviour, you might want to run your job through a debugger like [http://www.gnu.org/software/gdb/ gdb] or [http://valgrind.org/ valgrind]. In order to maximise your chances of getting useful output, you should build a 'Debug' version of the program you are having trouble. To do this, you can either change the ''CMAKE_BUILD_TYPE'' in ''ccmake'' to 'Debug' (press Return, change 'Release' to 'Debug' and press Return again), or on the command line like so for GMIN with AMBER 9 using the Intel ifort compiler:
<pre>
FC=ifort cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_AMBER=1 ../../source
make -j8
</pre>

You can then run the binary ''through'' gdb or valgrind as follows:
<pre>
gdb A9GMIN
</pre>
or
<pre>
valgrind A9GMIN
</pre>

I won't cover debugging with these tools here as it's a science in itself! Do some Googling and ask for help as needed :)

==Debugging compilation problems==
There are many ways to try and track down why your code is not compiling. Before you start changing compilers, building a 'Debug' version or changing machines, you might want to try running make again with the ''VERBOSE'' option enabled. This will dump a lot of potentially useful output:
<pre>
VERBOSE=1 make
</pre>

One possible gotcha: all .f and .f90 files in the relevant source directories will be compiled and added to a library. This is quite different from the old Makefile way of doing things, where source files were explicitly specified for compilation (via their corresponding .o file). So, if you are testing something by for instance copying code.f90 to code.myhack.f90 and code.orig.f90, then slightly editing a line or two of code.myhack.f90 and copying it back to code.f90 for use, this will probably cause linking problems due to multiply-defined subroutines (from all three files). The solution, if you must have alternative versions of the same file hanging round, is to differentiate the filenames by a suffix AFTER the .f[90] .
 
Another occasional issue is the unexplained compiler bug - a problem with the version of the compiler you happen to be using. You can can an idea for which compiler versions we expect to work by checking the Jenkins build-bot output, as described in the 'Seeing console output' section of the [[Jenkins CI]] page. If you are using a different version of the compiler in question, consider swapping to the version Jenkins is using with 'module swap'.

If the error message you are getting doesn't make sense to you after some Googling, go and ask someone - we all have these problems. Things you can try first include trying a different compiler version, or an entirely different compiler e.g. pgi rather than ifort for example. You should bear in mind that as mentioned above, not all versions of each code will compile with every compiler. Make sure you're not trying to build something that isn't expected to work.

To build the executables with the QUIP interface, it may be necessary to do make clean in the QUIP directory.

==Extra command line build examples==
The below commands are absolutely not an exhaustive list, but should give you an idea of what is possible. You can use ''ccmake'' as described above to discover which variables (e.g. WITH_AMBER) can be manipulated on the command line like this. All of these examples assume your git repository is set up in ''/home/CRSID/softwarewales'' - make the appropriate modifications if you have it elsewhere.

===GMIN===
'''A12GMIN''' (GMIN with AMBER12) using ifort:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/ifort_amber12
cd !$
FC=ifort cmake -DWITH_AMBER12=1 ../../source
make -j8
</pre>

'''C35GMIN''' (GMIN with CHARMM 35) using pgi:
<pre>
mkdir -p ~/softwarewales/GMIN/builds/pgi_charmm35
cd !$
FC=pgf90 cmake -DWITH_CHARMM35=1 ../../source
make -j8
</pre>

'''CUDAGMIN''' (GMIN leveraging GPU minimisation via the AMBER 12 interface) using ifort:
<pre>
module load cuda/5.5
mkdir -p ~/softwarewales/GMIN/builds/ifort_cuda
cd !$
FC=ifort cmake -DWITH_CUDA=1 ../../source
make -j8
</pre>
This will only work on machines with specific NVIDIA GPUs, for example when submitting jobs on the pat cluster. There is some additional information on the [[Using GMIN with GPUs]] page.

'''DFTBGMIN''' (GMIN with DFTBP) using Intel on nest:
<pre>
module load mkl/64/2022/0/0 cmake/3.23.2 ifort/64/2020/4/304
mkdir -p ~/softwarewales/GMIN/builds/ifort_dftbp
cd !$
FC=ifort CC=icc CXX=icc cmake ../../source -DWITH_DFTBP=yes
make -j8
</pre>

or using GCC on nest:
<pre>
module load cmake/3.23.2 gcc/12.2.0
mkdir -p ~/softwarewales/GMIN/builds/gfortran_dftbp
cd !$
cmake ../../source -DWITH_DFTBP=yes -DWITH_MYBLAS=no -DBLAS_LIBRARIES=/usr/lib64/libopenblas.so.0
make -j8
</pre>

===OPTIM===
'''A9OPTIM''' (OPTIM with AMBER9) using ifort:
<pre>
mkdir -p ~/softwarewales/OPTIM/builds/ifort_amber
cd !$
FC=ifort cmake -DWITH_AMBER9=1 ../../source
make -j8
</pre>

'''C35OPTIM''' (OPTIM with CHARMM 35) using pgi:
<pre>
mkdir -p ~/softwarewales/OPTIM/builds/pgi_charmm35
cd !$
FC=pgf90 cmake -DWITH_CHARMM35=1 ../../source
make -j8
</pre>

'''CUDAOPTIM''' (OPTIM leveraging GPU via the AMBER 12 interface) using ifort:
<pre>
module load cuda/5.5
mkdir -p ~/softwarewales/OPTIM/builds/ifort_cuda5.5
cd !$
FC=ifort CC=icc cmake -DWITH_CUDA=1 ../../source
make -j8
</pre>
This will only work on machines with specific NVIDIA GPUs, for example when submitting jobs on the pat cluster. There is some additional information on the [[Using GMIN and OPTIM with GPUs]] page.

'''DFTBOPTIM''' (OPTIM with DFTBP) using Intel on nest:
<pre>
module load mkl/64/2022/0/0 cmake/3.23.2 ifort/64/2020/4/304
mkdir -p ~/softwarewales/OPTIM/builds/ifort_dftbp
cd !$
FC=ifort CC=icc CXX=icc cmake ../../source -DWITH_DFTBP=yes
make -j8
</pre>

or using GCC on nest:
<pre>
module load cmake/3.23.2 gcc/12.2.0
mkdir -p ~/softwarewales/OPTIM/builds/gfortran_dftbp
cd !$
cmake ../../source -DWITH_DFTBP=yes -DWITH_MYBLAS=no -DBLAS_LIBRARIES=/usr/lib64/libopenblas.so.0
make -j8
</pre>

===PATHSAMPLE===
There are very few options for [[PATHSAMPLE]] as we don't need to worry about interfacing with a particular potential. As a result, every binary is simply called ''PATHSAMPLE''.

Using nagfor (the NAG fortran compiler - check you have the module loaded - very strict!):
<pre>
mkdir -p ~/softwarewales/PATHSAMPLE/builds/nagfor
cd !$
FC=nagfor cmake ../../source
make -j8
</pre>

Using pgi (much more generous with coding slips/non-standard uses):
<pre>
mkdir -p ~/softwarewales/PATHSAMPLE/builds/pgi
cd !$
FC=pgf90 cmake ../../source
make -j8
</pre>

==Configuring defaults - for developers==

Fortran compilers and their corresponding default settings are all controlled by the file $SVN/CMakeModules/FindFORTRANCOMPILER.cmake ($SVN is your svn root directory). In particular, we may wish to edit the flags used for each set of compilers and build type. These are contained in the following block:

<pre>
if(NOT COMPILER_FLAGS_WERE_SET)
   message("Setting initial values for compiler flags")
   if(COMPILER_SWITCH MATCHES "pgi")
      set (CMAKE_Fortran_FLAGS "-Mextend" CACHE TYPE STRING FORCE)
      set (CMAKE_Fortran_FLAGS_RELEASE "-O3 -Munroll -Mnoframe" CACHE TYPE STRING FORCE)
      set (CMAKE_Fortran_FLAGS_DEBUG "-Mextend -C -g -gopt -Mbounds -Mchkfpstk -Mchkptr -Mchkstk -Mcoff -Mdwarf1 -Mdwarf2 -Melf -Mpgicoff -traceback" CACHE TYPE STRING FORCE)
      set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-Mextend -C -g -gopt -Mbounds -Mchkfpstk -Mchkptr -Mchkstk -Mcoff -Mdwarf1 -Mdwarf2 -Mdwarf3 -Melf -Mpgicoff -traceback" CACHE TYPE STRING FORCE)
      set (FORTRAN_FREEFORM_FLAG "-Mfree" CACHE TYPE STRING)
   elseif(COMPILER_SWITCH MATCHES "gfortran")
      set (CMAKE_Fortran_FLAGS "-ffixed-line-length-200 -ffree-line-length-0" CACHE TYPE STRING FORCE)
      set (CMAKE_Fortran_FLAGS_RELEASE "-O3" CACHE TYPE STRING FORCE)
#      set (CMAKE_Fortran_FLAGS_DEBUG "-g -fbounds-check -Wuninitialized -O -ftrapv -fimplicit-none -fno-automatic" CACHE TYPE STRING FORCE)
      set (CMAKE_Fortran_FLAGS_DEBUG "-g -fbounds-check -Wuninitialized -O -ftrapv -fno-automatic" CACHE TYPE STRING FORCE)
      set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "${CMAKE_Fortran_FLAGS_DEBUG} -fimplicit-none" CACHE TYPE STRING FORCE)
      set (FORTRAN_FREEFORM_FLAG "-ffree-form" CACHE TYPE STRING)
   elseif(COMPILER_SWITCH MATCHES "nag")
      set (CMAKE_Fortran_FLAGS "-132 -kind=byte -maxcontin=3000" CACHE TYPE STRING FORCE)
      set (CMAKE_Fortran_FLAGS_RELEASE "-mismatch_all -O4" CACHE TYPE STRING FORCE)
      set (CMAKE_Fortran_FLAGS_DEBUG "-g -mismatch_all -ieee=stop" CACHE TYPE STRING FORCE)
      set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-C=all -mtrace=all -gline -g -mismatch_all -ieee=stop" CACHE TYPE STRING FORCE)
      set (FORTRAN_FREEFORM_FLAG "-free" CACHE TYPE STRING) # js850> is this ever used?
   elseif(COMPILER_SWITCH MATCHES "ifort")
      set (CMAKE_Fortran_FLAGS "-132 -heap-arrays -assume byterecl" CACHE TYPE STRING FORCE)
      set (CMAKE_Fortran_FLAGS_RELEASE "-O3" CACHE TYPE STRING FORCE)
# Warnings about temporary argument creation and edit descriptor widths are disabled with the final flags.
      set (CMAKE_Fortran_FLAGS_DEBUG "-C -g -traceback -debug full -check all,noarg_temp_created -diag-disable 8290,8291" CACHE TYPE STRING FORCE)
      set (CMAKE_Fortran_FLAGS_DEBUG_SLOW "-debug all -check all -implicitnone -warn unused -fp-stack-check -ftrapuv -check pointers -check bounds" CACHE TYPE STRING FORCE)
      set (FORTRAN_FREEFORM_FLAG "-free" CACHE TYPE STRING)
   else()
      message(FATAL_ERROR "unknown comiler switch: ${COMPILER_SWITCH}")
   endif()
    SET(COMPILER_FLAGS_WERE_SET yes CACHE TYPE INTERNAL)
endif(NOT COMPILER_FLAGS_WERE_SET)
</pre>

The main if/elseif blocks correspond to compiler switches. Inside these, there are the default flags for each of our build types (release, debug and debug_slow), which are configured using ccmake. These can be edited, if we wish to change the default behaviour (e.g. a recent addition of -check all,noarg_temp_created -diag-disable 8290,8291 to disable annoying warning messages for ifort).