Catherine Pitt
Catherine is one of the systems administrators for the theory sector.
Proposed changes to backup and archiving
useful stuff
export LD_DEBUG=all
is my friend
^[a-z][a-z0-9]{1,7}$
is a regexp for crsids.
cat ${PBS_NODEFILE} | uniq -c | awk '{print $2":"$1}'
I saw this on the Torque mailing list and can't believe I didn't think of it
dsh -a "rm -rf /var/adm/YaST/InstSrcManager/IS_CACHE_0x00000001" dsh -a 'installation_sources -a "nfs://192.168.1.254/install/suse/x86_64/9.3"'
fixing up node installation sources on a Streamline cluster
Library linking aide memoire
LIBRARY_PATH is used by many gccs at the link stage as a list of directories to search for shared libraries at link time. I have also tweaked the local PGI and Intel compilers to honour it (Pathscale and nag are gcc frontends so honour it out of the box). This is equivalent to the -L option that most compilers support.
There is a note in one of the CV modulefiles on tardis saying that Pathscale doesn't use bog-standard 'ld' when in -Ofast mode, and so LIBRARY_PATH doesn't work in that case, and indeed this is correct. For Pathscale one must use PSC_GENFLAGS or whatever to make things work with -Ofast. See (for example) the workstation acml modules.
LD_LIBRARY_PATH is used by the runtime linker as a list of directories to search for shared libraries at RUNTIME. Should be avoided where possible, as setting it means that every binary you call must search its contents which is a) slow and b) means you might pick up an incorrect library version if you don't manage it carefully. Especially dangerous if you set it to include NFS paths.
-rpath is a linker option that sets the RPATH in the ELF header of the resulting binary when you link the thing. At runtime the RPATH is used by the runtime linker as a list of directories to search for shared libraries. I have tweaked the local commercial compiler drivers to pass appropriate -rpath options that match the setting of LIBRARY_PATH, usually plus some compiler-specific ones (eg the particular compiler's own runtime libs)
Some people don't like rpath but LD_LIBRARY_PATH is even worse, see above. (Also that article applies to using it in a Linux distro that can be upgraded in place, whereas we're using it to try to stitch together a system comprising one Linux distro and a great many multi-versioned third party packages whose authors expect people to put up with LD_LIBRARY_PATH, and I never intend to upgrade in place)
As well as RPATH, ELF also has RUNPATH. RUNPATH is allegedly better because it's searched after LD_LIBRARY_PATH, giving the user a way to override it. *goes and tests* On my 32-bit SuSE 10.2 system this turns out to be misleading; LD_LIBRARY_PATH appears to override RPATH. Maybe RUNPATH behaving differently is a Solaris thing? Wait, if RUNPATH and RPATH are both set (as GNU ld does with -rpath) then RPATH is ignored so RUNPATH (set) is overridden by LD_LIBRARY_PATH if present. So that does actually make sense, and makes use of -rpath a lot less evil.
And then there is LD_RUN_PATH. This is an environment variable used by the linker at link time, if and ONLY if you didn't use -rpath (gah!). Its contents are then used to set the RPATH (and the RUNPATH? Who knows? Well I just tested it and it sets both as one would hope and expect). I suspect use of -rpath is better as it's more finely tuned- setting LD_RUN_PATH generally rather than using rpath per compiler would involve every binary you created having a huge RPATH, containing stuff for all the compilers you had kicking around.
chrpath is a neat little program that hacks at the ELF header of an existing binary and allows you to replace or delete the RPATH or RUNPATH. Source is in ~cen1001/src/utils on liberator or you can pull it out of Debian with apt-get source.
Then we get into libraries that need other libraries (this happens with MKL and also with ACML) and here it gets really nasty. Say executable file 1 needs library 1 which in turn needs library 2. Even if you compile exe1 with an rpath/runpath that will find both lib1 and lib2, this will not work (on Linux) unless lib1 has its own rpath/runpath that will find lib2, because the runtime linker only uses the runpath of the thing that has the dependency (ie the thing with the NEED in the ELF header). Solaris behaves differently in this respect, I am told (I should try this on CUS before they take it away). So you're totally stuck with commercial libraries, because the vendor does not know where you're going to install them and so can't set the runpath of lib1 so that it can find lib2, and LD_LIBRARY_PATH is your only hope. This is where chrpath might be useful.
CPATH is used by many C compiler drivers as a list of directories to search for headers. I think FPATH may do something similar. Equivalent to -I .