Catherine Pitt
Catherine is one of the systems administrators for the theory sector.
Proposed changes to backup and archiving
Intel Trace Analyzer and Collector
useful stuff
export LD_DEBUG=all
is my friend, as is http://www.linux.org/docs/ldp/howto/Program-Library-HOWTO/shared-libraries.html
^[a-z][a-z0-9]{1,7}$
is a regexp for crsids.
cat ${PBS_NODEFILE} | uniq -c | awk '{print $2":"$1}'
I saw this on the Torque mailing list and can't believe I didn't think of it
dsh -a "rm -rf /var/adm/YaST/InstSrcManager/IS_CACHE_0x00000001" dsh -a 'installation_sources -a "nfs://192.168.1.254/install/suse/x86_64/9.3"'
fixing up node installation sources on a Streamline cluster
alt-printsreen is good
module snippets from http://www.lrz-muenchen.de/services/software/utilities/modules/modules_admin.html
# make application usage profile entry
# (do not use for very heavily used stuff like compilers)
if { [module-info mode] == "load" } {
exec /lrz/sys/bin/application-control pac $VERSION $env(HOSTNAME)
}
# need to be member of group "pac" to use
set groups [ exec groups $env(USER) ]
if { [ lsearch -exact $groups pac ] == -1 && [module-info mode] == "load" }
{
puts stderr "Your account is not in the UNIX group pac.
Please contact LRZ support to have it entered. Aborting module load."
break
}
Library linking aide memoire
LIBRARY_PATH is an environment variable used by many gccs at the link stage as a list of directories to search for shared libraries at link time. I have also tweaked the local PGI and Intel compilers to honour it (Pathscale and nag are gcc frontends so honour it out of the box). This is equivalent to the -L option that most compilers support.
There is a note in one of the CV modulefiles on tardis saying that Pathscale doesn't use bog-standard 'ld' when in -Ofast mode, and so LIBRARY_PATH doesn't work in that case, and indeed this is correct. For Pathscale one must use PSC_GENFLAGS or whatever to make things work with -Ofast. See (for example) the workstation acml modules.
LD_LIBRARY_PATH is used by the runtime linker as a list of directories to search for shared libraries at RUNTIME. Should be avoided where possible, as setting it means that every binary you call must search its contents which is a) slow and b) means you might pick up an incorrect library version if you don't manage it carefully. Especially dangerous if you set it to include NFS paths.
-rpath is a linker option that sets the RPATH and RUNPATH in the ELF header of the resulting binary when you link the thing. At runtime the RPATH is used by the runtime linker as the first list of directories to search for shared libraries. I have tweaked the local commercial compiler drivers to pass appropriate -rpath options that match the setting of LIBRARY_PATH, usually plus some compiler-specific ones (eg the particular compiler's own runtime libs)
Some people don't like rpath but LD_LIBRARY_PATH is even worse, see above. (Also that article applies to using it in a Linux distro that can be upgraded in place, whereas we're using it to try to stitch together a system comprising one Linux distro and a great many multi-versioned third party packages whose authors expect people to put up with LD_LIBRARY_PATH, and I never intend to upgrade in place)
As well as RPATH, ELF also has the RUNPATH. RUNPATH is allegedly better than RPATH because it's searched after LD_LIBRARY_PATH, giving the user a way to override it. *goes and tests* On my 32-bit SuSE 10.2 system this turns out to be misleading; LD_LIBRARY_PATH appears to override RPATH. Maybe RUNPATH behaving differently is a Solaris thing? Wait, if RUNPATH and RPATH are both set (as GNU ld does with -rpath) then RPATH is ignored so RUNPATH (set) is overridden by LD_LIBRARY_PATH if present. So that does actually make sense, and makes use of -rpath a lot less evil than you might think.
Unfortunately some elderly versions of GNU ld only set RPATH and not RUNPATH when passed -rpath (eg the one on zero). Curiously enough the gcc on zero doesn't honour LIBRARY_PATH either.
And then there is LD_RUN_PATH. This is an environment variable used by the linker at link time, if and ONLY if you didn't use a -rpath option. Its contents are then used to set the RPATH (and the RUNPATH? Who knows? Well I just tested it and it sets both as one would hope and expect). I suspect use of -rpath is better as it's more finely tuned- setting LD_RUN_PATH generally rather than using rpath per compiler would involve every binary you created having a huge RPATH, containing stuff for all the compilers you had kicking around.
chrpath is a neat little program that hacks at the ELF header of an existing binary and allows you to replace or delete the RPATH or RUNPATH. Source is in ~cen1001/src/utils on liberator or you can pull it out of Debian with apt-get source.
Then we get into libraries that need other libraries (this happens with MKL and also with ACML) and here it gets really nasty. Say executable file 1 needs library 1 which in turn needs library 2. Even if you compile exe1 with an rpath/runpath that will find both lib1 and lib2, this will not work (on Linux) unless lib1 has its own rpath/runpath that will find lib2, because the runtime linker only uses the runpath of the thing that has the dependency (ie the thing with the NEED in the ELF header). Solaris behaves differently in this respect, I am told (although my own experiments on CUS, which I believe runs Solaris 9, do not bear this out. Incidentally the Solaris equivalent of readelf is elfdump. I have certainly created a program calling a library which needs another library and despite a correct RPATH pointing to both libs in the program, the second one is not found without LD_LIBRARY_PATH). So you're totally stuck with commercial libraries, because the vendor does not know where you're going to install them and so can't set the runpath of lib1 so that it can find lib2, and LD_LIBRARY_PATH is your only hope. This is where chrpath might be useful.
ld has a -rpath-link option for linking a library that needs another library, which tells the linker to search the contents of -rpath-link for things like lib2, but does not add stuff to the RPATH, RUNPATH, or any other thing in the ELF header that might help any exe calling the first library find the second one again at runtime! The point of this option escapes me completely at present. Anything in an -rpath option when doing this sort of link is used after the -rpath-link option contents and inserted into the header in the usual way.
I need to investigate $ORIGIN, whatever that is. People on the Internet occasionally sing its praises for fixing this kind of problem, but I have a feeling it's a Solaris thing.
And finally and not very relevantly CPATH is an environment variable used by many C compiler drivers as a list of directories to search for headers. I think FPATH may do something similar. Equivalent to -I .