Installation of PandaRoot on the Virgo cluster at GSI¶
Please refer to https://hpc.gsi.de/virgo on how to access and operate on the Cluster.
Log in to Virgo¶
The default entry point is on a CentOS7 container:
ssh virgo-centos7.hpc.gsi.de
Alternatively, you may use Debian8, which used to be the default system on the old cluster.
ssh virgo-debian8.hpc.gsi.de
The drawback is that Debian8 will be phased out in mid 2021.
The entry point will be a submit node, which is not designed to run any larger job. In fact your processes may get killed automatically if they occupy too many resources.
The cluster at GSI has a special place for mass storage of data, the lustre file system. You can store your large files at /lustre/panda/<username>. This file system can be mounted via sshfs (see here), which you can use to copy data to a local machine for further analysis.
The lustre file system is not the place to compile code. It has trouble managing many small files. However, the user home /u/<username> is not mounted on the cluster nodes. So it is best to store the source code on the user home directory, build on /tmp/ on the local machine for speed and install the libraries then on lustre.
PandaRoot on Virgo¶
Set up the environment for CentOS7¶
You will receive a shell within a container. The default FairSoft and FairRoot versions will be already installed on the CVMFS. It is strongly advised to use the same compiler that was used for FairSoft and FairRoot.
source scl_source enable devtoolset-8
spack load cmake%gcc arch=x86_64
ls /cvmfs/fairsoft.gsi.de
export SIMPATH=/cvmfs/fairsoft.gsi.de/centos7/gcc8/fairsoft/apr21
export FAIRROOTPATH=/cvmfs/fairsoft.gsi.de/centos7/gcc8/fairroot/v18.6.3_fs_apr21
export PATH=$SIMPATH/bin:$PATH
Set up the environment for Debian 8¶
You will receive a shell within a container. The default FairSoft and FairRoot versions will be already installed on the CVMFS. It is strongly advised to use the same compiler that was used for FairSoft and FairRoot.
export PATH=/cvmfs/vae.gsi.de/debian8/opt/linux-debian8-x86_64/gcc-8.1.0/cmake-3.17.3-szubqxkqi5zheszujgkq4d4xx7nrm5om/bin:$PATH
export PATH=/cvmfs/vae.gsi.de/debian8/opt/linux-debian8-x86_64/gcc-4.9.2/gcc-8.1.0-rqvu3k7qemjclnoxrymtq5wuhrep6pbe/bin:$PATH
export CC=gcc
export CXX=g++
ls /cvmfs/fairsoft.gsi.de
export SIMPATH=/cvmfs/fairsoft.gsi.de/debian8/fairsoft/nov20/
export FAIRROOTPATH=/cvmfs/fairsoft.gsi.de/debian8/fairroot/v18.6.1_fairsoft-nov20/
export PATH=$SIMPATH/bin:$PATH
Get PandaRoot code¶
For example if you want the latest release, just do
cd /u/<username>
git clone https://git.panda.gsi.de/PandaRootGroup/PandaRoot.git <source-folder>
cd <source-folder>
If you want the dev branch, do also
git checkout dev
Sometimes it may be necessary to use a specific release. For that we have Tags. Those can be listed and then you may choose to switch to a tagged version branch.
git tag
git checkout -b <yourtag> <yourtag>
Build¶
This follows the general build and Installation process.
mkdir /tmp/<build-folder>
cd /tmp/<build-folder>
cmake -DCMAKE_INSTALL_PREFIX=/lustre/panda/<username>/<install-folder> /u/<username>/<source-folder>
make -j16
make install
Now you could safely remove the sources (if necessary) and build artifacts (recommended):
rm -rf /u/<username>/<source-folder>
rm -rf /tmp/<build-folder>
Load PandaRoot¶
. /lustre/panda/<username>/<install-folder>/bin/config.sh -p
Here it is important to use the -p flag. Otherwise the environment would be missing important parts on the batch nodes.
Compatibility to Kronos¶
If you have to have full compatibility to the discontinued Kronos cluster, you have to choose Debian8.
The production script examples can be found in macro/production.
Good practice with disk access¶
If possible, use the /tmp folder on the nodes for your simulation/reconstruction job output. The local disks are fast and meant to be used in this manner. Copy the finalized files to your space in /lustre/panda/ at the end of each job. Minimizing massive file access to lustre will greatly improve the performance.