Docker Images for Panda ======================= Disclaimer ---------- This directory is a work in progress. I want to provide you with a starting point if you want to try PandaRoot in a Docker container, but there is still a lot to do here. TL;DR ----- On a Workstation ~~~~~~~~~~~~~~~~ 1. Download PandaRoot Source 2. Run ``rklasen/cvmfs`` docker container which provides FairSoft/FairRoot via CVMFS on ``/cvmfs`` .. code-block:: bash $ docker run -d --name cvmfs --privileged --cap-add SYS_ADMIN --device /dev/fuse --volume /cvmfs:/cvmfs:shared rklasen/cvmfs The container continues to run in background. Check after about a minute whether it works .. code-block:: bash $ ls /cvmfs/fairsoft.gsi.de centos7 centos8 debian10 debian8 macos-10.15 ubuntu1804 3. Run ``rklasen/fairroot:mini`` container that includes all that's needed for PandaRoot .. code-block:: bash $ docker run -u $(id -u):$(id -g) -it --rm --mount type=bind,source=/path/to/PandaRoot,target=/mnt/work/PandaRoot --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --volume /cvmfs:/cvmfs:shared --env="DISPLAY" rklasen/fairroot:mini Done. Compile PandaRoot and start simulations in the second container ("rklasen/fairroot:mini"). Everything you need is already there. On a HPC System with Singularity ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you cannot run the docker image (for example on cluster systems), it is easily possible to convert the ``mini`` image to a Singularity image. Because on these HPC systems the ``cvmfs`` mount often is already present, most of the work is already done. On your workstation, install `Singularity `_ and convert the ``mini`` docker image: .. code-block:: bash singularity build fairroot-mini.sif docker://rklasen/fairroot:mini Copy that to the HPC system. On there, make sure that the PandaRoot source directory is at ``~/PandaRoot`` (it will then automatically be found). On the cluster, you can execute the image: .. code-block:: bash module load tools/Singularity singularity exe fairroot-mini.sif bash And build PandaRoot directly: .. code-block:: bash cd PandaRoot mkdir build cd build cmake ../ -DUSE_DIFFERENT_COMPILER=true Nothing else is required, that's already it. A detailed explanation is [:ref:`down below `]. PandaRoot in a Docker Container ------------------------------- You can compile, install and run PandaRoot in a Docker container. The advantages are: - You don't have to install all needed packages system-wide - You can use system versions that are incompatible with PandaRoot (either too old or too new) - You can keep multiple different PandaRoot versions side by side And probably others. What is or should be included ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Usually, a complete PandaRoot installation meant you have to download FairSoft, compile and install it, download FairRoot, compile and install it and download PandaRoot and compile it (and of course yous system needs to have the needed packages installed). But since FairSoft and FairRoot are being built at the GSI and made available via CVMFS mount, not all of these steps are required anymore. Currently, these scenarios are considered (`see also these slides `_): - All-inclusive package. That means FairSoft/FairRoot and PandaRoot are included and compiled in this image and it's ready from the get-go. **Pros:** All inclusive, no internet connection is required to run. **Cons:** Large image size, PandaRoot code can't be (easily) modified. - Everything except PandaRoot. This is an image that includes FairSoft/FairRoot and all necessary packages, but not the PandaRoot Code. The PandaRoot code must be present on the host system and passed to the container, where it can be compiled and run. **Pros:** Edit PandaRoot code on host with favorite IDE, easy and reliable installation. **Cons:** Image is still quite large. - Base System and required packages, but nothing else. FairSoft/FairRoot must be present in the host system (for example via CVMFS mount) and passed to the container. **Pros:** Small image, no further configuration required to compile PandaRoot. **Cons:** Requires FairSoft/FairRoot from host. Either compiled and installed on host, which defeats the point, or via CVMFS mount, which requires internet connection. For now, only the second case is implemented here. That means you still have to download the PandaRoot source code to your machine and pass it to the docker container. This is shown below, and an example Dockerfile is provided in this directory. Building ~~~~~~~~ You can build the docker images with: .. code-block:: bash docker build -t fairroot:full -f Dockerfile-Full . for the full image that includes the FairSoft/FairRoot source code, or: .. code-block:: bash docker build -t fairroot:latest -f Dockerfile-Small . For a reduced images that contains only the compiled FairSoft/FairRoot binaries. Note that this requires an internet connection to clone the FairSoft/FairRoot repositories. Build time may take **several hours**. Because of that, an image is already built and uploaded to the Docker Hub as ``rklasen/fairroot:latest``. Pre-Built Docker Images ----------------------- There are three docker images ready for you: - Full: Includes FairSoft/FairRoot source and compiled - Small: Includes FairSoft/FairRoot compiled, but without source - Mini: Includes only required system packages for PandaRoot. FairSoft/FairRoot must be passed as shared volume. The "Mini" is probably the best all-rounder; it is small and still includes everything needed to compile and run PandaRoot. Mini ~~~~ Because this image needs an external FairSoft/FairRoot installation, you must already have those in ``/cvmfs/fairsoft.gsi.de/...``. On most HPC systems (HimsterII, GSI Compute farm), this should already be there. On your workstations, you can mount those two ways. Either, you do it from the host system, or you run a second docker container that mounts this to your host. Only this method is described here. Because this second container uses the hosts ``fuse``, it requires the ``--privileged`` flag. .. code-block:: bash docker run --privileged --cap-add SYS_ADMIN --device /dev/fuse --volume /cvmfs:/cvmfs:shared rklasen/cvmfs On your host system, the mount ``/cvmfs/fairsoft.gsi.de/`` should now be present. Pass it to the ``fairroot:mini`` container: .. code-block:: bash docker run -u $(id -u):$(id -g) -it --rm --mount type=bind,source=/path/to/PandaRoot,target=/mnt/work/PandaRoot --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" --volume /cvmfs:/cvmfs:shared rklasen/ --env="DISPLAY" fairroot:mini And you should be good to go. Note that this image is based on Debian, whereas the other two are based on Ubuntu. Building mini image ^^^^^^^^^^^^^^^^^^^ Build the image with .. code-block:: bash docker build -t fairroot:mini -f Dockerfile-Mini . Small ~~~~~ Still a work in progress. Full ~~~~ A built image is ready at the `Docker hub `_. You can test it out with: .. code-block:: bash docker run -u $(id -u):$(id -g) -it --rm --mount type=bind,source=/path/to/PandaRoot,target=/mnt/work/PandaRoot --env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw" rklasen/fairroot Some explanations for these flags: - ``-it --rm`` Run the container interactively and remove the container once you left it. Because nothing in the image is modified, it's not really necessary to keep it. - ``--mount type=bind,source=/path/to/PandaRoot,target=/mnt/work/PandaRoot`` pass your local copy of the code at ``/path/to/PandaRoot`` to the container. It is then accessible on the host system for development and in the container to compile and run. - ``-u $(id -u):$(id -g)`` By default, the docker daemon runs as root, so everything created inside the container is owned by hosts ``root``. That means all compiled artifacts and simulation data. Using user and group mapping, we can change that so that everything created in the container is owned by the hosts user instead of ``root``. - ``--env="DISPLAY" --volume="/tmp/.X11-unix:/tmp/.X11-unix:rw"`` Enables X11 pass-through from the container to you host. This way, you can use the ``TBrowser`` for example. - ``rklasen/fairroot`` The ready-to-use fairroot image into which PandaRoot can be built. Building a full image ^^^^^^^^^^^^^^^^^^^^^ Build the image with .. code-block:: bash docker build -t fairroot:full -f Dockerfile-Full . .. _example-build-pandaRoot-on-himster2: Example: Build PandaRoot on Himster2 ------------------------------------ Short Recipe ~~~~~~~~~~~~ - Convert Docker image to ``FairRoot-Mini.sif`` Singularity Image - On Login Node, clone PandaRoot code into your home directory (Singularity mounts home by default) - On Login Node, go into Singularity container and run ``cmake`` (do this on login node, cmake downloads some stuff and compute nodes don't have internet access) - Request interactive job on compute node to compile - On Compute Node, go into Singularity container and run ``make -j64`` Convert Docker image to ``FairRoot-Mini.sif`` Image ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Do this on your workstation, not on the Himster2. Get Docker image: .. code-block:: bash docker pull rklasen/fairroot:mini Install `Singularity `_ and convert to Singularity image: .. code-block:: bash singularity build fairroot-mini.sif docker://rklasen/fairroot:mini And copy that one to the Himster2. On Login Node, clone PandaRoot into your Home Directory ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This is easiest because Singularity mounts home by default and we don't have to pass volumes to it. So make sure your PandaRoot code is at ``~/PandaRoot/``. On Login Node, run ``cmake`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: bash module load tools/Singularity singularity exe fairroot-mini.sif bash You should be in the singularity container. .. code-block:: bash cd PandaRoot mkdir build cd build cmake ../ -DUSE_DIFFERENT_COMPILER=true The compiler option may be unnecessary. CMake should finish without errors and the build config should be written. We could also compile here, but we are limited to 4 cores, which means long compile time. Instead, we compile on a compute node. Request Interactive Job ~~~~~~~~~~~~~~~~~~~~~~~ .. code-block:: bash srun --pty -p <> -A <> bash --cpus-per-task 32 -i You should get an interactive session soon. On Himster2, ``<>`` is **not** your username. Verify that you are on a compute node: .. code-block:: bash hostname Should yield something like ``x2002.mogon``, or any other compute node. Just not a login node. Run the singularity container again: .. code-block:: bash module load tools/Singularity singularity exe fairroot-mini.sif bash In there, go to the build dir and start compiling: .. code-block:: bash cd PandaRoot/build make -j32 Bask in the glory of 32 threads compiling. It should finish without errors. Example Simulations ------------------- TODO (Merge requests welcome!) Conversion to Singularity Image ------------------------------- TODO (Merge requests welcome!)