Installation Guide¶

This documentation aims to enable the user to install DINO on any HPC system. As an example, the SuperMUC-NG HPC system of the Leibniz Rechenzentrum is used. Different steps of the installation can be distinguished:

System Requirements
Download and Installation of Dependencies
- mpi
- sundials
- cantera
- hdf5
- lapack
- hypre
- fftw
DINO Installation

On modern HPC systems, these libraries might already be available as modules that you can load to setup your environment. First you can check, which versions of the libraries are available and then load the correct one, for example:

    module avail lapack
    module load lapack

Note

Make sure, that the supported versions of all libraries are used. If they are not available, you can install the dependencies locally in your working directory. How you can do this is described below. If not all of the necessary libraries are available, you can install them individually and load the right available modules for all other dependencies.

System Requirements¶

Before compiling DINO, ensure that the following system requirements are met:

A Unix-like operating system (Linux or macOS; HPC clusters are fully supported)
A Fortran 90 compiler (e.g., gfortran, ifort, or nvfortran)
CMake ≥ 2.8

Check that cmake is available on your system:

cmake --version

If CMake is not installed, you can find installation instructions on the CMake website.

Download and Installation of Dependencies¶

Compilers must be available for the compilation and installation of DINO and its dependencies. On modern HPC systems, you can configure them via modules, that are preinstalled and available. On SuperMUC, the necessary compilers are:

    module load intel/21.4.0
    module load mpi.intel/2019-12

To compile DINO, required libraries (so called dependencies) must be installed and available. It is advised to create the two directories:

    mkdir dino_libs_sources
    mkdir dino_libs_install

The 'dino_libs_sources' folder contains all relevant source files, while the executables are installed in the dino_libs_installfolder. Dependencies can thus be easily accessed and a clear path system is available to compile the code.

In general, the source files of all dependencies is downloaded via git.

Info

If your HPC system of choice does not grant access to the internet and you can not use gitlab, the source code must be copied from your local computer, from which you have access to the HPC system. You can do so by using the command:

    rsync -raulvhpc sundials <username>@cluster.adress.com:/path/to/dino_libs_sources

MPI¶

The message passing interface (MPI) is developed and maintained by a consortium of academic, researc, and industry partners. Thus resources from all across the High Performance Computing community are combined to build the best MPI library available. You can download the Open MPI source code from the main Open MPI web site.

Open MPI uses a traditional configurescript paired with make to build. Typicall installs can be done by:

    ./configure --prefix=/path/to/dino_lib_install/openmpi [...options...]
    make -jN all
    make install

Also VPATH builds are fully suported. For example:

    cd openmpi
    mkdir build
    cd build
    ../configure --prefix=/path/to/dino_libs_install/openmpi

Sundials¶

The fist dependency that must be installed is the sundials library. It provides solvers for differential equations that are used for advanced time integration.

Note

Sundials must be compiled before Cantera.

Warning

We actively work on using the most recent version of Cantera. However, Compatibility of Sundials and Cantera is an important issue. Thus we recommend using these combinations:

Sundials 2.3 & Cantera 1.8
Sundials 2.6.2 & Cantera 2.2.0
Sundials 3.0.0 & Cantera 2.4.0
Sundials 6.0.0 or 7.0.0 & Cantera 3.1.0

The code can be downloaded from the official git repository or any source files available on the website of the Lawrence Livermore National Laboratory can be downloaded..

To download the code using git you can use the following command:

    git clone https://github.com/LLNL/sundials/tree/v3.0.0

Make sure, you are on the right branch:

    git branch -a
    git checkout v3.0.0

The compilation of sundials is done via CMake, which is usually available on HPC systems. You may check that by using

    module avail cmake

and load the corresponding module:

    module load cmake

You will then need to create a build folder

    mkdir build
    cd build

Finally, you can setup the compilation:

    cmake -DCMAKE_INSTALL_PREFIX=/beegfs1/software/dino_libs/sundials -DCMAKE_BUILD_TYPE=Release -DCMAKE_Fortran_COMPILER=ifort -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icpc ../

and compile SUNDIALS:

    make -j N

where N is the number of processors used for the compilation. To install it to your installation folder (CMAKE_INSTALL_PREFIX, that should point to your dino_libs_install), use

    make install

Add the installation path to your environment:

    export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:$HOME/dino_lib_install/sundials/lib"

It is advised to add this line to your ~/.bashrc to grant permanent access to this installation.

Cantera¶

2.4.03.1.0

Cantera provides thermodynamic and kinetic data for chemical species and reactions. It is user in DINO to compute reaction rates and thermodynamic properties. Since Cantera is a large program package, different external libraries must be available as well.

Warning

Currently, cantera version 1.8.0, 2.2.0, 2.4.0 and 3.1.0 are used in DINO and compatible with sundials 1.8, 2.6.2, 3.0.0 and 6.0.0/7.0.0, respectively. Make sure to install the right sundials version for the available cantera and use similar compiler flags to avoid conflicts. Cantera 3.1.0 is the version that should be installed by default. It accelerates computations by around 10% compared to 2.4.0.

Again, you can download the source code via git:

    git clone https://github.com/Cantera/cantera.git
    git checkout v2.4.0

External libraries are usually downloaded in the $cantera_home/ext/ directory via git. However, if your HPC cluster does not allow any access to the internet, you need to copy the source codes into the respective directories, if they are not installed elsewhere. The external libraries are:

sundials (Use the sundials that you installed previously for compatibility, change the cantera.conf to point to the right sundials installation)
eigen
fmt
googletest

Note

You do not need to install these libraries, only copy the root repository to the respective cantera/ext folder. If you have these libraries already installed, you can point to the right repositories in the cantera.conf file.

To install Cantera 2.4.0 the SCons compilation library is used, that is slightly different to CMake. However, more recent versions allow for both, CMake and SCons.

Any installation flags are contained in the cantera.conf file, that is available in the $cantera_home directory. For compilation on SuperMUC, the following cantera.conf is used:

CXX = 'icpc'
CC = 'icc'
prefix = '$HOME/dino_lib_install/cantera'
python2_package = 'none'
python3_package = 'minimal'
f90_interface = 'y'
FORTRAN = 'ifort'
sundials_include = '$HOME/dino_lib_install/sundials/include'
sundials_libdir = '$HOME/dino_lib_install/sundials/lib'
env_vars = 'all'
boost_inc_dir = '/dss/lrzsys/sys/spack/release/22.2.1/opt/skylake_avx512/boost/1.75.0-intel-6a2uo5l/include'

Warning

Make sure to use the same compilers for C,C++ and fortran as for sundials.

The installation follows by execution fo the scons build command:

    scons build

and executables are installed via:

    scons install

Add the following lines to your ~/.bashrc to make Cantera available in your environment:

    # cantera 
    source $HOME/dino_lib_install/cantera/bin/setup_cantera
    export PATH=$PATH:$HOME/dino_libs_install/cantera/include/cantera

Cantera provides thermodynamic and kinetic data for chemical species and reactions. It is used in DINO to compute reaction rates and thermodynamic properties. Since Cantera is a large program package, different external libraries must be available as well.

Warning

Currently, cantera version 1.8.0, 2.2.0, 2.4.0 and 3.1.0 are used in DINO and compatible with sundials 1.8, 2.6.2, 3.0.0 and 6.0.0/7.0.0, respectively. Make sure to install the right sundials version for the available cantera and use similar compiler flags to avoid conflicts. Cantera 3.1.0 is the version that should be installed by default. It accelerates computations by around 10% compared to 2.4.0.

The easiest way to install Cantera 3.1.0 is to use Anaconda. Please make sure that Anaconda or Miniconda is installed. You can find the installation instructions of Miniconda here.

    conda --version

If Anaconda is installed you can create an environment and installing the Cantera Development (C++ & Fortran 90) Interface with the following command:

    conda create --name ct-dev --channel conda-forge libcantera-devel=3.1.0

Add the following lines to a bash script to define the library paths to Cantera:

    cmake \
-DUSE_CANTERA=ON \
-DCANTERA_VERSION=3 \
-DCANTERA_DIR=/<path to conda envs>/ct-dev \ 
-DCANTERA_INCLUDE_DIR=${DCANTERA_DIR}/include/cantera \
-DCANTERA_LIBRARIES=${DCANTERA_DIR}/lib \

LAPACK¶

The LAPACK library offers optimized routines for linear algebra operations.

Note

Requires version 3.5.0 or newer.

It is downloaded via git:

    git clone https://github.com/Reference-LAPACK/lapack.git
    git checkout v3.10.0

LAPACK can be installed via make, and the configuration must be set in the make.inc file. Given examples are in given in the INSTALL directory. Use the one that corresponds to your compiler. On SuperMUC, the ifort compiler is used for DINO, so that the corresponding example script can be used:

    cp INSTALL/make.inc.ifort ./make.inc

You only need to set the installation prefix paths to your preference:

    BLASLIB      = $HOME/dino_libs_install/lapack/librefblas.a
    CBLASLIB     = $HOME/dino_libs_instal/lapack/libcblas.a
    LAPACKLIB    = $HOME/dino_libs_install/lapack/liblapack.a
    TMGLIB       = $HOME/dino_libs_install/lapack/libtmglib.a
    LAPACKELIB   = $HOME/dino_libs_install/lapack/liblapacke.a

After that, you can compile the library::

    make

Alternatively, a CMake compilation is also possible:

    mkdir build
    cd build
    cmake -DCMAKE_INSTALL_LIBDIR=$HOME/.local/lapack ..
    cmake --build . -jN --target install

Warning

Be aware that you might need to change CMake flags such as the used compilers etc.

HDF5¶

HDF5 is the prefered file format of DINO. Solutions are stored as dino_solution.h5 files and thus the HDF5 library is needed. You can download the source code via git and use a release version:

    git clone https://github.com/HDFGroup/hdf5.git
    git checkout hdf5-1_10_0

Warning

HDF5 requires at least version 1.8.x for DINO. Compatibility issues with more recent versions may be encountered.

To compile and install hdf5, it is necessary to run the configure:

    ./configure --prefix=/path/to/dino_libs_install/hdf5 --enable-fortran --enable-static --enable-parallel --enable-build-mode=production

After configuring the installation, hdf5 can be compiled and installed:

    make -j N
    make install

Finally, you need to export the paths of the installation to your environment:

    export PATH=$PATH:$HOME/dino_lib_install/hdf5/bin/
    export PATH=$PATH:$HOME/dino_lib_install/hdf5/include/
    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/dino_lib_install/hdf5/lib

To do this permanently, add these lines to your ~/.bashrc.

hypre¶

The hypre library offers scalable solvers and preconditioners for large sparse linear systems.

Warning

DINO needs at least hypre cersion 2.9

The sources can be downloaded via git:

    git clone https://github.com/hypre-space/hypre.git
    git checkout V2-9-0b

After downloading the sources, you need to configure the compiler and compile the code:

    cd ./src
    ./configure --prefix=/path/to/dino_libs_install/hypre
    make -jN
    make install

FFTW¶

FFTW is a library that offers fast fourier transformation functions for spectral solvers, which is used in DINO to solve the Poisson equation that coupled pressure and velocity fields.

The source code can be obtained via git:

    git clone https://github.com/FFTW/fftw3.git
    git checkout fftw-3.0

Warning

DINO requires FFTW versions >3.0.0.

FFTW is compiled similarly to hypre via a configuration script:

    ./configure --prefix=/path/to/dino_libs_install/fftw
    make -jN
    make install

Downloading DINO¶

Dino is distributed via a private git repository. Only registered users with granted access rights can clone the project. If you would like to onbtain access to DINO, please contact the authors.

Once you have been granted permission, you can download the code using one of the following commands:

    git clone git@code.ovgu.de:dino/dino.git

or

    git clone https://code.ovgu.de/dino/dino.git

You will be promted to enter your user name and password during authentication.

Updating DINO¶

To update your local copy and obtain the latest developments, use:

    git checkout master
    git pull

These commands ensure you are on the master branch and synchronize your local repository with the remote one. After updating, DINO must be recompiled to include all recent changes.

Modifying DINO¶

If you plan to modify or extend the code, it is recommended to work on a seperate local branch:

    git checkout -b <new branch name>

This command creates a new branch where you can safely commit your changes without affecting the stable master branch. To synchronize your changes with the main repository run:

    git fetch

You can then merge your dev elopments to the master branch, to make them available for everyone:

    git merge origin/master

After starting this merge request, your changes will be reviewed and validated by a code maintainer before integration to the master branch.

Instructions on how to contribute to DINO can be found in the Contributing section of this documentation.

Note

This instruction is only a short way to enable new users to work with DINO. If you are unfamiliar with git, you can learn basics with the gitlab tutorial

DINO Installation¶

Once all dependencies are available, navigate to your DINO installation directory.

Preparing the build environment¶

The main CMake configuration file is located in the project's top-level folder ($DINO_HOME/CMakeLists.txt). It is recommended to build DINOÜ in a dedicated build* directory to keep sources files clean and seperate from compiled objects.

    # go to the DINO repository
    cd $DINO_HOME
    # create and enter the build directory
    mkdir build && cd build

This directory will contain all intermediate build files and can safely be deleted and recreated without affecting the source code.

Configuring the Build with CMake¶

CMake is used to detect all dependencies, configure compiler options, and generate the appropriate Makefiles. The basic configuration command is:

    cmake -DUSE_EGLIB=OFF -DCMAKE_BUILD_TYPE=Release ../

Note

Common CMake options

-DUSE_EGLIB=OFF disables the EGlib module (Use ON to enable EGLib for transport properties. If disabled, Cantera or FPI tables must be used instead).
'-DCMAKE_BUILD_TYPE=Releaseenables an optimized build (UseDebug` for verbose output and debugging features).
../ points to the root of the DINO source tree

Example with a custom installation path:

    cmake -DUSE_EGLIB=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/path/to/install ../

If all dependencies are correctly located, CMake will generate the necessary Makefiles and report the detected configuration.

Note

On HPC systems, you may need to load modules for MPI, FFTW, HDF5, or compilers before running cmake.

Compiling the code¶

After successful configuration, compile the source code using:

make -jN

where Nis the number of CPU cores used for parralel compilation (e.g., make -j8)

Note

This step can take several minutes depending on system performance and compiler optimization level.

Installing Executables¶

Once compilation is complete, install the executables and libraries:

    make install

This will copy the compiled binaries to the bin/ directory (or to the directory specified by CMAKE_INSTALL_PREFIX). To verify that the installation was successful:

    ls $DINO_HOME/bin

You should see the main DINO executable and associated utilities.

Recompilation and Clean Builds¶

When updating DINO from the repository, always recompile the code to ensure compatibility with recent changes:

    git checkout master
    git pull
    cd build
    make clean
    cmake ../
    make -jN
    make install

Warning

If compilation issues occur, it is often sufficient to remove the entire build/ directory and repeat the configuration process.

Verification¶

After installation, run a simple test case to verify that DINO is working correctly. The autoignition test case in the /WORK/TESTS/ subfolder is a good start. You can run a case of your choice via a slurm submission script that is provided in the test case.

    sbatch run_test.sh

Warning

You might need to modify the submission script, depending on your HPC cluster.

Notes for Advanced Users¶

On HPC clusters, use provided CMake toolchains or host configuration files from the $DINO_HOME/CMAKE/ directory to simplify setup on different architectures. These files contain preconfigurations for different HPC clusters such as

SuperMUC (Cluster of the Leibniz Rechenzentrum)
SOFJA (Cluster of the University of Magdeburg "Otto-von-Guericke")
KARMAN (local cluster of LSS)

For development or debugging, compile with -DCMAKE_BUILD_TYPE=Debugand consider enabling verbose build output:

    make VERBOSE=1

If you plan to contribute to DINO, refer to the Contributing Guidelines for coding standards and merge procedures.

Specifying Library Search Paths¶

By default, CMake searches several hard-coded system directories and some directories that are specified in environment variables like PATH for the locations of the dependencies. Here, several issues might be encountered: If some dependencies are installed in unusual locations, CMake can not find them. Further, some dependencies should refer to a certain version of a library and not the one that is installed system wide. If dependencies are installed in non-standard locations, you may need to spevify the library search paths using CMAKE_PREFIX_PATH:

    cmake -DCMAKE_PREFIX_PATH="/path/to/lib1;/path/to/lib2"

Often, you want to set predefined library paths for an HPC system. This can be done with a bash script, that can be placed in the build directory.

#! /bin/bash
dino_libs=<path of the installed libraries>
cmake \
-DDERIV_HALO=ON \
-DUSE_CANTERA=ON \
-DUSE_EGLIB=OFF \
-DCMAKE_Fortran_COMPILER=gfortran \
-DCMAKE_C_COMPILER=gcc \
-DCMAKE_CXX_COMPILER=g++ \
-DCMAKE_BUILD_TYPE=Release \
-DHOSTTYPE=PC_LINUX_AMD64 \
-DCMAKE_PREFIX_PATH=\
"$dino_libs/lapack;\
$dino_libs/hypre;\
$dino_libs/fftw;\
$dino_libs/sundials;\
$dino_libs/cantera;\
$dino_libs/hdf5" ../

Warning

Make sure to set the right paths that lead CMake to the installed libraries that you whish to use, otherwhise compilation will fail!

Note

You can set the CMAKE_PREFIX_PATH in your .bashrc file if you want to refer to it permanently.

HOSTTYPES¶

To help setting up DINO on commonly used platforms, the HOSTTYPE variable can be set when first running CMake. Each HOSTTYPE sets up different variables that are specific to a certain platform, such as special compiler flags or library search paths. Currently, the available HOSTTYPEs are: - DEFAULT: Default compiler flags for most systems - PC_LINUX_AMD64: Flags for compilation on common Linux systems. It works for example with SuperMUC in lib. (Might work on 32-bit systems) - KARMAN: Flags for compilation on KARMAN cluster, local machine of our group in Magdeburg. Also sets library search paths specific to KARMAN. (might also work on CentOS-based systems)

To use one of the predefined HOSTTYPES, add -DHOSTTYPE=<HOSTTYPE> to the command line when running CMake. Replace <HOSTTYPE> to the HOSTTYPE of your choice, as shown in the bash script before.

To create a custom HOSTTYPE, you can copy and modify one of the existing ones. Go to CMAKE/modules/ in $DINO_HOME/ and copy the file DEFAULT-HostProfile.cmake. Each HOSTTYPE filename must follow the naming scheme <NAME>-HostProfile.cmake. The commands in the file will be executed when specifying -DHOSTTYPE=<NAME> on the command line. Currently the existing HOSTTYPES set some compiler flags based on the selected compiler and set library search paths.