Building b2000++pro on the HLRS system Vulcan for the NEC SX-Aurora TSUBASA

This is a step by step instruction to build b2000++ for the NEC SX-Aurora system on the HLRS Vulcan system specifically. Some details would vary on other systems, but the general procedure remains the same.

Currently there is only a version tested without Intel TBB, but with MPI parallelization.

Introduction and Prerequisites

Vulcan makes use of the workspace mechanism, and we will build all the software in such a workspace. To access the DLR repositories from the systm we need a reverse tunnel to get from Vulcan to the DLR gitlab servers:

ssh -R 8888:gitlab.dlr.de:22 vulcan

Where the ssh configuration contains a section describing the vulcan host:

Host vulcan
 Hostname vulcan.hww.hlrs.de
 User $yourhlrsaccount$
 ForwardAgent yes

and your system that you invoke the SSH command from does is allowed to access the hww network.

With this in place we can clone the three repositories on vulcan:

WS=`ws_allocate aurora 30`
echo "export WS=$WS" >> $HOME/.profile
mkdir -p $WS/sources
cd $WS/sources
git clone ssh://git@localhost:8888/b2000coop/b2000pp.git
git clone ssh://git@localhost:8888/b2000coop/memcom.git
git clone ssh://git@localhost:8888/b2000coop/CMakeModules.git

Here we establish the environment variable $WS to refer to the workspace subsequently. By registering the variable in the $HOME/.profile this setting is available also after logging out and in again.

To get a vector node for an hour of interactive use, run:

qsub -q vector -l select=1:node_type=aurora:mpiprocs=24,walltime=3600 -I

But in general it is probably more convenient to use batch scripts for the compilation.

To compile with cmake, NEC provides a toolchain for the vector engine (VE). See also their documentation on it. To make the installation process as smooth as possible, use the same INSTALL_PREFIX for all the software we need to compile and install.

Memcom

Compiling memcom with the provided toolchain is fairly straight forward and illustrated in this job script:

#!/bin/bash
#PBS -q vector
#PBS -N MCcomp
#PBS -l select=1:node_type=aurora:mpiprocs=24,walltime=36000
#PBS -m abe
#PBS -M your.name@world.org

source /opt/nec/ve/nlc/3.0.0/bin/nlcvars.sh
mkdir -p $WS/build/mc
cd $WS/build/mc
cmake -DCMAKE_INSTALL_PREFIX=$HOME/.local/ve \
  -DCMAKE_MODULE_PATH=$WS/sources/CMakeModules \
  -DCMAKE_TOOLCHAIN_FILE=/opt/nec/ve/share/cmake/toolchainVE-MPI.cmake \
  -DBUILD_PYMEMCOM_WITH_PYTHON3=OFF \
  $WS/sources/memcom 2>&1 | tee config.log
make -j 24 2>&1 | tee build.log
make install 2>&1 | tee install.log

This installs memcom into the $HOME/.local/ve directory. Of course, you could also choose to install it into some different location by changing the path. The three generated log files contain the screen messages of the the respective phases. The -m abe option to qsub tells the batch system to send you an email on abortion, begin and end of the job. We don’t support PyMemcom on the vector engines, so we need to turn that off.

Building arpack

Get arpack-ng from https://github.com/opencollab/arpack-ng.

With the autotools: after running sh bootstrap (on the vector node, either interactively or by job-script) to create the configure file, change the configure file to have ac_link use nfort instead of the C compiler, then run the job-script in the arpack-ng directory:

#!/bin/bash
#PBS -q vector
#PBS -N APcomp
#PBS -l select=1:node_type=aurora:mpiprocs=24,walltime=36000
#PBS -m abe
#PBS -M your.name@world.org

source /opt/nec/ve/nlc/3.0.0/bin/nlcvars.sh

CC=ncc CXX=nc++ FC=nfort F77=nfort ./configure --with-blas=/opt/nec/ve/nlc/3.0.0/lib/libblas_sequential.a --with-lapack=/opt/nec/ve/nlc/3.0.0/lib/liblapack.a --enable-icb --prefix=$HOME/.local/ve --enable-static --disable-shared 2>&1 | tee config.log
make -j 24 2>&1 | tee build.log
make install 2>&1 | tee install.log

Building nlopt

Get nlopt from https://github.com/stevengj/nlopt and put it into $WS/sources/nlopt. Building it is illustrated in the following job script:

#!/bin/bash
#PBS -q vector
#PBS -N NLcomp
#PBS -l select=1:node_type=aurora:mpiprocs=24,walltime=36000
#PBS -m abe
#PBS -M your.name@world.org

source /opt/nec/ve/nlc/3.0.0/bin/nlcvars.sh
export PKG_CONFIG_PATH=$HOME/.local/ve/lib/pkgconfig

mkdir -p $WS/build/nlopt
cd $WS/build/nlopt
cmake -DCMAKE_INSTALL_PREFIX=$HOME/.local/ve \
      -DCMAKE_TOOLCHAIN_FILE=/opt/nec/ve/share/cmake/toolchainVE-MPI.cmake \
      $WS/sources/nlopt
make -j 24 2>&1 | tee build.log
make install 2>&1 | tee install.log

Building metis

Metis (https://github.com/KarypisLab/METIS) requires the installation of GKlib (https://github.com/KarypisLab/GKlib) and we need those two libraries for both, the vector engine and the vector host, as Mumps uses VH offloading for Metis.

Changes in GKlib for compilation on the Aurora system:

Deactivate tests:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 9cd1b4b..6867293 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -22,8 +22,8 @@ if(UNIX)
   target_link_libraries(GKlib m)
 endif(UNIX)

-include_directories("test")
-add_subdirectory("test")
+#include_directories("test")
+#add_subdirectory("test")

 install(TARGETS GKlib
   ARCHIVE DESTINATION lib/${LINSTALL_PATH}

Avoid problem with signal.h (see https://github.com/KarypisLab/GKlib/issues/13#issuecomment-1327769070):

diff --git a/GKlibSystem.cmake b/GKlibSystem.cmake
index 31a1cf1..0be0268 100644
--- a/GKlibSystem.cmake
+++ b/GKlibSystem.cmake
@@ -111,6 +111,8 @@ if(GKRAND)
   set(GKlib_COPTIONS "${GKlib_COPTIONS} -DUSE_GKRAND")
 endif(GKRAND)

+set(GKlib_COPTIONS "${GKlib_COPTIONS} -D_POSIX_C_SOURCE=199309L")
+

 # Check for features.
 check_include_file(execinfo.h HAVE_EXECINFO_H)

Compile script for GKlib (put its sources into $WS/sources/GKlib):

#!/bin/bash
#PBS -q vector
#PBS -N GKcomp
#PBS -l select=1:node_type=aurora:mpiprocs=24,walltime=3600
#PBS -m abe
#PBS -M your.name@world.org

# for compiling on Vector engine: disable tests.
# see https://github.com/KarypisLab/GKlib/issues/13#issuecomment-1327769070
# for issue with signal.h
date
source /opt/nec/ve/nlc/3.0.0/bin/nlcvars.sh
export PKG_CONFIG_PATH=$HOME/.local/ve/lib/pkgconfig

# Build for VE
mkdir -p $WS/build/GKlib
cd $WS/build/GKlib
cmake -DCMAKE_TOOLCHAIN_FILE=/opt/nec/ve/share/cmake/toolchainVE.cmake \
      -DOPENMP=set \
      -DCMAKE_INSTALL_PREFIX=$HOME/.local/ve \
      $WS/sources/GKlib 2>&1 | tee config.log
make 2>&1 | tee build.log
make install 2>&1 | tee install.log

# Build for VH
mkdir -p $WS/build/vh/GKlib
cd $WS/build/vh/GKlib
cmake -DCMAKE_C_COMPILER=gcc \
      -DOPENMP=set \
      -DCMAKE_INSTALL_PREFIX=$HOME/.local/vh \
      $WS/sources/GKlib 2>&1 | tee config.log
make 2>&1 | tee build.log
make install 2>&1 | tee install.log
date

In Metis the Makefile actually does some additional work that we have to keep. Thus, rather than invoking cmake directly, we change the Makefile to adapt it for the vector engine compilation.

Changes in Metis Makefile to compile for the vector engine:

diff --git a/Makefile b/Makefile
index 8fe67e7..96b94c0 100644
--- a/Makefile
+++ b/Makefile
@@ -10,6 +10,7 @@ valgrind   = not-set
 openmp     = not-set
 shared     = not-set
 cc         = not-set
+necve      = not-set
 prefix     = ~/local
 gklib_path = ~/local

@@ -23,6 +24,12 @@ REALWIDTH = "\#define REALTYPEWIDTH 32"

 # Process configuration options.
 CONFIG_FLAGS = -DCMAKE_VERBOSE_MAKEFILE=1
+CONFIG_FLAGS += -DCMAKE_POSITION_INDEPENDENT_CODE=ON
+ifneq ($(necve), not-set)
+    CONFIG_FLAGS += -DCMAKE_TOOLCHAIN_FILE=/opt/nec/ve/share/cmake/toolchainVE.cmake
+else
+    BUILDDIR = build-vh
+endif
 ifneq ($(gklib_path), not-set)
     CONFIG_FLAGS += -DGKLIB_PATH=$(abspath $(gklib_path))
 endif

Compile script for Metis (put its sources into $WS/sources/METIS):

#!/bin/bash
#PBS -q vector
#PBS -N METIScomp
#PBS -l select=1:node_type=aurora:mpiprocs=24,walltime=3600
#PBS -m abe
#PBS -M your.name@world.org

date
source /opt/nec/ve/nlc/3.0.0/bin/nlcvars.sh
export PKG_CONFIG_PATH=$HOME/.local/ve/lib/pkgconfig

# Build for VE
rm -rf $WS/build/METIS
mkdir -p $WS/build
cp -r $WS/sources/METIS $WS/build
cd $WS/build/METIS
# To compile METIS for vector engine:
# Change the Makefile to include a necve option, and if it is set,
# set the toolchainVE.cmake accordingly.
make config prefix=$HOME/.local/ve necve=set 2>&1 | tee config.log
make install -j24 necve=set 2>&1 | tee build.log

# Build for the VH
rm -rf $WS/build/vh/METIS
mkdir -p $WS/build/vh
cp -r $WS/sources/METIS $WS/build/vh
cd $WS/build/vh/METIS
make config prefix=$HOME/.local/vh shared=set 2>&1 | tee config.log
make install -j24 2>&1 | tee build.log
date

Building mumps

Follow the instructions for NEC in the INSTALL file. Copy the content of the lib and include paths to the respective subdirectories in the INSTALL_PREFIX. Building takes quite long, best use multiple jobs (make -j 24).

Building b2000++pro

Compilation of b2000++pro with all the dependencies already installed as described above is illustrated in this job script:

#!/bin/bash
#PBS -q vector
#PBS -N b2comp
#PBS -l select=1:node_type=aurora:mpiprocs=24,walltime=36000
#PBS -m abe
#PBS -M your.name@world.org

export PKG_CONFIG_PATH=$HOME/.local/ve/lib/pkgconfig
source /opt/nec/ve/mpi/3.3.0/bin/necmpivars.sh
source /opt/nec/ve/nlc/3.0.0/bin/nlcvars.sh mpi

date
mkdir -p $WS/build/b2
cd $WS/build/b2
cmake -DCMAKE_INSTALL_PREFIX=$HOME/.local/ve \
      -DCMAKE_MODULE_PATH=$WS/sources/CMakeModules \
      -DCMAKE_TOOLCHAIN_FILE=/opt/nec/ve/share/cmake/toolchainVE-MPI.cmake \
      -DUSE_MPI=ON \
      -DMPI_HOME=/opt/nec/ve/mpi/3.3.0 \
      $WS/sources/b2000pp 2>&1 | tee config.log
make -j 24 VERBOSE=1 2>&1 | tee build.log
make install 2>&1 | tee install.log

NEC specific configuration settings are found in cmake/NECspecifics.cmake.

Running b2000++pro

Use mpiexec and run the executable in $HOME/.local/ve/libexec/b2000++.exe directly, rather than using the shell wrapper. Establish your environment with:

source /opt/nec/ve/mpi/3.3.0/bin/necmpivars.sh
source /opt/nec/ve/nlc/3.0.0/bin/nlcvars.sh mpi
export PKG_CONFIG_PATH=$HOME/.local/ve/lib/pkgconfig
export VE_LD_LIBRARY_PATH=/opt/nec/ve/nlc/3.0.0/lib:/opt/nec/ve/mpi/3.3.0/lib64/ve:/opt/nec/ve/mpi/3.3.0/lib64/ve:$HOME/.local/ve/lib:$HOME/.local/ve/lib64
export PATH=$PATH:$HOME/.local/ve/bin:$HOME/.local/ve/libexec

mpiexec -np 1 b2000++.exe input.b2m

Advice: run b2ip on mdl files on the host or frontend, rather than on the vector engine and use the database in runs on the vector engine.