LAMMPS Builds with GPU Support

There are 2 options for GPU support in LAMMPS: via the GPU package and the KOKKOS package. Depending on your simulation and the features that are available with these packages, you can decide which package to use at run time. To build these 2 packages in a single LAMMPS binary, you can following cmake configuration:

    mkdir build
    cd build
    cmake -D PKG_KSPACE=on -D PKG_MOLECULE=ON -D PKG_RIGID=ON \
      -D PKG_CLASS2=ON -D PKG_AMOBEA=ON -D PKG_ASPHERE=ON \
      -D PKG_REPLICA=ON -D PKG_GPU=ON -D GPU_API=cuda -D GPU_ARCH=sm_80 \
      -D BUILD_MPI=yes -D BUILD_OMP=ON \
      -D PKG_KOKKOS=ON -D Kokkos_ARCH_PASCAL60=off -D Kokkos_ARCH_NATIVE=yes \
      -D Kokkos_ARCH_AMPERE80=ON -D Kokkos_ENABLE_CUDA=yes -D Kokkos_ENABLE_OPENMP=yes \
      -DFFT_KOKKOS=CUFFT \
      -D CMAKE_CXX_COMPILER=`which mpicxx` -D MPI_C_COMPILER=`which mpicc`  ../cmake
    make -j4
    

Here you are using the default KISS FFT for the CPU side. You can use -D FFT=FFTW3 and specify the available FFTW3 libraries.

Note that you need -D Kokkos_ARCH_AMPERE80=ON for the KOKKOS package, the GPU package is actually built with all the architectures from 60 through 90.

If your cluster has Intel MKL, you can use FFT provided by MKL by loading the suitable module:

    module load mkl
    

which should set the environment variable MKLROOT and then specify -D FFTW3_INCLUDE_DIR=$MKLROOT/include/fftw. The cmake command would be

    cmake -D PKG_KSPACE=on -D PKG_MOLECULE=ON -D PKG_RIGID=ON \
      -D PKG_CLASS2=ON -D PKG_AMOBEA=ON -D PKG_ASPHERE=ON \
      -D PKG_REPLICA=ON -D PKG_GPU=ON -D GPU_API=cuda -D GPU_ARCH=sm_80 \
      -D BUILD_MPI=yes -D BUILD_OMP=ON \
      -D PKG_KOKKOS=ON -D Kokkos_ARCH_PASCAL60=off -D Kokkos_ARCH_NATIVE=yes \
      -D Kokkos_ARCH_AMPERE80=ON -D Kokkos_ENABLE_CUDA=yes -D Kokkos_ENABLE_OPENMP=yes \
      -DFFT_KOKKOS=CUFFT -DFFT=MKL -DFFTW3_INCLUDE_DIR=$MKLROOT/include/fftw \
      -D CMAKE_CXX_COMPILER=`which mpicxx` -D MPI_C_COMPILER=`which mpicc`  ../cmake
    

On OLCF Frontier with AMD GPUs and the HIP ROCm toolchain, I loaded the following modules and the linker flags:

    module load amd/6.3.1
    module load rocm/6.3.1
    module load cray-fftw
    module load hwloc
    module load cmake
    LINKFLAGS="-L${MPICH_DIR}/lib -lmpi_amd ${PE_MPICH_GTL_DIR_amd_gfx90a} ${PE_MPICH_GTL_LIBS_amd_gfx90a}"
    

Note that the MPI lib is libmpi_amd and the GTL library is needed for linking for both GPU and KOKKOS packages. Then I used the following cmake command to build with the GPU package:

    cmake -S cmake -B build-gpu -C cmake/presets/basic.cmake -D PKG_DIPOLE=on -D PKG_DPD-BASIC=on \
      -D PKG_SPH=on -DPKG_REPLICA=on -DPKG_MC=on -D PKG_GPU=yes -D GPU_API=hip -D HIP_ARCH=gfx90a \
      -D CMAKE_CXX_COMPILER=hipcc -D CMAKE_C_COMPILER=hipcc -DCMAKE_BUILD_TYPE=RelWithDebInfo \
      -DFFT=FFTW3 -D FFTW3_INCLUDE_DIRS=$FFTW_ROOT/include -D FFTW3_LIBRARIES=$FFTW_ROOT/lib \
      -D BUILD_OMP=on -D CMAKE_EXE_LINKER_FLAGS="${LINKFLAGS}"
    cmake --build build-gpu -j8
    

For the KOKKOS build, with the same modules loaded I used the following command

    cmake -S cmake -B build-kokkos -C cmake/presets/basic.cmake -D PKG_DIPOLE=on -D PKG_DPD-BASIC=on \
      -D PKG_SPH=on -DPKG_REPLICA=on -DPKG_MC=on -D PKG_KOKKOS=yes -D Kokkos_ENABLE_HIP=on -D Kokkos_ARCH_VEGA90A=on \
      -D Kokkos_ENABLE_HWLOC=on -D Kokkos_ENABLE_HIP_MULTIPLE_KENREL_INSTANTIATIONS=on -D Kokkos_ENABLE_SERIAL=on \
      -D CMAKE_CXX_COMPILER=hipcc -D CMAKE_C_COMPILER=hipcc -DCMAKE_BUILD_TYPE=RelWithDebInfo \
      -D FFT_KOKKOS=hipFFT -DFFT=FFTW3 -DFFTW3_INCLUDE_DIRS=$FFTW_ROOT/include -DFFTW3_LIBRARIES=$FFTW_ROOT/lib \
      -DBUILD_OMP=off -DCMAKE_EXE_LINKER_FLAGS="${LINKFLAGS}"
    cmake --build build-kokkos -j8
    

To build both GPU and KOKKOS packages, I used the following cmake command (note the KOKKOS OPENMP backend is on):

    cmake -S cmake -B build -C cmake/presets/basic.cmake -D PKG_DIPOLE=on -D PKG_DPD-BASIC=on \
      -D PKG_SPH=on -DPKG_REPLICA=on -DPKG_MC=on -D PKG_GPU=yes -D GPU_API=hip -D HIP_ARCH=gfx90a \
      -D CMAKE_CXX_COMPILER=hipcc -D CMAKE_C_COMPILER=hipcc -DCMAKE_BUILD_TYPE=RelWithDebInfo \
      -D PKG_KOKKOS=yes -D Kokkos_ENABLE_HIP=on -D Kokkos_ARCH_VEGA90A=on -D Kokkos_ENABLE_HWLOC=on \
      -D Kokkos_ENABLE_HIP_MULTIPLE_KENREL_INSTANTIATIONS=on -D Kokkos_ENABLE_OPENMP=on \
      -D FFT_KOKKOS=hipFFT -DBUILD_OMP=on -DCMAKE_EXE_LINKER_FLAGS="${LINKFLAGS}"
    cmake --build build -j8