1. Overview

General introduciton to initial GPE project can be found here.

2. GPU Hardware

The current GPU system at DESY (Zeuthen) for the GPE project consists of two servers with dual nVidia Tesla C2050 GPU cards. They are named gpu1 and gpu2 and are also used as a testbed for new developments in GPU-to-GPU networking with custom designed interconnects and InfiniBand.

3. Environment

Currently on the systems the newest version of the CUDA SDK 8.0 alongside with device drivers and libraries are installed. The Software Development Kit provides the following:

• CUDA driver 375.66
• CUDA SDK 8.0

• GPU Debugging & Profiling Tools

• GPU-Accelerated Math Libraries
• GPU-Accelerated Performance Primitives (Thrust library)

4. GPU Benchmarks

For evaluation and development a set of common benchmarks, as well as specially designed micro benchmarks were run on the gpu1/2 systems.

4.1. Custom low-level benchmarks

Custom designed benchmarks use OpenMPI and OpenMP for task parallelization and allocation on the host CPUs and evaluate the following performance metrics discussed below. For allocating benchmark threads to particular processor socets and cores specific options to mpirun were used. These include OpenMPI processor/memory affinity options described below:

mpirun --mca mpi_paffinity_alone 1 --mca rmaps_rank_file_path rank.cfg -np 4 executable

The option mpi_paffinity_alone=1 enables processor (and potentially memory) affinity. The options shown can be also defined via environment variables:

export OMPI_MCA_mpi_paffinity_alone=1
export OMPI_MCA_rmaps_rank_file_path=rank.cfg

• Sample rank files for different cores/processes/thread configurations are shown below:

#1: 1x2 cores / 2x processes / 1x thread per process
rank 0=znpnb90 slot=0
rank 1=znpnb90 slot=1

#2: 1x4 cores / 2x processes / 2x threads per process
rank 0=gpu1 slot=0-1
rank 1=gpu1 slot=2-3

#3: 2x4 cores / 2x processes / 4x threads per process
rank 0=gpu1 slot=0:*
rank 1=gpu1 slot=1:*

Using this a couple of benchmarks were implemented and run. These measure bandwidth of host do device memory transactions, as well as latencies of memory transactions for the case where two GPUs work simultaneously.

1. Memory Bandwidth for unpinned memory and synchronous / asynchronous transfers

   • Here the bandwidth of host to device memory copy operations is measured. The host memory areas used are allocated with common malloc() calls and are not pinned to physical page addresses - thus they are subject to page swapping. On the other hand memory is pinned when allocating it via the cudaHostAlloc() call. For differentiating between synchronous and asynchronous transfers we used cudaMemcpy and cudaMemcpyasync. The effects of both transfer type and memory pinning are scribed here.

2. Latency of host-ot-GPU memory copy operations for mulltiple GPUs
   • Here the latency for host to device memory copy operations is measured. This time however the memory regions of the host memory are pinned to physical addresses and only asynchronous memory transfer is used. The difference in the setup here is that both GPUs run the benchmark simultaneously, and we differe between two configurations - parallel (host process running on CPU socket 0 uses GPU 0, and process running on CPU socket 1 uses GPU 1) and cross (process on CPU socet 0 uses GPU 1 and vice versa).

   • Latecny of host-to-GPU memory copy operations for parallel configuration

   • Latency of host-to-GPU memory copy operations for corss configuration

3. Bandwidth and latency of GPU-to-GPU communication
   1. MPI send/recv vs. CUDA 4.0 peer-to-peer communication primitives

4. GPU-to-InfiniBand hardware datapath propagation delay

   • For measuring the hardware datapath latency between host CPU and InfiniBand network adapter, as well as beteen GPU and InfiniBand network adapter, a micro benchmarks was developed which utilizes the loopback capability of the INfiniband HCA. In such mode packets send from a process running on the host CPU are looped back "inwards".

   • For the current one-server setup such measurements are not possible however, because of the limitation the used QSFP loopback connector poses - if the HCA is not directly connected to other HCA or infiniBand switch, then the subnet manager throws an error for duplicate addresses.

4.2. Application-level benchmark suites

For ensuring consistency of performance with real-world applications, the Scalable Heter Ogeneous Computing (SHOC) benchmark suite was run on the gpu1 system too. This benchmark suite gives not only benchmark results for CUDA implementations of key algorithms but also corresponding implementations of the more general OpenCL parallel programming framework.

The SHOC benchmark suite currently contains benchmark programs, categoried based on complexity. Some measure low-level "feeds and speeds" behavior (Level 0), some measure the performance of a higher-level operation such as a Fast Fourier Transform (FFT) (Level 1), and the others measure real application kernels (Level 2).

• Level 0
  • Bus-Speed-Download: measures bandwidth of transferring data across the PCIe bus to a device.
  • Bus-Speed-Readback: measures bandwidth of reading data back from a device.
  • Device-Memory: measures bandwidth of memory accesses to various types of device memory including global, local, and image memories.
  • Kernel-Compile: measures compile time for several OpenCL kernels, which range in complexity
  • Max-Flops: measures maximum achievable floating point performance using a combination of auto-generated and hand coded kernels.
  • Queue-Delay: measures the overhead of using the OpenCL command queue.
• Level 1
  • FFT: forward and reverse 1D FFT.
  • MD: computation of the Lennard-Jones potential from molecular dynamics
  • Reduction: reduction operation on an array of single or double precision floating point values.
  • SGEMM: matrix-matrix multiply.
  • Scan: scan (also known as parallel prex sum) on an array of single or double precision floating point values.
  • Sort: sorts an array of key-value pairs using a radix sort algorithm
  • Spmv: sparse matrix-vector multiplication
  • Stencil2D: a 9-point stencil operation applied to a 2D data set. In the MPI version, data is distributed across MPI processes organized in a 2D Cartesian topology, with periodic halo exchanges.
  • Triad: a version of the STREAM Triad benchmark, implemented in OpenCL and CUDA. This version includes PCIe transfer time.
• Level 2
  • S3D: A computationally-intensive kernel from the S3D turbulent combustion simulation program

4.3. GPUDirect

The term GPUDirect refers to a mechanism which allows different drivers to share pinned memory pages. This is realised by a (minor) modification of the Linux kernel and change in the network device drivers (i.e. IB device driver). The pages are pinned via the CUDA driver and the drivers who want to use these pages have to implement a mechanism which allow these to be notified about any change.

• Kernel patch - The required patch has not made it into the current SL6 kernel, i.e. the kernel needs to be patched. Unfortunately the patch provided by NVIDIA needs to be changed in order to match the current SL6 version. This is doable but of course means that the performance can only be verified for an (temporary) experimental setup and cannot make it into the standard deployment channel.
• Using GPUDirect - programms just have to allocate memory using cudaMallocHost (instead of, e.g., malloc) and use the pointer for MPI send/recv functions. NVIDIA provides this example TODO.

5. GPU Applications

Recent applications utilizing the gpu1 system at DESY (Zeuthen) are Chroma-based LQCD munerical simulationd and applications from the field of Astro Particle Physics.

5.1. Debugger and Profiler Tools

• Compute Visual Profiler
• CUDA Debugger

6. Monitoring

• gpu1 in Nagios
• GPU cores temperature
• host and device free and/or used memory
• GPU core frequency
• GPU cores utilization/load

7. New GPU server

An example specification for a second GPU server can be as follows:

• Supermicro Barebone 7046GT-TRF, 1400 Watt
• Supermicro X8DTG-QF Mainboard
• 2x Intel Xeon X5667
• Dual Intel 5520 Rev. 22 (Tylersburg) chipset
• 48 GB DDR3-1333 with ECC
• 2x 147GB SAS HDD
• 2x 450GB SAS HDD
• LSI 8704ELP SAS/S-ATA RAID Controller incl. BBU
• 2x NVIDIA Fermi C2050 GPUs
• Supermicro 19" Rackschienen for Barebone 7046GT-TRF
• Mellanox MHQH19B-XTR ConnectX 2 Adapter Card, single Port QSFP, IB 40GB/s, PCI-Express x8

Optional components:

• 2x Intel Xeon X5675 (statt Intel X5667)
  • [6 instead of 4 cores pro CPU]
• 2x NVIDIA Fermi C2070 GPUs (statt C2050)

8. Success/failure stories

Peer-to-Peer GPU memory access restrictions on dual Intel IOH machines

NOTE: Sections marked with need further discussion or are work in progress