#pragma section-numbers on
----
<<TableOfContents>>

----

= Overview =
The Intel Cluster Toolkit is a collection of MPI related tools, which is helpful when debugging, fine tuning and analyzing large MPI applications. ICT includes:

 * Intel C++ Compiler 11.1
 * Intel Debugger 11.1
 * Intel MPI Library 3.2
 * Intel MPI Benchmarks 3.2
 * Intel Trace Analyzer1 and Trace Collector 7.2

= Using ICT =
Each of this parts and its use at the [[Cluster|cluster environment]] in DESY Zeuthen is described next.

== Setting up the environment ==
Intitialize the Intel compiler and Intel MPI  environment:

{{{#!c
ini ic openmpi_intel
}}}
Initialize ICT environment:

{{{#!c
 source /opt/intel/itac/7.2.2.006/bin/itacvars.sh
 source /opt/intel/impi/3.2.2.006/bin/mpivars.sh
 source /opt/products/idb/11.0/bin/ia32/idbvars.sh
 export I_MPI_CC=/opt/products/bin/icc
}}}
== Compiling MPI applications with Intel MPI ==
Add

{{{
LDFLAGS     = -L$(VT_LIB_DIR) -lmpi $(VT_ADD_LIBS)
}}}
to the IMB Makefile.

== MPD daemons ==
The first step in setting up the MPD daemons, which are actually the environment for strating the parallel applications, is to set up an SSH connectivity environment with the help of the sshconnectivity.exp script and the imachines.LINUX file. The script is available <<AttachList>>

TODO: mpd.hosts

Start MPD daemons on all hosts in mpd.hosts file:

{{{#!c
# ./sshconnectivity.exp machines.LINUX
# mpdboot -n 8 --rsh=ssh -f ./mpd.hosts
}}}
== Tracing information ==
Run an application for the trace analyzer:

{{{#!c
# export VT_LOGFILE_FORMAT=STF;
# export VT_PCTRACE=5;
# export VT_PROCESS="0:N ON";
# export VT_LOGFILE_PREFIX=IMB-MPI1_inst;
# rm -fr $VT_LOGFILE_PREFIX; mkdir $VT_LOGFILE_PREFIX;
# mpiexec -n 2 itcpin --run -- ./IMB-MPI1 -npmin 2 PingPong;
# traceanalyzer ./IMB-MPI1_inst/IMB-MPI1.stf &;
}}}
After you are done with running and analysing your application shut down the MPD daemons with the command:

{{{#!c
# mpdallexit
}}}
== Fine tuning ==
Invoke mpitune:

{{{#!c
# mpitune -f machines.LINUX -o ./ --app mpiexec -genv MPIEXEC_DEBUG 1 -n 2 ./IMB-MPI1 -np 2 PingPong
# mpiexec -tune app.conf -n 4 ./IMB-MPI1 -np 2 PingPong
}}}
== Debugging ==
Start execution of the MPI program in debugging mode on 8 nodes each running 2 processes:

{{{#!c
# mpiexec -idb -genv MPIEXEC_DEBUG 1 -n 8 ./IMB-MPI1 -np 2 PingPong
}}}