Combining OpenMP and MPI

Some jobs will run efficiently using a combination of OpenMP and MPI, but care needs to be taken to make sure that OpenMP threads are correctly placed onto the CPU cores. Intel MPI provides a mechanism to do this using the I_MPI_PIN_DOMAIN environment variable, as described in section 3.2.3 of the Intel MPI Reference manual for more details (this can be found at /sw/sdev/impi/3.2/doc/Reference_Manual.pdf on zen). For example adding -genv I_MPI_PIN_DOMAIN omp to the mpiexec command should ensure correct placement when running a hybrid code. An example job file can be found at /usr/local/examples/qsub_script/hybrid.pbs

Note on dplace

An alternative mechanism for controlling affinity is dplace. If you are using dplace to control placement then you will need to disable Intel's affinity mechanism by setting the environment variable KMP_AFFINITY to disable e.g. setenv KMP_AFFINITY disable for C-shell type shells.

-- DavidAcreman - 17 Oct 2008