Compiler Flags

The Intel compilers have several flags which can help with detecting errors (e.g. out of bounds array access). ifort has a -check all option which enables run-time checking; this slows the code down significantly but can be very useful when debugging. Adding -g -traceback flags will tell you where the error is occurring. See the ifort man page for more details of the available flags. For checking MPI you can use the -check_mpi flag (version 4.0 and later), see the CheckingMPI page for more details.

Segmentation Faults

If your code is generating a segmentation fault this advice from Intel may be useful.

The GNU Debugger

This is called gdb and can be used to debug an executable called myprog using the command

gdb myprog

This will load your executable into gdb, now type run to start it running. For more information see the gdb home page.

To start an MPI run under gdb use

mpirun -gdb -np 4 myprog

Intel Debugger

The intel debugger (idb) is available, with the graphical interface also available on zen-viz. Use idb to start the graphical interface or idbc to start the command line interface. The versions of Intel Cluster Studio provided by the ics2013 and ics2013sp1 module can run idb using the mpiexec command e.g. mpiexec -idb -np 4 myprog

Distributed Debugging Tool (DDT)

We have a license for ddt from Allinea, to start it load the ddt module and type ddt. This is licensed to run on the log in node (the one called zen) but not on zen-viz. If you need to use ddt with a program running on zen-viz then see the "Attaching to a remote process" section below.

Submitting a job to the queue using DDT

You will need a modified job submission file to allow your job to run in the queue using DDT.

Copy the template PBS file (/usr/local/examples/ddt/example-ddt.qtf) into the directory where your job will run and edit as required (e.g. to change into the required working directory). You do not need to specify the executable in the mpirun line as this is set up from DDT.

Specify the location of the job submission file in DDT (session -> options -> Job submission)

You can now use DDT to submit your job to the queue.

When using fewer than 8 MPI processes per node you need to change the value of PROCS_PER_NODE_TAG to the number of MPI processes per node. This can be found from the run window by clicking the Change button (on the MPI implementation line) and selecting the Job submission tab.

When debugging an OpenMP application you will need to select threaded memory checking. From the Advanced section of the run window click the Settings button next to Enable Memory Debugging. From the Language menu select C/Fortran, threads.

Attaching to a remote process

If your program is already running, either on a compute node or on zen-viz, you can attach ddt to it as follows.

Firstly run cp /usr/local/examples/ddt/remote-exec ~/.ddt (which copies a small script into your ddt configuration directory.) In the ddt window select Session->Options->Other. This will bring up a window called Other settings which contains a box called Attach hosts file. Enter /usr/local/examples/ddt/nodes in this box and click OK (if you are only using zen-viz then put /usr/local/examples/ddt/zen-viz in this box). Now select Session->New Session->Attach and enter the path to your executable in the Application box. You should see your executable appear in the window under the Process name column. Select the processes you want ddt to connect to and click Attach to selected processes.

-- JohnRowe - 28 Jan 2008