GPU621/Debugging OpenMP

Group Members

1. Jimmy Liu
2. Sami Ali
3. Leon Liu

// TODO: Create a tutorial on how to debug parallel programs using OpenMP in Visual Studio

// remember to add pictures if possible

// Steps

1. Read the Microsoft Documentation for your section

2. Paraphrase the contents of your section onto the Wiki

3. Test it yourself in Visual Studio and add examples

// Uploading Files

To upload a file, go to https://wiki.cdot.senecacollege.ca/wiki/Special:Upload

To include a file in a page, use [[File:File.jpg]] to use the full version of the file

Process and Thread

Process

A process is an instance of a program executed by the computer. Processes are started when programs are initiated, and they can be comprised of one or more threads.

On Windows operating systems, you can look at your running processes on the Task Manager.

Thread

A single thread is a set of instructions that the CPU schedules and executes independently. A process consists of one or more threads which are able to be run concurrently. This forms the basis of multithreading.

Multiple Processes

Background

Visual Studio supports the creation of solutions with multiple processes. Visual Studio's debugger will by default run the first project (selected in bold), but this can be changed in the Solution Explorer by right-clicking on a different project and selecting Set as Startup Project. This will allow you when debugging to switch between processes, as well as break, stop, and continue currently debugging processes.

Setup

Startup Project

To set the startup project or multiple projects:

1. Right-click the solution in the Solution Explorer and select Properties.
2. In the Properties window, select Common Properties > Startup Project.
3. Here you can change the Startup Project to either the currently selected project, a single startup project (this is selected by default), or multiple startup projects.
   1. When selecting Multiple startup projects, you can modify the execution order and action for each project between Start, Start without debugging, or None.
4. Save your settings by selected Apply or OK.

Command Scope

When debugging with multiple processes, the break, step, and continue debugger commands will affect all processes by default. This can have unintended consequences when debugging. To gain more control over the targets, you can and should change these settings. You can do so by going to Tools (or Debug) > Options > Debugging > General, and clearing the checkbox Break all processes when one process breaks.

Attach to Process

The Visual Studio debugger has the capability to attach to applications running in processes outside of Visual Studio locally and remotely. This is useful as you are able to debug external applications that are already built. You can use the debugger while attached to a separate app, though some features may be limited. To attach to a running process:

1. With the app and Visual Studio running, select Debug > Attach to Process.
2. Select the desired process from the list and select Attach.
   1. If you have multiple identical processes, use the w3wp process details or Command Line column to identify the correct process to use.
   2. In the Attach to field, change the setting to Automatic to the specific type if necessary. Automatic will work for most applications.

Keep in mind that Visual Studio will not attach to any child process spawned by the debugged process. You will have to manually attach it to the new child.

Registry Editor

The Windows Registry Editor can be used to automatically start a process in the debugger. This will allow you to debug the startup code for an app that is launched by another process, such as Windows services.

1. Start the Windows Registry Editor by running regedit.exe from the run window (Windows + R).
2. In the Registry Editor, go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options.
3. Select the folder of the app to start in the debugger.
   1. If the app isn't listed, right-click Image File Execution Options, select New > Key, and enter the application's name.
4. Right-click the new key in the tree and select New > String Value.
5. Change the name of the new value from New Value #1 to debugger.
6. Right-click the new value and select Modify. Change the value data to vsjitdebugger.exe and select OK.
7. In the Edit String dialog box, enter vsjitdebugger.exe in the Value data box, and then select OK.

Debugging

Switching Between Processes

While you've done all this work to be able to use multiple processes, only one process can be debugged at a time. You will need to break all processes in order to switch between processes. There are two ways to set the current process:

1. Open the Debug Location toolbar through View > Toolbars > Debug Location.
2. While you are debugging, select the process to set on the Debug Location toolbar.

The other method is to use the processes window.

1. While you are debugging, open the processes window through Debug > Windows > Processes.
2. Double-click the process you want to set as the current process. The current active process is marked by a yellow arrow.

You can now start debugging multiple processes with Visual Studio! You can break, step, and continue processes independent of each other, as well as being able to switch between multiple active processes in your solution. The next step is to familiarize yourself with the Visual Studio parallelization user interface.

Stopping Debugging

When you stop debugging, the process is ended by default. This will detach the debugger from any attached applications if necessary and then end the process. This will keep the previously attached process running. You can change this default setting by right-clicking the process in the Processes window and clearing the Detach when debugging stopped option.

User Interface

// TODO: Show the window and explain what it does for every single window below (medium)

// documentation for windows are here: https://learn.microsoft.com/en-us/visualstudio/debugger/debug-threads-and-processes?view=vs-2022

Primary tools include:

• Attach to Process Dialog
• Process Window
• Debug Location Bar

Tools in debugging threads include:

• Thread Windows
• Thread Markers in Source Windows
• Parallel Stacks Window
• Debug Location Toolbar
• Tasks Window
• GPU Threads Window

Attach to Process Dialog Box

In Attach to Process Dialog Box, you can attach:

• Process name
• Session number
• Menubar Title
• Process ID number
• Type (Managed v4.0; Managed v2.0, v1.1, v1.0; x86; x64; IA64)
• User Name

Actions you can perform include Select a process to attach to, Select a remote computer, and Change transport type for connecting to remote computers

Process Window

Processes you can attach include:

• Process Name
• Process ID number
• Path to process .exe
• Menubar Title
• State (Break. Running)
• Debugging (Native, Managed, and so on.)
• Transport type (default, native with no authentication)
• Transport Qualifier (remote computer)

Tools include:

• Attach
• Detach
• Terminate

Thread Window

Thread Window is a tool in Visual Studio that allows you to manage and monitor your program's threads during debugging.

It can be set up by adding breakpoints in your program in order to examine it's activity.

This is done by right-clicking the desired line and choose the Breakpoint option.

• • How to use breakpoints**

You can use this by going to Debug -> Windows -> Threads, or by pressing CTRL + ALT + H

This will allow the execution to stop at your breakpoints.

Walkthrough

Environment

IDE

• Microsoft Visual Studio 2019

Compiler

• Intel oneAPI DPC++/C++

Sample Code

#include <stdio.h>
#include <omp.h>

int main(int argc, char** argv) {
    int partial_Sum, total_Sum;

#pragma omp parallel private(partial_Sum) shared(total_Sum)
    {
        partial_Sum = 0;
        total_Sum = 0;

#pragma omp for
        {
            for (int i = 1; i <= 1000; i++) {
                partial_Sum += i;
            }
        }

        //Create thread safe region.
#pragma omp critical
        {
            //add each threads partial sum to the total sum
            total_Sum += partial_Sum;
        }
    }
    printf("Total Sum : % d\n", total_Sum);
    return 0;
}

Configuration

• Enable OpenMP for the current project file

Project - Property - C/C++ - Language [Intel C++] - OpenMP Support - Generate Parallel Code (/Qiopenmp)

Output

• The output of the sample program

Case A: Using the Thread Window

Serial Region

• When one of the breakpoints is reached and the debugger stops at the serial region, only the main thread exists

Parallel Region

• When one of the breakpoints is reached and the debugger stops at a parallel region, all threads are listed in the Thread window. (including main & worker threads)

Freeze Feature

• All threads can be frozen (including the main thread) so that the programmer can check the status of one specific thread
• Frozen thread can also be thawed at any time to make it back into service

Flag Feature

• All threads can be flagged in order to filter the list in the Threads window
• It is quite useful when there are too many threads
• By clicking `Show Flagged Threads Only`, only threads that are marked will be displayed

Search & Sort

• A specific thread can be searched using the Search bar located at the upper left corner

• By clicking the title of each column, existing threads can be sorted based on the selected column

Case B: Using the Parallel Stacks Window

Start Execution Until the 1st Breakpoint

When the program stops at the first break point, only the main thread exists in Parallel Stacks Window

Resume Execution Until the 2nd Breakpoint

• The Parallel Stacks Window reveals the current status of all threads in the program
• There are 2 Threads that come from `ntdll.dll`, which is a runtime library that we don't really care here
• There are 2 threads that are ahead of other worker threads
  • They have reached the barrier (`__kmp_linear_barrier_release_template` & `__kmp_yield`)
• There are 8 threads that are ready to move on

Resume Execution Until the 3rd Breakpoint

• When reaches the 3rd break point, Thanks to `#pragma omp critical`, all threads are waiting, trying to pass their result to main
• As shown in the picture, there are 11 threads are ready to pass their values
  • They are either in the current thread or in the waiting list (`__kmp_wait_4` / `__kmp_yield`)
• According to the code, we know that there will be 11 threads passing their calculated results to the main thread
• At this point, the value of `total_Sum` is 0

Resume Execution Until the 4th Breakpoint

• When the program reaches this step, all calculations have been completed
• As can be seen in the figure, all worker threads have reached `__kmp_linear_barrier_release_template`
• This means that their work is done
• And we can see that the result of the program is correct

Resume Execution Until the Last Breakpoint

• When the program ends, all threads are terminated and the program returns to only one main thread