54
edits
Changes
→Added Walkthrough section for Walkthrough Part I
We need to enable OpenMP support, but before we start configuring our projects, we need to switch our solution to ''Release'' mode configuration and then configure both projects in ''Release'' mode.
To switch our solution to ''Release'' mode configuration, right click on our solution (Solution 'GPU621 Walkthrough' (2 projects)) and select ''Configuration Manager...''. In ''Configuration Manager'' dialog, select ''Release'' from ''Active solution configuration'' dropdown and click ''OKClose'' to continue.
Now it is time to enable OpenMP support and disable optimization for both projects. Right click on project "Part ONE" in ''Solution Explorer'' and select ''Properties'' (Shortcut Alt + Enter when project "Part ONE" is selected). In ''Part ONE Property Pages'' dialog, expand ''Configuration Properties'' and select ''Language [Intel C++]'' sub-section under ''C/C++'' section (In the leftmost window), then enable OpenMP support by selecting ''Generate parallel Code (/Qopenmp)'' from ''OpenMP Support'' dropdown. Next, select Optimization sub-section under ''C/C++'' section, and disable optimization by selecting ''Disabled (/Od)'' from ''Optimization'' dropdown. Finally, enable OpenMP support and disable optimization for project "Part TWO" as we just did for project "Part ONE".
'''NOTE:''' Since we will be referring to the specific lines of code by its line numbers, it is important you enable line numbers in your Visual Studio (If it's not enabled already). To enable line numbers, select ''TOOLS > Options...'', then in ''Options'' dialog expand ''Text Editor'' section and select ''All Languages'' sub-section in the leftmost window. On the right side you will see ''Line Numbers'' as one of the checkboxes. Ensure that it is checked and click ''OK''.
====Walkthrough====
With both projects configured, we can start debugging our application. So lets begin with setting up the breakpoints in "mainPartOne" source file. For the purpose of this walkthrough, we would need to set up 10 breakpoints in "mainPartOne" source file on each line that contains /*** Break ... ***/ comment (Lines Numbers: 39, 42, 45, 79, 93, 101, 117, 137, 151, and 161). Once all the break points are in place, run the solution by selecting ''DEBUG > Start Debugging'' (Shortcut F5) and wait until the execution breaks at our first breakpoint.
At this point we need to enable tools that we will be using during debugging process. We will be taking advantage of the Threads, Processes, and Parallel Watch windows. Additionally, the Debug Location toolbar can be used to assist with debugging multiple threads. So lets enable these tools right now:
* To enable ''Processes'' window, select ''DEBUG > Windows > Processes'' or press Ctrl + Alt + Z;
* To enable ''Threads'' window, select ''DEBUG > Windows > Threads'' or press Ctrl + Alt + H;
* To enable ''Parallel Watch'' window, select ''DEBUG > Windows > Parallel Watch > Parallel Watch 1/2/3/4'' or press Ctrl + Shift + D, 1/2/3/4;
* To enable ''Debug Location'' toolbar, select ''VIEW > Toolbars > Debug Location'';
'''NOTE:''' It is important to position each window in a way that will allow to minimize switching between tabs to access required window information. In other words, you want to see all of the tools, that we just enabled, on the screen at the same time.
Assuming that the execution stopped at a breakpoint on line 39, we should discuss ''Processes'' window and ''Location Debug'' toolbar while both process are running.
Processes window allows us to view available processes, see current process, and switch between processes. It also allows us to attach, detach, and terminate process.
Debug Location toolbar allows us switch between processes, threads within processes, and available call stack frames. It also allows to flag threads and toggle between ''Show Only Flagged Threads'' and ''Show All Threads'' modes.
Now lets take a look at the Threads window, where we should see all active threads. Depending on the system, you may see either one thread called ''Main Thread'', or ''Main Thread'' and one or more system threads. Since we only care about the ''Main Thread'' at this point, lets highlight it (Flag it) by clicking on the flag icon in the leftmost column. Flagging our threads will help us to distinguish between our threads and system threads. It is always a good idea to flag the threads that are important and require a special attention. Click ''Continue'' or press F5 to proceed to the next breakpoint.
On line 42 we are entering a parallel region. This is where all the magic happens! However, before entering a parallel region, we should look at something very interesting... We declared and initialized five variables, some of which we will be using in a parallel region during our debugging process. We are interested in ''counterShared'', ''counterPrivate'', ''num'', and ''magnitude'' variables. So lets add these variables to our ''Parallel Watch'' window, so we can monitor what happens to each variable on each thread. To add a variable to the ''Parallel Watch'' window, you can either right click on the variable that you wish to add and select ''Add to Parallel Watch'' option, or you can enter the name of the variable directly into ''Parallel Watch'' by clicking on ''<Add Watch>'' column header and entering the name of the variable. Add all four variables to the ''Parallel Watch''.
'''NOTE:''' Since ''num'' is an array/pointer, we can only see the value of its first element. We can change that by double clicking on the ''num'' header (Or other array/pointer name) in the ''Parallel Watch'' window and appending ", {size}" to the name of the array/pointer (For example, if we wish to see first ten elements of the ''num'' array, we would change "num" to "num, 10").
Pay close attention to the value of the ''num'' array/pointer, because it is about to change as we enter the parallel region. Click ''Continue'' or press F5 to continue.
At this point you will notice that new threads are being forked (You may have to click ''Continue'' or press F5 a couple of times). While on line 45, you will notice two things:
# Now there are multiple rows in the ''Parallel Watch'' window, which means that multiple threads are being watched at the same time;
# Look at the value of the ''num'' array/pointer in the ''Parallel Watch'' window, it has changed and no longer valid. You probably think that this behaviour will break our program, but no, our program will be executed as expected with previously initialized values. However, we will not be able to monitor any array/pointer (That declared and initialized before entering a parallel region) using Watch, Parallel Watch, or Locals windows while in parallel region. This is where we can take advantage of the ''Memory'' window (Discussed at the end of this walkthrough). Click ''Continue'' or press F5 to proceed to the next breakpoint (Next breakpoint is on line 79, so you may have to click ''Continue'' or press F5 multiple times).
We need to flag all the remaining threads that appear in the ''Parallel Watch'' window by clicking on the flag icons in the leftmost column.
You will notice that threads that reach line 79 have their name changed to "GPU621 Thread - OpenMP ID: {OpenMP thread id}" (We take advantage of Microsoft's "void SetThreadName(DWORD dwThreadID, const char* threadName)" function to change the name of the thread). Ensure that all the flagged threads (Except "Main Thread") have their names changed (You may have to click ''Continue'' or press F5 multiple times). Changing the name of the thread may be very useful when debugging a multithreaded application with many threads, especially when using sorting by ''Name'' feature of the ''Threads'' window. Click ''Continue'' or press F5 to proceed to the next breakpoint.
On line 93 we can observe the accumulation into the thread private variable ''sum'' on each thread by using ''Parallel Watch'' window. After hitting the breakpoint on line 93 for the first time, you may notice that current thread (Yellow arrow in ''Threads'', and ''Parallel Watch'' windows indicate a current thread) has ''sum'' variable initialized to 0.0f, while other threads may have random (Garbage) value for ''sum'' variable. This happens because other threads did not reach line 87 (''sum'' declaration and initialization). You can observe the accumulation into ''sum'' on each thread later, when the execution hits line 93 breakpoint again.
It is important to understand what is happening in the following block of code (Line 95 to line 105), because it contains a common mistake that many new developers make when working with OpenMP. This block of code will be executed by each thread once at the most due to its condition, and all it will does is increments the ''counterShared'' once per thread and accumulates the result into ''counterPrivate''. Therefore, if ''counterPrivate'' is initialized to 0, it will always be equal to ''counterShared'' per thread. So the expression on line 101 (''if (counterPrivate != counterShared)'') should never be true.
'''NOTE:''' We initialize both counters (''counterPrivate'' and ''counterPrivate'') to 0 on lines 26 and 29. Then on line 42 we pass both counters to the parallel region as private (''private(counterPrivate)'') and shared (''shared(counterShared)'') variables.
Click ''Continue'' or press F5 to proceed to the next breakpoint (You may want to disable the breakpoint on line 93 after observing the accumulation into ''sum'' on different threads).
Once we hit the breakpoint on line 101, we can see that it will not be evaluated to true, since counterPrivate was not initialized and contains random (Garbage) value. Therefore, the expression on line 101 may or may not evaluate to true. The behaviour is unpredictable at this point and it is likely that we will hit the compiler intrinsic breakpoint (''__debugbreak ()'') on line 103, which is just a way to show that things could go wrong going forward.
Go ahead and click ''Continue'' or press F5 to see what happens.
So why ''counterPrivate'' is not initialized properly when we did initialize it on line 29, just before entering the parallel region. Well, it appears that we need to initialize it separately for each thread after entering a parallel region.
Lets uncomment line 89 to resolve this issue and try again. Now the expression on line 101 will never evaluate to true, and this is what we want.
Click ''Continue'' or press F5 until you reach a breakpoint on line 117 for the first time. ''Parallel Watch'' window observe the accumulation of thread private ''sum'' into shared ''magnitude'' variable. Click ''Continue'' or press F5 until execution breaks on line 137.
Now lets see how ''#pragma omp parallel for'' breaks up the work between threads. This can be done by adding ''i'' into ''Parallel Watch''. Do it now, and observe its values change as you hold F5 or click ''Continue'' many times. You will see that the initial value of ''i'' is different on each thread, but how different? Look at ''N_SIZE'' value... The work is equally divided. Awesome!
When you are ready to proceed, remove the breakpoint on line 137 and click ''Continue'' or press F5 multiple times to pass the barrier on line 151.
If everything went well, your program should get stuck. Introducing a deadlock condition (When none of the threads can execute, for more info visit [https://en.wikipedia.org/wiki/Deadlock Deadlock Wiki]). In this example, deadlock condition occurs, because one of the threads encounters one more barrier than the other threads. Therefore, barrier can never be placed inside ''parallel for'' region. Reference: [https://confluence.csiro.au/display/SC/Frequent+parallel+programming+errors Frequent Parallel Programming Errors].
Lets stop the execution and modify the expression to bypass this issue. On line 145, change the expression ''for (int i = 0; i <= numThreads; i++)'' to ''for (int i = 0; i < numThreads; i++)''. Now deadlock condition should not occur. Keep in mind that this is not a solution, but a workaround. You must NOT place barrier inside ''parallel for'' region.
Now run the solution again and click ''Continue'' or press F5 until you reach line 151.
One last thing that is worth mentioning about ''Parallel Watch'' window is the fact that variable values can be modified as we debug our program. Change the values of ''counterShared'' and ''counterPrivate'' (On each thread) variables to 101 by double clicking on variable values in the ''Parallel Watch'' window and modifying the value.
After clicking ''Continue'' or pressing F5 you will see that expression on line 154 (''if (counterPrivate == 101 && counterShared == 101)'') evaluates to true.
On line 161, check the value of ''num'' array/pointer. Incredible, it is initialized again, just like on line 39. Actually, is has always been there and the best way to see that is to use ''Memory'' window.
Enable ''Memory'' window by selecting ''DEBUG > Windows > Memory > Memory 1/2/3/4''.
Now restart our program, but make sure that there is a breakpoint on line 39. Once you hit the breakpoint on line 39, type "num" into ''Address'' field in ''Memory'' window and hit Enter. Then right click on the ''Memory'' window and select ''32-bit Floating Point''. Now you should see values of ''num'' array/pointer while you are in parallel region.
This concludes part one of this walkthrough.
