1
edit
Changes
→0.3 Release
== <font color="Peru">Project Name </font>==
PGO Related Bugs
== <font color="Peru">Project Description </font>==
Firefox crashes or experiences bugs after creating PGO builds on Windows. The focus of this project is to determine what is causing these crashes or bugs and either patch/fix them or find solutions to allow for the optimizations without incurring the problems.
What contributions are needed? *What should be profiled?*A write up on steps to produce profile.*Creating scripts to execute profile steps.<br />{| border="1"!style= "width:auto"|Name!style="width:25px"|!style="width:auto"|Contribution|-|Chris Bishop ( Example )||Script to open and close Firefox.|-|}<br /><br />*Need some bash scripting help.*I'm on IRC as chrisBee {| border="1"!style="width:auto"|Name!style="width:25px"|!style="width:auto"|Script Contribution|-|[[User:cdpatel1|Chinmay Patel]]||Created a small script to search and replace a piece of text in a file.|-|} == <font color="Peru">Project Tasks </font>==
<table style="width: 100%;" class="standard-table" cellborder=1 border=1 cellpadding=3 cellspacing=0>
<th style="width: 15%;">Task</th>
<th style="width: 5%;">Status</th>
<th style="width: 8%;">Target</th>
<th style="width: 25%;">Link(s)or Notes</th>
<tr>
<td>I need to reproduce the bugs or crashes other people were experiencing with their pgo builds.</td>
<td style="background-color: Orange; text-align: center; font-weight: bold; font-size: bigger">High</td>
<!-- color green when done --><td style="background-color: whitegreen; text-align: center; font-weight: bold;">Not StartedDone</td>
<td style="background-color: YellowGreen; text-align: center; font-weight: bold;">0.1 R</td>
<td>Unable to reproduce. Focusing on profiling Firefox. [http://dee132.blogspot.com/2008/10/change-of-direction.html Blog Post]</td>
</tr>
<tr>
<td>'''Compile Firefox PGO Tests'''</td>
<td>Compile Firefox with VS 2005 and Vista SDK and compile with VS 2008 w/o Vista SDK.</td>
<td style="background-color: Orange; text-align: center; font-weight: bold; font-size: bigger">lowLow</td>
<!-- color green when done --><td style="background-color: white; text-align: center; font-weight: bold;">Not Started</td>
<td style="background-color: YellowGreen; text-align: center; font-weight: bold;">0.3 R</td>
<td></td>
</tr>
<tr>
<td>'''Determine 0.1 Release'''</td>
<td>I need to determine what I'll be releasing for the 0.1 Release.</td>
<td style="background-color: Orange; text-align: center; font-weight: bold; font-size: bigger">High</td>
<!-- color green when done --><td style="background-color: green; text-align: center; font-weight: bold;">Done</td>
<td style="background-color: YellowGreen; text-align: center; font-weight: bold;">Sept 28th, 2008</td>
<td></td>
</tr>
<tr>
<td>'''Scenario Scripts'''</td>
<td>I need a script that will build multiple PGO builds in different directories using different profiling scenarios.</td>
<td style="background-color: Orange; text-align: center; font-weight: bold; font-size: bigger">High</td>
<!-- color green when done --><td style="background-color: green; text-align: center; font-weight: bold;">Done</td>
<td style="background-color: YellowGreen; text-align: center; font-weight: bold;">Oct 24th, 2008</td>
<td>I need some help from contributors to help me accomplish this. **Update** Thanks to Chinmay and Dave Humphrey I was able to create this script.</td>
</tr>
<tr>
<td>'''Create SQLite Upstream Bug'''</td>
<td>I have determined the pgo bug is actually with SQLite and it needs to be added to the upstream for it to be worked on.</td>
<td style="background-color: Orange; text-align: center; font-weight: bold; font-size: bigger">High</td>
<!-- color green when done --><td style="background-color: white; text-align: center; font-weight: bold;">On Going</td>
<td style="background-color: YellowGreen; text-align: center; font-weight: bold;">Oct 24th, 2008</td>
<td>Looking into how to actual do this</td>
</tr>
<tr>
<th colspan="6">0.2 Release</th>
</tr>
<tr>
<td>'''Setup Test Environment'''</td>
<td>I need to setup a test environment to duplicate the testing done on Thunderbird.</td>
<td style="background-color: Orange; text-align: center; font-weight: bold; font-size: bigger">High</td>
<!-- color green when done --><td style="background-color: green; text-align: center; font-weight: bold;">Done</td>
<td style="background-color: YellowGreen; text-align: center; font-weight: bold;">Nov 4th, 2008</td>
<td>I have this VM setup but I am not using yet since my XP OS that is not on a VM is much faster.</td>
</tr>
</table>
== <font color="Peru">Project Details </font>== === Conversations === {||width="15px"||December 6th|Ted Mielczarek|-||colspan="2"|I told Ted about how a large number of pgc (profile data) files were in a folder that gets deleted before the pgc files get used to optimize Thunderbird. It turns out that this folder gets clobbered to make sure if some one rebuilt their try after removing a component that it would not still be there. Ted suggest adding a flag that <code>ifndef MOZ_PROFILE_USE</code> then go ahead and clobber it otherwise don't.<br /><br />That is only one part of it. The pgc files need to be merged into their corresponding pgd (program database) files but the script that does this only looks in one directory. I think I can just tell it to look in both locations. The <code>pgomerge.py</code> does not return errors for files not found.<br /><br />Ted also said I should create another bug about this - how components can't be PGO'd on Win32.|} === PGO Details === {||width="15px"||[[PGO_Build_Details| Details]]|-||[[PGO_Related_Bugs/TB_Details|Thunderbird PGO Details]]|} ==== Thunderbird ===={||width="15px"||[[PGO_Build_Details/TB-Addrbook| Address Book enabled with PGO]]<br/>Turned on speed optimizations for the Address Book module. |}
{|class="collapsible collapsed" style= Project News "border-style:solid;border-width:thin;border-color:black"!align="left" style="background-color:#FF9933; font-weight:bold;border-style:solid;border-width:thin;padding: 2px 2px 2px 2px;"|The scripts I am using for running tests against my builds ( Oct 2nd, 2008 )|-|<code>#!/bin/sh</code><br/><span style="color:green;">''####### the .mozconfig file to use for this build''</span><br /><code>export MOZCONFIG=/f/mozilla/.mozconfigPGO</code><br /><span style="color:green;">''####### the subdirectory trunk holds my src files''</span><br /><code>cd trunk</code><br /><span style="color:green;">''####### time the profile build and output stdout and stderr to files that are timestamped''</span><br /><code>time make -f client.mk profiledbuild 1> /i/mozilla/pgo-output-`date +%s`.txt 2> /i/mozilla/pgo-errors-`date +%s`.txt''</code><br /><span style="color:green;">''####### after the build is completed done run the xpcshell tests and output stdout and stderr to file that are timestamped''</span><br /><code>make check 1> /i/mozilla/pgo-check-output-`date +%s`.txt 2> /i/mozilla/pgo-check-errors-`date +%s`.txt</code><br /><span style="color:green;">''####### change to the objdir directory then to the mochitest directory''</span><br /><code>cd /i/mozilla/ff-pgo/_tests/testing/mochitest</code><br /><span style="color:green;">''####### run the mochitest, run test right away, close the browser when done, log all output to a timestamped file''</span><br /><code>python runtests.py --autorun --close-when-done --log-file=/i/mozilla/pgo-mochitest-`date +%s`.log --file-level=DEBUG</code>|}<br />{|class="collapsible collapsed" style="border-style:solid;border-width:thin;border-color:black"!align="left" style="background-color:#FF9933; font-weight:bold;border-style:solid;border-width:thin;padding: 2px 2px 2px 2px;"|Here is the .mozconfig file ( Oct 2nd, 2008 )|-|<code>. $topsrcdir/browser/config/mozconfig</code><br /><code>mk_add_options MOZ_OBJDIR=/i/mozilla/ff-pgo</code><br /><code>mk_add_options MOZ_MAKE_FLAGS=-j5</code><br /><span style="color:green;">''####### this is the script to run after the first build to gather profile data''</span><br /><code>mk_add_options PROFILE_GEN_SCRIPT="sh /f/mozilla/pgo.sh"</code><br /><code>ac_add_options --disable-vista-sdk-requirements</code><br /><span style="color:green;">''####### this has to be enabled to perform the mochitests''</span><br /><code>ac_add_options --enable-tests</code><br /><code>ac_add_options --disable-debug</code><br /><code>ac_add_options --disable-accessibility</code>|}<br />{|class="collapsible collapsed" style="border-style:solid;border-width:thin;border-color:black"!align="left" style="background-color:#FF9933; font-weight:bold;border-style:solid;border-width:thin;padding: 2px 2px 2px 2px;"|Here is the script the profile build runs to gather profile data ( Oct 2nd, 2008 )|-|<pre>#!/bin/shexport NO_EM_RESTART=1mkdir $OBJDIR/_profileprofilecd /i/mozilla/ff-pgo/_tests/testing/mochitestpython runtests.py --autorun --close-when-done --log-file=/i/mozilla/pgo-profile-mochitest-`date +%s`.log --file-level=DEBUG</pre>|}
|} === Related Bugs === {||width="15px"||-|| Profiledbuild doesn't work on comm-central - [https://bugzilla.mozilla.org/show_bug.cgi?id=448515 448515]|-||Figure out why building jemalloc breaks with PGO enabled - [https://bugzilla.mozilla.org/show_bug.cgi?id=419470 419470]|-||sort out PGO-triggered bugs - [https://bugzilla.mozilla.org/show_bug.cgi?id=419893 419893]|-||win32 mozilla-central weirdness with PGO - [https://bugzilla.mozilla.org/show_bug.cgi?id=437002 437002]|-||To enable PGO on Thunderbird a module in the Mozilla core needs to have PGO disabled - [https://bugzilla.mozilla.org/show_bug.cgi?id=465592 465592] - Resolved|-||Components cannot be PGO'd on Windows - [https://bugzilla.mozilla.org/show_bug.cgi?id=468285 468285]|-||js3250!1.pgc remains in dist/bin during a PGO build - [https://bugzilla.mozilla.org/show_bug.cgi?id=467271 467271] |} === Patches ==={||width="15px"||-||Here is the [[PGO_Related_Bugs/Patches#465592|patch]] for bug [https://bugzilla.mozilla.org/show_bug.cgi?id=465592 465592].|-||Here is the proposed [[PGO_Related_Bugs/Patches#468285 | patches]] for bug [https://bugzilla.mozilla.org/show_bug.cgi?id=468285 468285].|} === Releases ==={||width="15px"||==== <span style="color:SaddleBrown;">0.1 Release</span> ==== {||width="15px"||{|!width="25px" |!|-|colspan="2"|'''Reproduce bug(s)'''|-||Was able to reproduce bug for sqlite pgo.|-|colspan="2"|'''Document how to reproduce bug(s).'''|-||To enable PGO for sqlite a line must be commented in a make file.<br />in path/to/src/db/sqlite/src/Makefile.in<br /> add a hash to the line <code>NO_PROFILE_GUIDED_OPTIMIZE = 1</code> <br />should look like <code>#NO_PROFILE_GUIDED_OPTIMIZE = 1</code><br />The build firefox with <code>make -f client.mk profiledbuild</code><br />in the .mozconfig file you'll need an extra parameter <code>mk_add_options PROFILE_GEN_SCRIPT="sh script/to/run.sh"</code><br /><em>My Script file</em> has the following: <pre>export NO_EM_RESTART=1 #this means 'do not restart for any reason' during automation we do not want any restarting mkdir $OBJDIR/_profileprofile cd $OBJDIR/_tests/testing/mochitestpython runtests.py --test-path=browser/ --autorun --close-when-done<ul/pre> |-|colspan="2"|'''Hone in on cause of bug(s).'''|-||Turns out the bug is in the sqlite3.c file. I can reproduce the error when creating a stand alone sqlite application and enabling PGO on the application.|-|colspan="2"|'''Running xpcshell and Mochitest against PGO build.'''|-||Since firefox right out crashes I am unable to run Mochitests but the xpcshell tests do run until they hit the storage tests.<br />{|class="collapsible collapsed" style="border-style:solid;border-width:thin;border-color:black"!align="left" style="background-color:#FF9933; font-weight:bold;border-style:solid;border-width:thin;padding: 2px 2px 2px 2px;"|SQLite PGO Errors in xpcshell ( Oct 16th, 2008 )|-|<pre>TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_bug-365166.js | test failed, see log../../_tests/xpcshell-simple/test_storage/unit/test_bug-365166.js.log:>>>>>>>*** Storage Tests: Trying to close! <<<<<<<TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_bug-393952.js | test failed, see log../../_tests/xpcshell-simple/test_storage/unit/test_bug-393952.js.log:>>>>>>>*** Storage Tests: Trying to close! <<<<<<<TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_bug-429521.js | test failed, see log../../_tests/xpcshell-simple/test_storage/unit/test_bug-429521.js.log:>>>>>>>*** Storage Tests: Trying to close! <<<<<<<TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_bug-444233.js | test failed, see log../../_tests/xpcshell-simple/test_storage/unit/test_bug-444233.js.log:>>>>>>>*** Storage Tests: Trying to close! <<<<<<<TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_connection_executeAsync.js | test failed, see log../../_tests/xpcshell-simple/test_storage/unit/test_connection_executeAsync.js.log:>>>>>>>*** Storage Tests: Trying to close! <<<<<<<TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_like.js | test failed, see log../../_tests/xpcshell-simple/test_storage/unit/test_like.js.log:>>>>>>>*** Storage Tests: Trying to close! <<<<<<<TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_like_escape.js | test failed, see log../../_tests/xpcshell-simple/test_storage/unit/test_like_escape.js.log:>>>>>>>*** Storage Tests: Trying to close! <<<<<<<make[2]: *** [check] Interruptmake[1]: *** [check] Interruptmake: *** [check] Interrupt</pre>|}||-|colspan="2"|'''Create strategy for profiling Firefox to improve performance. '''|-||I tried different approaches to profiling for Firefox to improve its performance but I was not able to help it at all. I have concluded that the existing script of opening and closing the browser is more than enough for profiling. The browser is rendered using the same components to render and load web pages this means profiling the loading and rendering of the browser will optimize the rendering and loading of web pages.|-|colspan="2"|'''Deliverable'''|-||I have updated the bug [https://bugzilla.mozilla.org/show_bug.cgi?id=419893 419893] with how to reproduce the bug, a text file of the errors, a zip file with the source code for my test app to produce pgo errors with sqlite as a stand alone app.|} |} ==== <span style="color:SaddleBrown;">0.2 Release</span> ===={||width="15px"|| {|!width="25px" |!|-|colspan="2"|'''Setup Testing Environment'''|-||Created Windows 2003 Server VM - but I am still using my faster Windows XP OS since it is not a VM. The test environment will be used to verify performance data once testing of PGO begins. Either in 0.3 or beyond.|-|colspan="2"|'''Compile Thunderbird with PGO'''|-||I successfully compiled Thunderbird with PGO. It required modifying a file. I blogged about it [http://dee132.blogspot.com/2008/11/success.html here].|-|colspan="2"|'''Compile Steps'''|-||I also have the steps to compile Thunderbird with PGO [[PGO_Related_Bugs/Thunderbird|here]].|-|colspan="2"|'''Contribute to Existing Bug'''|-|| The bug '''Profiledbuild doesn't work on comm-central''' [https://bugzilla.mozilla.org/show_bug.cgi?id=448515 448515] has been re-opened and I have posted my success results on this bug.|-|colspan="2"|'''Add PGO to one module'''|-||This will be pushed back to 0.3 Release|} <!-- end of inside 0.2 release -->|} ==== <span style="color:SaddleBrown;">0.3 Release</span> ===={||width="15px"||{|!width="25px" |!|-|colspan="2"|'''Patch to enable PGO for Thunderbird on Windows'''|-||I created a new [https://bugzilla.mozilla.org/show_bug.cgi?id=465592 bug] for allow PGO for Thunderbird on Windows and my patch has made it into the mozilla-core.<br/>This is the [[PGO_Related_Bugs/Patches | patch]] that landed.|-|colspan="2"|'''Add PGO to one module'''|-||It turns out that by default each module is optimized for size if using PGO. The problem that arose from this was the profile data was being collect but not used. The profile data was getting deleted at the beginning of the build cycle.|-|colspan="2"|'''Create a new bug for deleted profile data'''|-||Problem 1 is the profile data in the dist/bin/components folder is getting deleted. Problem 2 is even if the profile data wasn't getting deleted the pgomerge.py script is not looking for profile data in the dist/bin/components folder anyways. This means that any component that are profiled are not getting optimized.<br />Here are the [[PGO_Related_Bugs/Patches#468285|patches]] I'm proposing.<br/>I created a new [https://bugzilla.mozilla.org/show_bug.cgi?id=468285 bug] for this problem.|-|colspan="2"|'''Details of optimizing Thunderbird'''|-||Using the proposed patches 74 objects were profiled and 69 were optimized. Here are the [[PGO_Related_Bugs/TB_Details|details]].|} <!-- end 0.3 release -->|} |}<!-- end of table indent --> === Errors ==={||width="15px"||*Build Environment:**Windows XP (32-bit)**Visual Studio 2008**Vista SDK
CONTEXT:
Eax = 00000000 Esp = 0012ED18
Ebx = 02D5D215 Ebp = 0012ED2C
Ecx = 02D96601 Esi = 02D5D280
Edx = 02DAAE12 Edi = 00000305
Eip = 511953E5 EFlags = 00010246
SegCs = 0000001B SegDs = 00000023
SegSs = 00000023 SegEs = 00000023
SegFs = 0000003B SegGs = 00000000
Dr0 = 00000000 Dr3 = 00000000
Dr1 = 00000000 Dr6 = 00000000
Dr2 = 00000000 Dr7 = 00000000
</pre>
|}
make[6]: *** Deleting file `mozlcms.dll'
make[5]: *** [libs] Error 2
make: *** [profiledbuild] Error 2
</pre>
|}
<br />
|} <!-- end of table indent -->
Thunderbird PGO Errors<br />
The lcms module breaks pgo. [[PGO_Related_Bugs/TB_PGO_Errors|Found Here]]
<ol type="1">
<li>Simple Table: Uses an array and a linear search.
|3.6x
|} <!-- End result Table -->