Difference between revisions of "PGO Related Bugs"

From CDOT Wiki
Jump to: navigation, search
(→‎0.2 Release)
(→‎0.3 Release)
 
(25 intermediate revisions by the same user not shown)
Line 120: Line 120:
  
 
== <font color="Peru">Project Details </font>==
 
== <font color="Peru">Project Details </font>==
 +
 +
=== Conversations ===
 +
 +
{|
 +
|width="15px"|
 +
|December 6th
 +
|Ted Mielczarek
 +
|-
 +
|
 +
|colspan="2"|
 +
I told Ted about how a large number of pgc (profile data) files  were in a folder that gets deleted before the pgc files get used to optimize Thunderbird.  It turns out that this folder gets clobbered to make sure if some one rebuilt their try after removing a component that it would not still be there.  Ted suggest adding a flag that <code>ifndef MOZ_PROFILE_USE</code> then go ahead and clobber it otherwise don't.<br /><br />
 +
That is only one part of it.  The pgc files need to be merged into their corresponding pgd (program database) files but the script that does this only looks in one directory.  I think I can just tell it to look in both locations.  The <code>pgomerge.py</code> does not return errors for files not found.<br /><br />
 +
Ted also said I should create another bug about this - how components can't be PGO'd on Win32.
 +
|}
  
 
=== PGO Details ===
 
=== PGO Details ===
Line 126: Line 140:
 
|width="15px"|
 
|width="15px"|
 
|[[PGO_Build_Details| Details]]
 
|[[PGO_Build_Details| Details]]
 +
|-
 +
|
 +
|[[PGO_Related_Bugs/TB_Details|Thunderbird PGO Details]]
 
|}
 
|}
  
 +
==== Thunderbird ====
 +
{|
 +
|width="15px"|
 +
|[[PGO_Build_Details/TB-Addrbook| Address Book enabled with PGO]]<br/>
 +
Turned on speed optimizations for the Address Book module. 
 +
|}
  
 
=== Testing Scripts ===
 
=== Testing Scripts ===
Line 201: Line 224:
 
|
 
|
 
|win32 mozilla-central weirdness with PGO - [https://bugzilla.mozilla.org/show_bug.cgi?id=437002 437002]
 
|win32 mozilla-central weirdness with PGO - [https://bugzilla.mozilla.org/show_bug.cgi?id=437002 437002]
 +
|-
 +
|
 +
|To enable PGO on Thunderbird a module in the Mozilla core needs to have PGO disabled - [https://bugzilla.mozilla.org/show_bug.cgi?id=465592 465592] - Resolved
 +
|-
 +
|
 +
|Components cannot be PGO'd on Windows - [https://bugzilla.mozilla.org/show_bug.cgi?id=468285 468285]
 +
|-
 +
|
 +
|js3250!1.pgc remains in dist/bin during a PGO build  - [https://bugzilla.mozilla.org/show_bug.cgi?id=467271 467271]
 +
|}
 +
 +
=== Patches ===
 +
{|
 +
|width="15px"|
 +
|-
 +
|
 +
|Here is the [[PGO_Related_Bugs/Patches#465592|patch]] for bug [https://bugzilla.mozilla.org/show_bug.cgi?id=465592 465592].
 +
|-
 +
|
 +
|Here is the proposed [[PGO_Related_Bugs/Patches#468285 | patches]] for bug [https://bugzilla.mozilla.org/show_bug.cgi?id=468285 468285].
 
|}
 
|}
  
Line 207: Line 250:
 
|width="15px"|
 
|width="15px"|
 
|
 
|
−
==== 0.1 Release ====
+
==== <span style="color:SaddleBrown;">0.1 Release</span> ====
  
 
{|
 
{|
Line 309: Line 352:
 
|}
 
|}
  
−
==== 0.2 Release ====
+
==== <span style="color:SaddleBrown;">0.2 Release</span> ====
 
{|
 
{|
 
|width="15px"|
 
|width="15px"|
Line 326: Line 369:
 
|-
 
|-
 
|
 
|
−
|I successfully compiled Thunderbird with PGO.  It required modifying a file.  I blogged about it [http://dee132.blogspot.com/2008/11/success.html here]. <br />I also have the steps to compile Thunderbird with PGO [[PGO_Related_Bugs/Thunderbird|here]].
+
|I successfully compiled Thunderbird with PGO.  It required modifying a file.  I blogged about it [http://dee132.blogspot.com/2008/11/success.html here].
 +
|-
 +
|colspan="2"|'''Compile Steps'''
 +
|-
 +
|
 +
|I also have the steps to compile Thunderbird with PGO [[PGO_Related_Bugs/Thunderbird|here]].
 
|-
 
|-
 
|colspan="2"|'''Contribute to Existing Bug'''
 
|colspan="2"|'''Contribute to Existing Bug'''
Line 342: Line 390:
 
|}
 
|}
  
−
==== 0.3 Release ====
+
==== <span style="color:SaddleBrown;">0.3 Release</span> ====
 
{|
 
{|
 
|width="15px"|
 
|width="15px"|
 
|
 
|
−
To be Announced...
+
{|
 +
!width="25px" |
 +
!
 +
|-
 +
|colspan="2"|'''Patch to enable PGO for Thunderbird on Windows'''
 +
|-
 +
|
 +
|I created a new [https://bugzilla.mozilla.org/show_bug.cgi?id=465592 bug] for allow PGO for Thunderbird on Windows and my patch has made it into the mozilla-core.<br/>
 +
This is the [[PGO_Related_Bugs/Patches | patch]] that landed.
 +
|-
 +
|colspan="2"|'''Add PGO to one module'''
 +
|-
 +
|
 +
|It turns out that by default each module is optimized for size if using PGO.  The problem that arose from this was the profile data was being collect but not used.  The profile data was getting deleted at the beginning of the build cycle.
 +
|-
 +
|colspan="2"|'''Create a new bug for deleted profile data'''
 +
|-
 +
|
 +
|
 +
Problem 1 is the profile data in the dist/bin/components folder is getting deleted. Problem 2 is even if the profile data wasn't getting deleted the pgomerge.py script is not looking for profile data in the dist/bin/components folder anyways.  This means that any component that are profiled are not getting optimized.<br />
 +
Here are the [[PGO_Related_Bugs/Patches#468285|patches]] I'm proposing.<br/>
 +
I created a new [https://bugzilla.mozilla.org/show_bug.cgi?id=468285 bug] for this problem.
 +
|-
 +
|colspan="2"|'''Details of optimizing Thunderbird'''
 +
|-
 +
|
 +
|Using the proposed patches 74 objects were profiled and 69 were optimized.  Here are the [[PGO_Related_Bugs/TB_Details|details]].
 +
|} <!-- end 0.3 release -->
 
|}
 
|}
  
Line 412: Line 487:
  
 
|} <!-- end of table indent -->
 
|} <!-- end of table indent -->
 +
 +
Thunderbird PGO Errors<br />
 +
The lcms module breaks pgo. [[PGO_Related_Bugs/TB_PGO_Errors|Found Here]]
  
 
=== Performance ===
 
=== Performance ===
Line 475: Line 553:
  
 
== <font color="Peru">Project News </font>==
 
== <font color="Peru">Project News </font>==
 +
 +
=== December 2008===
 +
{|
 +
!width="100px"|Date
 +
!width="15px"|
 +
!Comments
 +
|-valign="top"
 +
|style="color:green;font-weight:bold" |Dec 8nd, 2008
 +
|
 +
|I successfully optimized the components but I broke something small.  I [http://dee132.blogspot.com/2008/12/thunderbird-components-getting.html blogged] about it.
 +
|-valign="top"
 +
|style="color:green;font-weight:bold" |Dec 6nd, 2008
 +
|
 +
|I posted a new bug about components not getting optimized for Windows. This is the [https://bugzilla.mozilla.org/show_bug.cgi?id=468285 bug].
 +
|-valign="top"
 +
|style="color:green;font-weight:bold" |Dec 6nd, 2008
 +
|
 +
|After talking with Ted Mielczarek on IRC I have found out why the files I'm profiling are not getting merged into the optimized build. [[PGO_Related_Bugs#Conversations|Here]] is the conversation.
 +
|-valign="top"
 +
|style="color:green;font-weight:bold" |Dec 6nd, 2008
 +
|
 +
|My patch has made into mozilla-central.  Here is the [http://dee132.blogspot.com/2008/12/patch-landed.html blog] posting.
 +
|-valign="top"
 +
|style="color:green;font-weight:bold" |Dec 5nd, 2008
 +
|
 +
|My patch has been reviewed.  It needed some changes but was approved. This is the [https://bugzilla.mozilla.org/show_bug.cgi?id=465592 bug].
 +
|-valign="top"
 +
|style="color:green;font-weight:bold" |Dec 5nd, 2008
 +
|
 +
|Still trying to get the address book optimized.  Discovered that all the modules are getting optimized for size by default.  I [http://dee132.blogspot.com/2008/12/optimizing-thunderbird-address-book_05.html blogged] about it.
 +
|-valign="top"
 +
|style="color:green;font-weight:bold" |Dec 3nd, 2008
 +
|
 +
|Added speed optimizations to the address book module in Thunderbird.  This library gets linked into the mail.dll.
 +
|-valign="top"
 +
|style="color:green;font-weight:bold" |Dec 2nd, 2008
 +
|
 +
|I got a patch up that will disable pgo for the mozlcms module when it is not linked into the xul.dll.
 +
|}
  
 
=== November 2008===
 
=== November 2008===
Line 482: Line 599:
 
!width="15px"|
 
!width="15px"|
 
!Comments
 
!Comments
 +
|-valign="top"
 +
|style="color:green;font-weight:bold" |Nov 24th, 2008
 +
|
 +
|I've been looking at the Thunderbird Makefiles.  I'm trying to understand how the build process works.  I [http://dee132.blogspot.com/2008/11/understanding-thunderbird-build-process.html blogged] about it.
 +
|-valign="top"
 +
|style="color:green;font-weight:bold" |Nov 18th, 2008
 +
|
 +
|I created a new [https://bugzilla.mozilla.org/show_bug.cgi?id=465592 bug] to get input on how to enable PGO for Thunderbird without modifying the Mozilla core files. Blog is [http://dee132.blogspot.com/2008/11/lets-start-converation.html here].
 
|-valign="top"
 
|-valign="top"
 
|style="color:green;font-weight:bold" |Nov 12th, 2008
 
|style="color:green;font-weight:bold" |Nov 12th, 2008

Latest revision as of 22:26, 9 December 2008

Project Name

PGO Related Bugs

Project Description

Firefox crashes or experiences bugs after creating PGO builds on Windows. The focus of this project is to determine what is causing these crashes or bugs and either patch/fix them or find solutions to allow for the optimizations without incurring the problems.

As of Oct 2nd:
Profiling Firefox to increase the number of functions optimized in the PGO build of Firefox.

Project Leader(s)

Project Contributor(s)

Contributors are needed for help on profiling Firefox to improve PGO on Firefox.

What contributions are needed?

  • What should be profiled?
  • A write up on steps to produce profile.
  • Creating scripts to execute profile steps.


Name Contribution
Chris Bishop ( Example ) Script to open and close Firefox.



  • Need some bash scripting help.
  • I'm on IRC as chrisBee
Name Script Contribution
Chinmay Patel Created a small script to search and replace a piece of text in a file.

Project Tasks

Task Details Priority Status Target Link(s) or Notes
Compile Firefox PGO Compile the trunk of Firefox using make -f client.mk profiledbuild. High Done 0.1 R
Reproduce PGO Bugs or Crashes I need to reproduce the bugs or crashes other people were experiencing with their pgo builds. High Done 0.1 R Unable to reproduce. Focusing on profiling Firefox. Blog Post
Compile Firefox PGO Tests Compile Firefox with VS 2005 and Vista SDK and compile with VS 2008 w/o Vista SDK. Low Not Started 0.3 R
Determine 0.1 Release I need to determine what I'll be releasing for the 0.1 Release. High Done Sept 28th, 2008
Scenario Scripts I need a script that will build multiple PGO builds in different directories using different profiling scenarios. High Done Oct 24th, 2008 I need some help from contributors to help me accomplish this. **Update** Thanks to Chinmay and Dave Humphrey I was able to create this script.
Create SQLite Upstream Bug I have determined the pgo bug is actually with SQLite and it needs to be added to the upstream for it to be worked on. High On Going Oct 24th, 2008 Looking into how to actual do this
0.2 Release
Setup Test Environment I need to setup a test environment to duplicate the testing done on Thunderbird. High Done Nov 4th, 2008 I have this VM setup but I am not using yet since my XP OS that is not on a VM is much faster.

Project Details

Conversations

December 6th Ted Mielczarek

I told Ted about how a large number of pgc (profile data) files were in a folder that gets deleted before the pgc files get used to optimize Thunderbird. It turns out that this folder gets clobbered to make sure if some one rebuilt their try after removing a component that it would not still be there. Ted suggest adding a flag that ifndef MOZ_PROFILE_USE then go ahead and clobber it otherwise don't.

That is only one part of it. The pgc files need to be merged into their corresponding pgd (program database) files but the script that does this only looks in one directory. I think I can just tell it to look in both locations. The pgomerge.py does not return errors for files not found.

Ted also said I should create another bug about this - how components can't be PGO'd on Win32.

PGO Details

Details
Thunderbird PGO Details

Thunderbird

Address Book enabled with PGO

Turned on speed optimizations for the Address Book module.

Testing Scripts




Related Bugs

Profiledbuild doesn't work on comm-central - 448515
Figure out why building jemalloc breaks with PGO enabled - 419470
sort out PGO-triggered bugs - 419893
win32 mozilla-central weirdness with PGO - 437002
To enable PGO on Thunderbird a module in the Mozilla core needs to have PGO disabled - 465592 - Resolved
Components cannot be PGO'd on Windows - 468285
js3250!1.pgc remains in dist/bin during a PGO build - 467271

Patches

Here is the patch for bug 465592.
Here is the proposed patches for bug 468285.

Releases

0.1 Release

Reproduce bug(s)
Was able to reproduce bug for sqlite pgo.
Document how to reproduce bug(s).
To enable PGO for sqlite a line must be commented in a make file.

in path/to/src/db/sqlite/src/Makefile.in
add a hash to the line NO_PROFILE_GUIDED_OPTIMIZE = 1
should look like #NO_PROFILE_GUIDED_OPTIMIZE = 1
The build firefox with make -f client.mk profiledbuild
in the .mozconfig file you'll need an extra parameter mk_add_options PROFILE_GEN_SCRIPT="sh script/to/run.sh"
My Script file has the following: 

export NO_EM_RESTART=1 #this means 'do not restart for any reason' during automation we do not want any restarting

mkdir $OBJDIR/_profileprofile 
cd $OBJDIR/_tests/testing/mochitest
python runtests.py --test-path=browser/ --autorun --close-when-done
Hone in on cause of bug(s).
Turns out the bug is in the sqlite3.c file. I can reproduce the error when creating a stand alone sqlite application and enabling PGO on the application.
Running xpcshell and Mochitest against PGO build.
Since firefox right out crashes I am unable to run Mochitests but the xpcshell tests do run until they hit the storage tests.
Create strategy for profiling Firefox to improve performance.
I tried different approaches to profiling for Firefox to improve its performance but I was not able to help it at all. I have concluded that the existing script of opening and closing the browser is more than enough for profiling. The browser is rendered using the same components to render and load web pages this means profiling the loading and rendering of the browser will optimize the rendering and loading of web pages.
Deliverable
I have updated the bug 419893 with how to reproduce the bug, a text file of the errors, a zip file with the source code for my test app to produce pgo errors with sqlite as a stand alone app.

0.2 Release

Setup Testing Environment
Created Windows 2003 Server VM - but I am still using my faster Windows XP OS since it is not a VM. The test environment will be used to verify performance data once testing of PGO begins. Either in 0.3 or beyond.
Compile Thunderbird with PGO
I successfully compiled Thunderbird with PGO. It required modifying a file. I blogged about it here.
Compile Steps
I also have the steps to compile Thunderbird with PGO here.
Contribute to Existing Bug
The bug Profiledbuild doesn't work on comm-central 448515 has been re-opened and I have posted my success results on this bug.
Add PGO to one module
This will be pushed back to 0.3 Release

0.3 Release

Patch to enable PGO for Thunderbird on Windows
I created a new bug for allow PGO for Thunderbird on Windows and my patch has made it into the mozilla-core.

This is the patch that landed.

Add PGO to one module
It turns out that by default each module is optimized for size if using PGO. The problem that arose from this was the profile data was being collect but not used. The profile data was getting deleted at the beginning of the build cycle.
Create a new bug for deleted profile data

Problem 1 is the profile data in the dist/bin/components folder is getting deleted. Problem 2 is even if the profile data wasn't getting deleted the pgomerge.py script is not looking for profile data in the dist/bin/components folder anyways. This means that any component that are profiled are not getting optimized.
Here are the patches I'm proposing.
I created a new bug for this problem.

Details of optimizing Thunderbird
Using the proposed patches 74 objects were profiled and 69 were optimized. Here are the details.

Errors

  • Build Environment:
    • Windows XP (32-bit)
    • Visual Studio 2008
    • Vista SDK



Thunderbird PGO Errors
The lcms module breaks pgo. Found Here

Performance

Last updated: October 11th, 2008

PGO Performance data


Project News

December 2008

Date Comments
Dec 8nd, 2008 I successfully optimized the components but I broke something small. I blogged about it.
Dec 6nd, 2008 I posted a new bug about components not getting optimized for Windows. This is the bug.
Dec 6nd, 2008 After talking with Ted Mielczarek on IRC I have found out why the files I'm profiling are not getting merged into the optimized build. Here is the conversation.
Dec 6nd, 2008 My patch has made into mozilla-central. Here is the blog posting.
Dec 5nd, 2008 My patch has been reviewed. It needed some changes but was approved. This is the bug.
Dec 5nd, 2008 Still trying to get the address book optimized. Discovered that all the modules are getting optimized for size by default. I blogged about it.
Dec 3nd, 2008 Added speed optimizations to the address book module in Thunderbird. This library gets linked into the mail.dll.
Dec 2nd, 2008 I got a patch up that will disable pgo for the mozlcms module when it is not linked into the xul.dll.

November 2008

Date Comments
Nov 24th, 2008 I've been looking at the Thunderbird Makefiles. I'm trying to understand how the build process works. I blogged about it.
Nov 18th, 2008 I created a new bug to get input on how to enable PGO for Thunderbird without modifying the Mozilla core files. Blog is here.
Nov 12th, 2008 I have updated my 0.2 Release.
Nov 12th, 2008 Nomis who is working on enabling PGO for TB on Mac is having different errors than me. He is posting to the same bug as I am.
Nov 12th, 2008 A closed bug for PGO on Thunderbird has been re-opened. I am posting to it with new information and findings.
Nov 7th, 2008 I posted results from using PGO.
Nov 7th, 2008 I successfully compiled Thunderbird with PGO enabled. I had to disable PGO for the mozlcms.dll.
Nov 4th, 2008 Made change to the client.mk file for Thunderbird and compiled it with PGO enable. I received a compile error.
Nov 1st, 2008 Installed Windows 2003 for creating a testing environment.

October 2008

Date Comments
Oct 28th, 2008 Focusing on adding PGO to Thunderbird. Built Thunderbird from source.
Oct 17th, 2008 I have written up my 0.1 Release and have blogged about it.
Oct 16th, 2008 After compiling Firefox with sqlite pgo enabled I was able to reproduce the crash as reported on the bug. I also received errors in xpcshell which I posted to the bug on bugzilla. I also created a test application for sqlite and found it crashed when compiled with PGO as a stand alone application.
Oct 14th, 2008 Through the contribution of a fellow student I was able to create a script to automate multiple PGO Builds.
Oct 11th, 2008 I checked my performance tests again this time including a pgo build with no profiling and Google Maps was 45% on both Minefields. Regardless of the profiling.
Oct 10th, 2008 Updated the contributions section for help with creating bash shell scripts.
Oct 10th, 2008 Made a pgo performance chart. Google Maps was 45% faster in my pgo build after I profiled the site.
Oct 8th, 2008 Still receiving 35 failed tests even when I have no windows open and am not at the computer.
Oct 8th, 2008 Ted mentioned on IRC that the 35 failed tests may of happened because I was moving my mouse during the tests. Which I was. I'm going to try the tests again.
Oct 7th, 2008 Using pgomgr I am able to see what profiling data has been collected.
Oct 4th, 2008 A build of Firefox without PGO resulted in 35 failed Tests in Mochitest. Xpcshell passed just fine. At this time the tree is closed so I cannot get the latest bug free version.
Oct 2nd, 2008 After talking with Ted and Dave Humphrey I am going to focus my project on increasing the profiling of Firefox to increase its speed.

I am also enabling the modules that have PGO disable and running the xpcshell and Mochitests on them to determine if they are still broken or if they are now working.

September 2008

Date Comments
Sept 30th, 2008 Failed to reproduce bugs. I followed Ted's suggestions on my blog but still not experiencing any problems with my pgo build.
Sept 28th, 2008 Ted pointed me in the right direction on how to reproduce the PGO bugs. His post is here.
Sept 26th, 2008 Successfully compiled Firefox with PGO using Visual Studio 2005 compiler. Tests and debug are disabled.
Sept 26th, 2008 On IRC some guys suggested compiling with Visual Studio 2005 instead.
Sept 24th, 2008 Attempted to compile with debug and test disabled. Still Failed.
Sept 22th, 2008 Failed to compile Firefox with PGO on Visual Studio 2008.
Sept 20th, 2008 Successfully compiled one old assignment using PGO in Visual Studio. Results in Project Details.
Sept 19th, 2008 Tried to compile some old assignments using PGO in Visual Studio. Two assignments failed.
Sept 18th, 2008 Created this Project Page.
Sept 17th, 2008 Created First Build of Firefox.
Sept 15th, 2008 Met with Ted Mielczarek and discuss PGO Project.
Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=PGO_Related_Bugs&oldid=21453"