PGO Related Bugs

Project Name

PGO Related Bugs

Project Description

Firefox crashes or experiences bugs after creating PGO builds on Windows. The focus of this project is to determine what is causing these crashes or bugs and either patch/fix them or find solutions to allow for the optimizations without incurring the problems.

As of Oct 2nd:
Profiling Firefox to increase the number of functions optimized in the PGO build of Firefox.

Project Leader(s)

Chris Bishop

Project Contributor(s)

Contributors are needed for help on profiling Firefox to improve PGO on Firefox.

What contributions are needed?

What should be profiled?
A write up on steps to produce profile.
Creating scripts to execute profile steps.

Name		Contribution
Chris Bishop ( Example )		Script to open and close Firefox.

Need some bash scripting help.
I'm on IRC as chrisBee

Name		Script Contribution
Chinmay Patel		Created a small script to search and replace a piece of text in a file.

Project Tasks

Task	Details	Priority	Status	Target	Link(s) or Notes
Compile Firefox PGO	Compile the trunk of Firefox using `make -f client.mk profiledbuild`.	High	Done	0.1 R
Reproduce PGO Bugs or Crashes	I need to reproduce the bugs or crashes other people were experiencing with their pgo builds.	High	Done	0.1 R	Unable to reproduce. Focusing on profiling Firefox. Blog Post
Compile Firefox PGO Tests	Compile Firefox with VS 2005 and Vista SDK and compile with VS 2008 w/o Vista SDK.	Low	Not Started	0.3 R
Determine 0.1 Release	I need to determine what I'll be releasing for the 0.1 Release.	High	Done	Sept 28th, 2008
Scenario Scripts	I need a script that will build multiple PGO builds in different directories using different profiling scenarios.	High	Done	Oct 24th, 2008	I need some help from contributors to help me accomplish this. Update Thanks to Chinmay and Dave Humphrey I was able to create this script.

Project Details

PGO Details

Details

Testing Scripts

The scripts I am using for running tests against my builds ( Oct 2nd, 2008 )

#!/bin/sh
####### the .mozconfig file to use for this build
export MOZCONFIG=/f/mozilla/.mozconfigPGO
####### the subdirectory trunk holds my src files
cd trunk
####### time the profile build and output stdout and stderr to files that are timestamped
time make -f client.mk profiledbuild 1> /i/mozilla/pgo-output-`date +%s`.txt 2> /i/mozilla/pgo-errors-`date +%s`.txt
####### after the build is completed done run the xpcshell tests and output stdout and stderr to file that are timestamped
make check 1> /i/mozilla/pgo-check-output-`date +%s`.txt 2> /i/mozilla/pgo-check-errors-`date +%s`.txt
####### change to the objdir directory then to the mochitest directory
cd /i/mozilla/ff-pgo/_tests/testing/mochitest
####### run the mochitest, run test right away, close the browser when done, log all output to a timestamped file
python runtests.py --autorun --close-when-done --log-file=/i/mozilla/pgo-mochitest-`date +%s`.log --file-level=DEBUG

Here is the .mozconfig file ( Oct 2nd, 2008 )

. $topsrcdir/browser/config/mozconfig
mk_add_options MOZ_OBJDIR=/i/mozilla/ff-pgo
mk_add_options MOZ_MAKE_FLAGS=-j5
####### this is the script to run after the first build to gather profile data
mk_add_options PROFILE_GEN_SCRIPT="sh /f/mozilla/pgo.sh"
ac_add_options --disable-vista-sdk-requirements
####### this has to be enabled to perform the mochitests
ac_add_options --enable-tests
ac_add_options --disable-debug
ac_add_options --disable-accessibility

Here is the script the profile build runs to gather profile data ( Oct 2nd, 2008 )

#!/bin/sh
export NO_EM_RESTART=1
mkdir $OBJDIR/_profileprofile
cd /i/mozilla/ff-pgo/_tests/testing/mochitest
python runtests.py --autorun --close-when-done --log-file=/i/mozilla/pgo-profile-mochitest-`date +%s`.log --file-level=DEBUG

Related Bugs

	Figure out why building jemalloc breaks with PGO enabled - 419470
	sort out PGO-triggered bugs - 419893
	win32 mozilla-central weirdness with PGO - 437002

Releases

0.1 Release

Reproduce bug(s)

Was able to reproduce bug for sqlite pgo.

Document how to reproduce bug(s).

To enable PGO for sqlite a line must be commented in a make file.

in path/to/src/db/sqlite/src/Makefile.in
add a hash to the line NO_PROFILE_GUIDED_OPTIMIZE = 1
should look like #NO_PROFILE_GUIDED_OPTIMIZE = 1
The build firefox with make -f client.mk profiledbuild
in the .mozconfig file you'll need an extra parameter mk_add_options PROFILE_GEN_SCRIPT="sh script/to/run.sh"
My Script file has the following:

export NO_EM_RESTART=1 #this means 'do not restart for any reason' during automation we do not want any restarting

mkdir $OBJDIR/_profileprofile 
cd $OBJDIR/_tests/testing/mochitest
python runtests.py --test-path=browser/ --autorun --close-when-done

Hone in on cause of bug(s).

Turns out the bug is in the sqlite3.c file. I can reproduce the error when creating a stand alone sqlite application and enabling PGO on the application.

Running xpcshell and Mochitest against PGO build.

Since firefox right out crashes I am unable to run Mochitests but the xpcshell tests do run until they hit the storage tests.

SQLite PGO Errors in xpcshell ( Oct 16th, 2008 )

TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_bug-365166.js | test failed, see log
../../_tests/xpcshell-simple/test_storage/unit/test_bug-365166.js.log:
>>>>>>>
*** Storage Tests: Trying to close!

<<<<<<<
TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_bug-393952.js | test failed, see log
../../_tests/xpcshell-simple/test_storage/unit/test_bug-393952.js.log:
>>>>>>>
*** Storage Tests: Trying to close!

<<<<<<<
TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_bug-429521.js | test failed, see log
../../_tests/xpcshell-simple/test_storage/unit/test_bug-429521.js.log:
>>>>>>>
*** Storage Tests: Trying to close!

<<<<<<<
TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_bug-444233.js | test failed, see log
../../_tests/xpcshell-simple/test_storage/unit/test_bug-444233.js.log:
>>>>>>>
*** Storage Tests: Trying to close!

<<<<<<<
TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_connection_executeAsync.js | test failed, see log
../../_tests/xpcshell-simple/test_storage/unit/test_connection_executeAsync.js.log:
>>>>>>>
*** Storage Tests: Trying to close!

<<<<<<<
TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_like.js | test failed, see log
../../_tests/xpcshell-simple/test_storage/unit/test_like.js.log:
>>>>>>>
*** Storage Tests: Trying to close!

<<<<<<<
TEST-UNEXPECTED-FAIL | ../../_tests/xpcshell-simple/test_storage/unit/test_like_escape.js | test failed, see log
../../_tests/xpcshell-simple/test_storage/unit/test_like_escape.js.log:
>>>>>>>
*** Storage Tests: Trying to close!

<<<<<<<
make[2]: *** [check] Interrupt
make[1]: *** [check] Interrupt
make: *** [check] Interrupt

Create strategy for profiling Firefox to improve performance.

I tried different approaches to profiling for Firefox to improve its performance but I was not able to help it at all. I have concluded that the existing script of opening and closing the browser is more than enough for profiling. The browser is rendered using the same components to render and load web pages this means profiling the loading and rendering of the browser will optimize the rendering and loading of web pages.

Deliverable

I have updated the bug 419893 with how to reproduce the bug, a text file of the errors, a zip file with the source code for my test app to produce pgo errors with sqlite as a stand alone app.

0.2 Release

To be Announced...

0.3 Release

To be Announced...

Errors

Build Environment:
- Windows XP (32-bit)
- Visual Studio 2008
- Vista SDK

Firefox Compilation Errors ( Sept 26th, 2008 )

e:\mozilla\trunk\modules\lcms\src\cmscgats.c(875) : fatal error C1001: An internal error has occurred in the compiler.
(compiler file 'f:\dd\vctools\compiler\utc\src\p2\main.c[0x511953E5:0x00000008]', line 182)
 To work around this problem, try simplifying or changing the program near the locations listed above.
Please choose the Technical Support command on the Visual C++ 
 Help menu, or open the Technical Support help file for more information

LINK : fatal error LNK1000: Internal error during IMAGE::BuildImage

  Version 9.00.21022.08

  ExceptionCode            = C0000005
  ExceptionFlags           = 00000000
  ExceptionAddress         = 511953E5 (51030000) "e:\Program Files\Microsoft Visual Studio 9.0\VC\BIN\c2.dll"
  NumberParameters         = 00000002
  ExceptionInformation[ 0] = 00000000
  ExceptionInformation[ 1] = 00000008

CONTEXT:
  Eax    = 00000000  Esp    = 0012ED18
  Ebx    = 02D5D215  Ebp    = 0012ED2C
  Ecx    = 02D96601  Esi    = 02D5D280
  Edx    = 02DAAE12  Edi    = 00000305
  Eip    = 511953E5  EFlags = 00010246
  SegCs  = 0000001B  SegDs  = 00000023
  SegSs  = 00000023  SegEs  = 00000023
  SegFs  = 0000003B  SegGs  = 00000000
  Dr0    = 00000000  Dr3    = 00000000
  Dr1    = 00000000  Dr6    = 00000000
  Dr2    = 00000000  Dr7    = 00000000

Firefox Compilation Errors ( Sept 22th, 2008 )

make[6]: *** [mozlcms.dll] Error 232
make[6]: *** Deleting file `mozlcms.dll'
make[5]: *** [libs] Error 2
make[4]: *** [libs_tier_external] Error 2
make[3]: *** [tier_external] Error 2
make[2]: *** [default] Error 2
make[1]: *** [build] Error 2
make: *** [profiledbuild] Error 2

Performance

Last updated: October 11th, 2008

PGO Performance data

PGO Build Results ( Sept 20th, 2008 )

The program loads 15,000 random records from a text file into 4 tables each using a different method for storage and searching.

Simple Table: Uses an array and a linear search.
Chain Table: Uses an array of link lists. Search is done with a Hash Key.
Hash Table: Uses an array and searches with linear probing. Also uses a Hash Key.
Tree Table: Uses a binary search tree.

After the data is loaded into a table, 30 different tests are run against it. The same 30 tests for each table. Here are the Results.

	Non-PGO	PGO	Improved By
Simple Table:	28 Secs	4 Secs	7x
Chain Table:	0.109 Secs	0.047 Secs	2.3x
Hash Table:	0.141 Secs	0.047 Secs	2.3x
Tree Table:	0.287 Secs	0.078 Secs	3.6x

Project News

October 2008

Date		Comments
Oct 17th, 2008		I have written up my 0.1 Release and have blogged about it.
Oct 16th, 2008		After compiling Firefox with sqlite pgo enabled I was able to reproduce the crash as reported on the bug. I also received errors in xpcshell which I posted to the bug on bugzilla. I also created a test application for sqlite and found it crashed when compiled with PGO as a stand alone application.
Oct 14th, 2008		Through the contribution of a fellow student I was able to create a script to automate multiple PGO Builds.
Oct 11th, 2008		I checked my performance tests again this time including a pgo build with no profiling and Google Maps was 45% on both Minefields. Regardless of the profiling.
Oct 10th, 2008		Updated the contributions section for help with creating bash shell scripts.
Oct 10th, 2008		Made a pgo performance chart. Google Maps was 45% faster in my pgo build after I profiled the site.
Oct 8th, 2008		Still receiving 35 failed tests even when I have no windows open and am not at the computer.
Oct 8th, 2008		Ted mentioned on IRC that the 35 failed tests may of happened because I was moving my mouse during the tests. Which I was. I'm going to try the tests again.
Oct 7th, 2008		Using pgomgr I am able to see what profiling data has been collected.
Oct 4th, 2008		A build of Firefox without PGO resulted in 35 failed Tests in Mochitest. Xpcshell passed just fine. At this time the tree is closed so I cannot get the latest bug free version.
Oct 2nd, 2008		After talking with Ted and Dave Humphrey I am going to focus my project on increasing the profiling of Firefox to increase its speed. I am also enabling the modules that have PGO disable and running the xpcshell and Mochitests on them to determine if they are still broken or if they are now working.

September 2008

Date		Comments
Sept 30th, 2008		Failed to reproduce bugs. I followed Ted's suggestions on my blog but still not experiencing any problems with my pgo build.
Sept 28th, 2008		Ted pointed me in the right direction on how to reproduce the PGO bugs. His post is here.
Sept 26th, 2008		Successfully compiled Firefox with PGO using Visual Studio 2005 compiler. Tests and debug are disabled.
Sept 26th, 2008		On IRC some guys suggested compiling with Visual Studio 2005 instead.
Sept 24th, 2008		Attempted to compile with debug and test disabled. Still Failed.
Sept 22th, 2008		Failed to compile Firefox with PGO on Visual Studio 2008.
Sept 20th, 2008		Successfully compiled one old assignment using PGO in Visual Studio. Results in Project Details.
Sept 19th, 2008		Tried to compile some old assignments using PGO in Visual Studio. Two assignments failed.
Sept 18th, 2008		Created this Project Page.
Sept 17th, 2008		Created First Build of Firefox.
Sept 15th, 2008		Met with Ted Mielczarek and discuss PGO Project.

CDOT Wiki β

PGO Related Bugs

Project Name

Project Description

Project Leader(s)

Project Contributor(s)

Project Tasks

Project Details

PGO Details

Testing Scripts

Related Bugs

Releases

0.1 Release

0.2 Release

0.3 Release

Errors

Performance

PGO Performance data

Project News

October 2008

September 2008

CDOT Wiki ^β