Firefox Performance Testing Lab Fall 2010

Goal

The goal of this lab is twofold. First, to provide students with a real-world experience of working collaboratively in an open community; and second, to work on a cutting-edge, but manageable project within the Mozilla community.

Objective

To conduct A/B performance tests of the Chrome Experiments using nightly builds of both Firefox and Chrome, in order to identify performance bottlenecks in Firefox. Also, to profile and file bugs in order to fix these issues.

Method

• As a group determine a method for dividing the Chrome Experiments so they all get tested
• Install both the Firefox and Chrome nightly builds
• Test each experiment in the two browsers, looking for various issues:
  • Speed - is Firefox as fast as Chrome at rendering the graphics?
  • Smoothness - are the graphics as smooth as in Chrome? Do you notice a lot of pauses, jerkiness, etc.?
  • Responsiveness - does Firefox remain responsive while you run the demo? Is it pegging your CPU(s)?
• Record your findings, as well as browser and computer info ([about:support], [about:]) in the Results Spreadsheet.

Resources

Links

• Chrome Experiments
• Firefox (i.e., Minefield) nightly builds
• Chrome (i.e., Chromium) nightly builds
• Example performance bug

Misc Info

One of the questions that comes up a lot when filing these bugs, especially on Windows, is whether or not you have Direct2D (i.e., D2D) or Direct3D (D3D) enabled. The graphics system in Firefox has 3 states related to hardware acceleration:

1. no hardware acceleration
2. D3D but not D2D
3. All hardware acceleration

You can control this using various preferences, which you can change by going to about:config (type this into your address bar and press enter):

• D3D Layers Preferences
  • layers.accelerate-all
  • layers.accelerate-none
• Font Rendering
  • gfx.font_rendering.directwrite.enabled
• Direct2D
  • gfx.direct2d.force-enabled

Tests: Initial First Round

(humph) Let's refine this a bit more, such that we track:

Name of Experiment | URL of Experiment | Your Name | Date of Test | Hardware Info | Browser Info | Firefox Performance - Speed | Firefox Performance - Smoothness | Firefox Performance - Responsiveness | Notes and other Details |

Test No.	Name	Results
1-10	Carl	Results
11-20	Kevin Lasconia	Results
21-30	Andrew Condinho	Results
31-40	Stephen Bologna	Results
41-50	Kenneth Pangilinan	Results
51-60	Pete Leaning	Results
61-70	Brian Law	Results
71-80	Steven Weerdenburg	Results
81-90	James Evangelista	Results
91-100	Kaitlyn Callow	Results
101-110	Crystal de Nobrega	Results
111-120	Chris DeCairos	Results
1-26 (Linux)	Konstantin Novichikhin	Results

Tests: Second Round

For each experiment you tested as part of the first-round of testing that was not as fast or faster than Chrome, please create an entry in the table below. Include details about what you are seeing, what is failing, etc. Also, if you need to file a bug, include the link to the bug you filed.

Link to Xperf Profiler
MSDN blog detailing how to use xperf results

Test	Tester	Problem	Additional Info
Lorenz 84	Stephen Bologna	In the 32bit version of Minefield on Vista the browser froze for several seconds when the page loaded, and any attempt to interact with the test caused it to freeze again. In the 64bit version of Minefield on Window 7, the page took several minutes to load, and the image was not drawn properly.
Google Sphere	Stephen Bologna	Overall look in Minefield is very sloppy compared to Chromium. When the 2d text gets close to the screen, it becomes less legible. It also runs slower on Minefield.	Runs fine in Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
Cheloniidae Live	Kaitlyn Callow	Not running.	Returning error: Error: console is not defined Source File: http://spencertipping.com/beta/cheloniidae-live-b1/script.js Line: 2
Water Type	Kaitlyn Callow	MUCH SLOWER in Minefield on school computers. Speeds seems compairable to Chrome on my home computer. At school browser significantly slowed down while viewing this experiment.	Not sure which is better, but Chrome looks more blurry or anti-aliased maybe then Mindfield.
Plane Deformations	Kaitlyn Callow	At school Mindfield is slower, 12fps vs Chrome at 20fps. At home running must faster in Mindfield, 50 fps (25 in Chrome).
BallDroppings	Carl D	far more audio clipping and visual chop when at a 'max' ball drop speed when compared to chrome
VP Puzzle	Carl D	far greater delay generating picture puzzle windows	Firefox was much faster and smoother then Chrome when creating the one video puzzle and its windows
Ball Pool	Carl D	Choppy at 50+% ball saturation when moving the window	Chromes breakpoint was far later
ChromeDrones	Carl D	Video and Audio chop when 8+ notes were introduced
Liquid Particles	Steven Weerdenburg	Minefield particle dot redraw very slow, manages approx 10 frames/sec	Chromium has no delay when rendering particle dot movement. Both have difficulty in letter rendering
Browser Pong	Steven Weerdenburg	Minefield calculates the "ball" window height as twice that of Chromium
Realtime Video->ASCII Conversion	Steven Weerdenburg	Seems to have very difficult time at higher "resolutions" (smaller fonts). Scales up in canvas size ok, but becomes unresponsive over prolonged periods (5+ minutes) of use.
Colorscube	Kevin Lasconia	When the cube is rotated in any direction the animation is very choppy. There is also a delay between moving the mouse in one direction and the actual cube moving in that direction.	Chrome was very smooth, and responsive.
Monster	Kevin Lasconia	In Firefox, when more complex objects are generated the spinning animation of the object becomes increasingly more choppy. During transformations the animation would freeze for a few seconds then continue.	Chrome did not experience any of the outlined problems above.
Browser Ball	Kevin Lasconia	There is an issue when multiple windows are spawned in Firefox for this experiment. When moving the ball from the main window to another the ball will get stuck. Even when trying to "throw" the ball to another window it will still get stuck.	In Chrome, the ball can actually be moved to and from new windows.
Gravity	Kevin Lasconia	This experiment does not work in the Mozilla/5.0 (Windows NT 6.0; WOW64; rv:2.0b7pre) Gecko/20100916 Firefox/4.0b7pre nightly build.	It works fine in Chrome. It also works in Firefox 3.6.10.
Animated Harmonograph	Steven Weerdenburg	Image rendering much slower than on Chromium. Drawing of complex patterns on Minefield would sometimes cause centre to "wobble". CPU utilization seems (near) identical on both.	Memory observations: Chromium mallocs a large buffer for entire page initially, while Minefield incrementally mallocs as needed.
Wilderness Downtown	Chris DeCairos	NEW Minefield crashes on my computer at the same point in the song every time! I have had it happen 3 times in a row, my first crash reprot was not sent but two others were. It happens after the first verse.	see my first crash report and second crash report Update: I've reproduced it again got a third crash report will file bug report once I know exactly what component this is crashing in. also, I would test it on Chrome nightly, but their current build still cannot play the video at all. Filed a Bug

Bug Reports

NOTE: All bugs related to what we find should have [chromeexperiments] in the bug's Whiteboard field, so we can track them. The following Bugzilla search will list them all: https://bugzilla.mozilla.org/buglist.cgi?quicksearch=[chromeexperiments]

• Sand Trap bug(dupe)
• Open Sand Trap bug
• Sand Trap graphics bug
• Sand Trap painting perf bug
• Wilderness Downtown Canvas bug
• Wilderness Downtown Crash bug (xull.dll)