View source for Firefox Performance Testing Lab Fall 2010

== Goal ==

The goal of this lab is twofold.  First, to provide students with a real-world experience of working collaboratively in an open community; and second, to work on a cutting-edge, but manageable project within the Mozilla community.

== Objective ==

To conduct A/B performance tests of the Chrome Experiments using nightly builds of both Firefox and Chrome, in order to identify performance bottlenecks in Firefox.  Also, to profile and file bugs in order to fix these issues.

== Method ==

* As a group determine a method for dividing the Chrome Experiments so they all get tested
* Install both the Firefox and Chrome nightly builds
* Test each experiment in the two browsers, looking for various issues:
** Speed - is Firefox as fast as Chrome at rendering the graphics?
** Smoothness - are the graphics as smooth as in Chrome?  Do you notice a lot of pauses, jerkiness, etc.?
** Responsiveness - does Firefox remain responsive while you run the demo?  Is it pegging your CPU(s)?  
* Record your findings, as well as browser and computer info ([about:support], [about:]) in the Results Spreadsheet.

== Resources ==

===Links===

* [http://chromeexperiments.com Chrome Experiments]
* [http://nightly.mozilla.org/ Firefox (i.e., Minefield) nightly builds]
* [http://build.chromium.org/buildbot/snapshots/ Chrome (i.e., Chromium) nightly builds]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=594920 Example performance bug]

===Misc Info===

One of the questions that comes up a lot when filing these bugs, especially on Windows, is whether or not you have Direct2D (i.e., D2D) or Direct3D (D3D) enabled.  The graphics system in Firefox has 3 states related to hardware acceleration:

# no hardware acceleration
# D3D but not D2D
# All hardware acceleration

You can control this using various preferences, which you can change by going to '''about:config''' (type this into your address bar and press enter):

* D3D Layers Preferences
** layers.accelerate-all
** layers.accelerate-none
* Font Rendering
** gfx.font_rendering.directwrite.enabled
* Direct2D
** gfx.direct2d.force-enabled

You can also use built-in Firefox options to toggle hardware acceleration between the "None" and "All" states by (un)checking “Use hardware acceleration when available” in the Advanced section of the Preferences/Options dialog. Alternately, you can run Firefox in safe mode to disable hardware acceleration [http://blog.mozilla.com/joe/2010/09/15/testing-hardware-acceleration/]. When filing a bug related to hardware acceleration, please include the Graphics card information from '''about:support''' in your browser.

== Tests: Initial First Round ==

(humph) Let's refine this a bit more, such that we track:

Name of Experiment | URL of Experiment | Your Name | Date of Test | Hardware Info | Browser Info | Firefox Performance - Speed | Firefox Performance - Smoothness | Firefox Performance - Responsiveness | Notes and other Details |

{| border="1" cellpadding="5"
! Test No. !! Name <br /> !! Results
|-
| 1-10 || [[User:cwdesautels|Carl]] || [[Firefox_Performance_Testing_Lab_Fall_2010_cwdesautels | Results]]  
|-
| 11-20 || [[User:kclascon|Kevin Lasconia]] || [[Firefox_Performance_Testing_Lab_Fall_2010_kclascon | Results]] 
|-
| 21-30 || [[User:ajcondinho | Andrew Condinho]] || [[Firefox_Performance_Testing_Lab_Fall_2010_ajcondinho | Results]] 
|-
| 31-40 || [[User:sbologna|Stephen Bologna]] || [[Firefox_Performance_Testing_Lab_Fall_2010_sbologna | Results]]
|-
| 41-50 || [[User:kpangilinan|Kenneth Pangilinan]] || [[Firefox_Performance_Testing_Lab_Fall_2010_kpangilinan | Results]]  
|-
| 51-60 || [[User:peleaning|Pete Leaning]] || [[Firefox_Performance_Testing_Lab_Fall_2010_peleaning | Results]] 
|-
| 61-70 || [[User:blaw1|Brian Law]] ||  [[Firefox_Performance_Testing_Lab_Fall_2010_blaw1 | Results]] 
|-
| 71-80 || [[User:sweerdenburg|Steven Weerdenburg]] || [[Firefox_Performance_Testing_Lab_Fall_2010_sweerdenburg | Results]]  
|-
| 81-90 || [[User:jevangel|James Evangelista]] ||  [[Firefox_Performance_Testing_Lab_Fall_2010_jevangel | Results]] 
|-
| 91-100 || [[User:dacallow|Kaitlyn Callow]] || [[Firefox_Performance_Testing_Lab_Fall_2010_dacallow| Results]]  
|-
| 101-110 || [[User:cldenobrega|Crystal de Nobrega]] || [[Firefox_Performance_Testing_Lab_Fall_2010_cldenobrega | Results]]
|-
| 111-120 || [[User:cadecairos|Chris DeCairos]] || [[Firefox_Performance_Testing_Lab_Fall_2010_cadecairos | Results]] 
|-
| 1-26 (Linux) || [[User:knovichikhi|Konstantin Novichikhin]] || [[Firefox_Performance_Testing_Lab_Fall_2010_knov | Results]] 
|}

== Tests: Second Round ==

For each experiment you tested as part of the first-round of testing that was not as fast or faster than Chrome, please create an entry in the table below.  Include details about what you are seeing, what is failing, etc.  Also, if you need to file a bug, include the link to the bug you filed.

[https://developer.mozilla.org/En/Profiling_with_Xperf Link to Xperf Profiler]<br />
[http://blogs.msdn.com/b/pigscanfly/archive/2009/08/06/stack-walking-in-xperf.aspx MSDN blog detailing how to use xperf results]

{| border="1" cellpadding="5"
! Test !! Tester !! Problem !! Additional Info <br />
|-
| [http://www.chromeexperiments.com/detail/lorenz-84/ Lorenz 84]
|| [[User:sbologna|Stephen Bologna]]
|| In the 32bit version of Minefield on Vista the browser froze for several seconds when the page loaded, and any attempt to interact with the test caused it to freeze again.  In the 64bit version of Minefield on Window 7, the page took several minutes to load, and the image was not drawn properly.
||
|-

| [http://www.chromeexperiments.com/detail/google-sphere/ Google Sphere]
|| [[User:sbologna|Stephen Bologna]]
|| Overall look in Minefield is very sloppy compared to Chromium.  When the 2d text gets close to the screen, it becomes less legible.  It also runs slower on Minefield.
|| Runs fine in Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8
|-

| [http://www.chromeexperiments.com/detail/cheloniidae-live/ Cheloniidae Live] 
|| [[User:dacallow|Kaitlyn Callow]] 
|| Not running. 
|| Returning error:
Error: console is not defined
Source File: http://spencertipping.com/beta/cheloniidae-live-b1/script.js
Line: 2
|-




| [http://www.chromeexperiments.com/detail/water-type/ Water Type] 
|| [[User:dacallow|Kaitlyn Callow]] 
|| MUCH SLOWER in Minefield on school computers. Speeds seems compairable to Chrome on my home computer.  At school browser significantly slowed down while viewing this experiment.
|| Not sure which is better, but Chrome looks more blurry or anti-aliased maybe then Mindfield.
|-


| [http://www.chromeexperiments.com/detail/plane-deformations/ Plane Deformations] 
|| [[User:dacallow|Kaitlyn Callow]] 
|| At school Mindfield is slower, 12fps vs Chrome at 20fps.  At home running must faster in Mindfield, 50 fps (25 in Chrome).
||
|-


| [http://www.chromeexperiments.com/detail/balldroppings/ BallDroppings]
|| [[User:cwdesautels|Carl D]]
|| far more audio clipping and visual chop when at a 'max' ball drop speed when compared to chrome
||
|-

| [http://www.chromeexperiments.com/detail/video-picture-puzzle/ VP Puzzle]
|| [[User:cwdesautels|Carl D]]
|| far greater delay generating picture puzzle windows
|| Firefox was much faster and smoother then Chrome when creating the one video puzzle and its windows
|-

| [http://www.chromeexperiments.com/detail/liquid-particles/ Liquid Particles]
|| [[User:Sweerdenburg |Steven Weerdenburg]]
|| Minefield particle dot redraw very slow, manages approx 10 frames/sec. Chromium has no delay when rendering particle dot movement. Both have difficulty in letter rendering
|| Investigation yields a [https://bugzilla.mozilla.org/show_bug.cgi?id=564643 duplicate]
|-

| [http://www.chromeexperiments.com/detail/animated-harmonograph/ Animated Harmonograph]
|| [[User:Sweerdenburg |Steven Weerdenburg]]
|| Image rendering much slower than on Chromium. Drawing of complex patterns on Minefield would sometimes cause centre to "wobble". CPU utilization seems (near) identical on both.
|| Hardware acceleration makes no difference
|-

| [http://www.chromeexperiments.com/detail/browser-pong/ Browser Pong]
|| [[User:Sweerdenburg |Steven Weerdenburg]]
|| Minefield calculates the "ball" window height as twice that of Chromium
|| 
|-

| [http://www.chromeexperiments.com/detail/realtime-video-ascii-conversion/ Realtime Video->ASCII Conversion]
|| [[User:Sweerdenburg |Steven Weerdenburg]]
|| Seems to have very difficult time at higher "resolutions" (smaller fonts). Scales up in canvas size ok, but becomes unresponsive over prolonged periods (5+ minutes) of use. Handles larger canvas with same resolution better than chrome at default settings.
|| 
|-

| [http://www.chromeexperiments.com/detail/colorscube/ Colorscube]
|| [[User:Kclascon |Kevin Lasconia]]
|| When the cube is rotated in any direction the animation is very choppy. There is also a delay between moving the mouse in one direction and the actual cube moving in that direction.
|| Chrome was very smooth, and responsive. Filed a bug [https://bugzilla.mozilla.org/show_bug.cgi?id=599351 here].
|-

| [http://www.chromeexperiments.com/detail/monster/ Monster]
|| [[User:Kclascon |Kevin Lasconia]]
|| In Firefox, when more complex objects are generated the spinning animation of the object becomes increasingly more choppy. During transformations the animation would freeze for a few seconds then continue.
|| Found a bug about the same experiment [https://bugzilla.mozilla.org/show_bug.cgi?id=503470 here].
|-

| [http://www.chromeexperiments.com/detail/browser-ball/ Browser Ball]
|| [[User:Kclascon |Kevin Lasconia]]
|| There is an issue when multiple windows are spawned in Firefox for this experiment. When moving the ball from the main window to another the ball will get stuck. Even when trying to "throw" the ball to another window it will still get stuck.
|| In Chrome, the ball can actually be moved to and from new windows.
|-

| [http://www.chromeexperiments.com/detail/gravity/ Gravity]
|| [[User:Kclascon |Kevin Lasconia]]
|| This experiment does not work in the  Firefox/4.0b7pre nightly build.
|| It works fine in Chrome. It also works in Firefox 3.6.10, however the objects cannot be dragged around like in Chrome. A bug filed [https://bugzilla.mozilla.org/show_bug.cgi?id=570922 here] outlines the dragging issues. Did some regression testing to determine the builds where the experiment worked and stopped working. Added some additional info to this bug [https://bugzilla.mozilla.org/show_bug.cgi?id=595541 here].
|-


| [http://www.thewildernessdowntown.com Wilderness Downtown]
|| [[User:cadecairos |Chris DeCairos]]
|| '''NEW''' Minefield crashes on my computer at the same point in the song every time! I have had it happen 3 times in a row, my first crash reprot was not sent but two others were. It happens after the first verse.
|| see my first [http://crash-stats.mozilla.com/report/index/bp-e5ff5685-4816-4787-b8f0-25de02100921 crash report] and second [http://crash-stats.mozilla.com/report/index/bp-4c74c6af-2bd4-4e52-b8ac-65a402100921 crash report] '''Update''': I've reproduced it again got a [http://crash-stats.mozilla.com/report/index/49bba790-767f-4662-b83f-1ad092100921 third crash report] will file bug report once I know exactly what component this is crashing in.
also, I would test it on Chrome nightly, but their current build still cannot play the video at all.
Filed a [https://bugzilla.mozilla.org/show_bug.cgi?id=598498 Bug]
|-


| [http://www.chromeexperiments.com/detail/depth-of-field/ Depth of Field]
|| [[User:kpangilinan |Kenneth Pangilinan]]
||  Minefield freezes up PC for a few seconds before running. When attempting to move the window around, PC freezes up again.
|| Experiment works fine in Chromium, moving the window around is fine as well.
|-

| [http://www.chromeexperiments.com/detail/3d-javascript-with-sandy-hx/ 3D JS w/ Sandy DX]
|| [[User:kpangilinan |Kenneth Pangilinan]]
|| Experiment did not load in Minefield or Chromium.
||
|-

| [http://www.chromeexperiments.com/detail/javascript-voxel-spacing/ JS Voxel Spacing]
|| [[User:kpangilinan |Kenneth Pangilinan]]
|| In Minefield it runs at 2-4FPS, in Chromium it runs at 22-24FPS, about 10 times faster!
||
|-

| [http://www.chromeexperiments.com/detail/gear/ Gear]
|| [[User:blaw1 |Brian Law]]
|| This experiment is very choppy in Minefield, whereas Chromium runs smoothly. In Minefield you can see arrows in the boxes which shouldn't be visible.
||
|-

| [http://www.chromeexperiments.com/detail/waterfall/ Waterfall]
|| [[User:blaw1 |Brian Law]]
|| Minefield quickly begins to lag as more balls enter the screen. Chromium will run smoothly for much longer.
||
|-

| [http://www.chromeexperiments.com/detail/pocket-full-of-canvas/ Pocket Full of Canvas]
|| [[User:peleaning |Pete Leaning]]
|| Minefield draws black triangles in the following effects in 'pocket full of canvas':
Elipse, Colorrects, Mario, Colormunch, imagewaves, fire, wave(de)form and imagemagnifier. These artifacts are not present in Chromium
||The two functions that seem to be responsible for this are renderTriangle() and drawImage
|-

| [http://www.chromeexperiments.com/detail/kaleidscope/ Kaleidscope]
|| [[User:ajcondinho |Andrew Condinho]]
|| Physical window becomes jerky and laggy whenever you re-size the window.
|| Tested this out on my desktop and lag disappeared, looks like it might be an issue with lesser hardware
|-

| [http://www.chromeexperiments.com/detail/physicsketch/ physicSketch]
|| [[User:ajcondinho |Andrew Condinho]]
|| Objects don't draw, and after a few failed attempts experiment stops responding to any attempts to draw.
|| Chrome has no problems running this experiment.
|-

|}

==Bug Reports==

'''NOTE:''' All bugs related to what we find should have '''[chromeexperiments]''' in the bug's Whiteboard field, so we can track them.  The following Bugzilla search will list them all: <nowiki>https://bugzilla.mozilla.org/buglist.cgi?quicksearch=[chromeexperiments]</nowiki>

* [https://bugzilla.mozilla.org/show_bug.cgi?id=596677 Sand Trap bug(dupe)]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=595963 Open Sand Trap bug]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=596707 Sand Trap graphics bug]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=596746 Sand Trap painting perf bug]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=597186 Wilderness Downtown Canvas bug]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=598498 Wilderness Downtown Crash bug (xull.dll)]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=598831 Liquid Particles perf bug (dupe)]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=598834 Animated Harmonograph perf bug]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=598361 Pocket Full of Canvas triangle bug]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=599210 Ball Dropping bug]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=599236 Video->ASCII Conversion perf bug]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=599318 Javascript Voxel Spacing bug]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=599350 Gear bug]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=599351 Colorscube bug]
* [https://bugzilla.mozilla.org/show_bug.cgi?id=595541 Google Gravity bug (added additional info)]