New Firefox Performance Testing Documentation

Introduction

This documentation is still in progress.

• This document assumes that you have the following:
  • msvcp71.dll (NOTE: If you have Visual Studio 2003 installed, you've already got it.)
    • Microsoft® C Runtime Library, v. 7.10.3077.0
    • You need a copy of this file to run Python with all the libraries.
• Put it in your C: directory if you have to download msvcp71.dll

• This documentation is the revised version of Annie Sullivan's Readme.txt for the Firefox Performance Testing.

• Unfortunately, the Firefox Performance Testing is only catered to Windows users at this time.

Quick start

3 easy steps to run Firefox Performance Testing:

• Install Cygwin
• Save the script
• Run the script

Cygwin Setup

• Cygwin
  • Download cygwin.
  • Install cygwin with the DOS-style line endings.
  • At minimum, you are required to have the following additional tools:
    • unzip (Archive category)
    • wget (Web category)

unzip (Archive category)


wget (Web category)

Note: For some reason, we can't get output from Firefox unless it's wrapped in a Cygwin sh shell. So we use cygwin for that, and to sync between Firefox runs.

Automated Configuration

Save the script

• Save this script at a desired location
  • Script: Script to configure Firefox Performance Testing Framework
• Edit the script to your own choosing. You may provide a list of extensions to install and test in the config.yaml section.
• Note: This script will run the performance testing. If don't wish for that to happen, simply comment out the last line of the script (to comment out, use #):

#./run_tests.py config.yaml;

• Refer to Run Firefox Performance Tests if you wish to run the performance tests later.

Run the script

• In cygwin, browse to the location where you saved your script.
• Before you run the script, refer to What happens when you run Firefox Performance Tests
• From cygwin, run the script.

For example:

$./perfconfig.sh

What the script does

• Creates directories needed for the performance testing
  • Directory that holds the files for the performance testing
  • Reports directory to hold the results generated
• Downloads and installs pre-requisites
• Write to config.yaml

Config File (config.yaml)

Config File Info

When you run the perfconfig.sh script, a config.yaml file will be written to C:\mozilla\testing\performance\win32\ by default.

Your config.yaml should consist of these items.

• filename
• title
• (Test Profile name) - any name
  • firefox
  • preferences
  • extensions

Refer to sample.config:

     # Filename will be appended to the timestamp in the report filename.
     # Use letters and underscores only
     filename: yourfilename
     # The title of the report
     title: 1 GHz Celeron

     # Name of profile to test
     Test profile 1:
       # Path to Firefox to test
       firefox: C:\Program Files\Mozilla Firefox 2 Beta 1\firefox.exe

       # Preferences to set in the test (use "preferences: {}" for no prefs)
       preferences:
         javascript.options.showInConsole : true
         xpinstall.enabled : true

       # Extensions to install in test (use "extensions: {}" for none)
       extensions:
         # Make sure that the path is directed to the desired location of the extension
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi

Config File Instructions

1. Verify that the firefox path is pointing to the right Firefox instance on your system.

• You will hit a zero division error and some other errors if you don't point it to the right path

2. preferences:

• If you don't have preferences, change to:

  preferences: {}

  (make sure there is a space between ":" and "{" or else it will cause a syntax error)
• If you do have preferences, refer to sample.config

3. extensions:

• If you don't have extensions to install, change to:

  extensions: {}

  (make sure there is a space between ":" and "{" or else it will cause a syntax error)
• If you do have extensions to install, refer to this sample:

       extensions:
         # Make sure that the path is directed to the desired location of the extension
         # Need quotes around guid because of curly braces
         "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi
         foo@sample.com : c:\path\to\other\unzipped\xpi 

• Make sure that the path to the extension is right

4. Change filename and title to whatever you desire

Run Firefox Performance Tests

• By default, the run_tests.py file that you need to execute to run the Firefox Performance Testing is located at c:\mozilla\testing\performance\win32\
• You can run run_tests.py either from the DOS prompt or Cygwin and you may add as many config file arguments as you need to for your Performance Testing:

DOS

$run_tests.py config.yaml <anotherConfig.yaml>

Cygwin

./run_tests.py config.yaml <anotherConfig.yaml>

Things you can configure manually

Configuring the paths (paths.py)

• You may set up the appropriate paths for the framework manually.
• The current paths are set by default
• After you run perfconfig.sh, the paths.py file is located in C:\mozilla\testing\performance\win32\ by default
• The following are the constants in paths.py:
  • CYGWIN
  • SYNC
  • BASE_PROFILE_DIR
  • REPORTS_DIR
  • INIT_URL
  • TS_URL
  • TP_URL

CYGWIN

By default: CYGWIN = r'c:\cygwin\bin\bash.exe -c'

• The path to cygwin bash to get output from dump() in Firefox
• If you didn't specify a different root directory for the Cygwin to be installed, it will be installed in C:\ by default

SYNC

By default: SYNC = r'c:\cygwin\bin\sync'

• Runs sync between Ts runs
• If you didn't specify a different root directory for the Cygwin to be installed, it is in C:\

BASE_PROFILE_DIR

By default: BASE_PROFILE_DIR = r'C:\mozilla\testing\performance\win32\base_profile'

• IMPORTANT: Check that there is content in the base_profile directory

• (Optional) For the page load test to work, modify the hostperm.1 file
  • Set to allow urls with scheme:file to open in new windows
  • Set the preference to open new windows in a tab should be off.

REPORTS_DIR

By default: REPORTS_DIR = r'C:\mozilla\testing\performance\win32\extension_perf_reports'

• The perfconfig.sh script creates a extension_perf_reports dir in C:\mozilla\testing\performance\win32\

INIT_URL

By default: INIT_URL = 'file:///c:/mozilla/testing/performance/win32/initialize.html'

• The path to the file url to load when initializing a new profile
• Note: Don't change the local file url to a file path

TS_URL

By default: TS_URL = 'file:///c:/mozilla/testing/performance/win32/startup_test/startup_test.html?begin='

• The path to the file url to load for startup test (Ts)
• Note: Don't change the local file url to a file path

TP_URL

By default: TP_URL = 'file:///c:/mozilla/testing/performance/win32/page_load_test/cycler.html'

• The path to the file url to load for page load test (Tp)
• Note: Don't change the local file url to a file path

Configuring Config file

• If you run the perfconfig.sh script, a config.yaml file will be written to C:\mozilla\testing\performance\win32\ config.yaml or refer to sample.config
• You may have as many config files as you need to use for your Performance Testing
• The framework takes as many arguments as you provide when you run run_tests.py
  • Refer to Run Firefox Performance Tests
• For more information, refer to Config File (config.yaml)

What happens when you run Firefox Performance Tests

• All opened browser windows will be closed automatically!
• You might run into this message on your browser: This page should close Firefox. If it does not, please make sure that the dom.allow_scripts_to_close_windows preference is set to true in about:config
  • Type auto:config in the address bar and double click on dom.allow_scripts_to_close_windows to change it to true
• If you see processing... Performance Testing is starting, performance testing is running..
• You can check the progress by the progress bar