Difference between revisions of "Firefox Performance Testing : A Python framework for Windows Documentation"
(→Framework Configuration) |
|||
Line 20: | Line 20: | ||
=Framework Configuration= | =Framework Configuration= | ||
+ | ==Configure the environment== | ||
#Make sure the [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows_Documentation#Framework_Prerequisites | prerequisites]] are installed. | #Make sure the [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows_Documentation#Framework_Prerequisites | prerequisites]] are installed. | ||
− | #Make a directory for the framework: | + | #Make a directory for the framework:<pre>$mkdir C:\mozilla</pre> |
− | <pre>$mkdir C:\mozilla</pre> | ||
#Download entire framework (directories and all subdirectories) and put it in '''C:\mozilla''' | #Download entire framework (directories and all subdirectories) and put it in '''C:\mozilla''' | ||
− | |||
− | |||
− | |||
− | |||
− | + | ==Configure the paths (paths.py)== | |
− | * | + | #To setup the appropriate paths for the framework, open the '''paths.py''' file in '''c:\mozilla\testing\performance\win32\''' and configure the paths: |
− | + | ===CYGWIN=== | |
+ | *Path to cygwin bash to get output from dump() in Firefox | ||
+ | *By default, the Cygwin you just installed is in your C:\ drive | ||
− | + | ===SYNC=== | |
− | * | + | *Run sync between Ts runs |
− | * | + | *By default, the Cygwin you just installed is in your C:\ drive |
− | |||
− | |||
− | |||
− | *REPORTS_DIR | + | ===BASE_PROFILE_DIR=== |
− | + | *Path to the base profile directory used for testing. | |
− | **Create C:\ | + | *Check that BASE_PROFILE_DIR is pointing to the '''base_profile''' directory |
+ | *If you didn't alter the dir structure, it would be: '''C:\mozilla\testing\performance\win32\base_profile''' | ||
+ | *Check there is content in the base_profile directory | ||
+ | *(Optional) For the page load test to work, modify the '''hostperm.1''' file | ||
+ | **Set to allow urls with scheme:file to open in new windows | ||
+ | **Set the preference to open new windows in a tab should be off. | ||
+ | |||
+ | ==REPORTS_DIR== | ||
+ | *Path to the directory that the generated reports will go into | ||
+ | **Create '''extension_perf_reports''' in '''C:\mozilla\testing\performance\win32\''' | ||
+ | |||
+ | OR | ||
+ | |||
+ | **Configure REPORT_DIR path to a directory where you want the reports to go into | ||
*INIT_URL | *INIT_URL | ||
**Path to the file url to load when initializing a new profile | **Path to the file url to load when initializing a new profile | ||
+ | **Check that '''initialize.html''' is in '''C:\mozilla\testing\performance\win32\''' | ||
+ | **Note: Don't change the local file url to a file path | ||
+ | <pre>INIT_URL = 'file:///c:/mozilla/testing/performance/win32/initialize.html'</pre> | ||
*TS_URL | *TS_URL | ||
**The path to the file url to load for startup test (Ts) | **The path to the file url to load for startup test (Ts) | ||
+ | **Check that '''startup_test.html''' is in '''C:\mozilla\testing\performance\win32\startup_test''' | ||
+ | **Note: Don't change the local file url to a file path | ||
+ | <pre>TS_URL = 'file:///c:/mozilla/testing/performance/win32/startup_test/startup_test.html?begin='</pre> | ||
+ | |||
*TP_URL | *TP_URL | ||
− | **The path to the file url to load for page load test (Tp) | + | **The path to the file url to load for page load test (Tp) |
+ | **Check that '''cycler.html''' is in '''C:\mozilla\testing\performance\win32\page_load_test''' | ||
+ | **Note: Don't change the local file url to a file path | ||
+ | <pre>TP_URL = 'file:///c:/mozilla/testing/performance/win32/page_load_test/cycler.html'</pre> | ||
+ | |||
#Create a YAML config file (config.yaml) with info about the profiles you want to test. Copy and paste the following into config.yaml: | #Create a YAML config file (config.yaml) with info about the profiles you want to test. Copy and paste the following into config.yaml: | ||
<pre> | <pre> | ||
# Filename will be appended to the timestamp in the report filename. | # Filename will be appended to the timestamp in the report filename. | ||
# Use letters and underscores only | # Use letters and underscores only | ||
− | filename: | + | filename: yourfilename |
# The title of the report | # The title of the report | ||
title: 1 GHz Celeron | title: 1 GHz Celeron | ||
Line 80: | Line 99: | ||
***use "preferences : {}" for no prefs | ***use "preferences : {}" for no prefs | ||
***use "extensions: {}" for none | ***use "extensions: {}" for none | ||
+ | **filename and title can be anything | ||
− | 5. Run '''run_tests.py''' with the name of your config file as an argument. You can use a space-separated list of config files, to generate a report of startup and page load times. The report will be placed in the REPORTS_DIR directory you specified in paths.py. | + | 5. Run '''run_tests.py''' with the name of your config file as an argument: <pre>$run_tests.py config.yaml</pre> |
− | + | *You can use a space-separated list of config files, to generate a report of startup and page load times. The report will be placed in the REPORTS_DIR directory you specified in paths.py. | |
=Directory Structure of Framework= | =Directory Structure of Framework= |
Revision as of 23:20, 27 November 2006
Contents
Framework Prerequisites
You'll need to download these before running performance testing:
- Python 2.4
- The scripts all run from Python 2.4. You will need the windows version (not the cygwin version).
- Python Win32 Extensions
- These extensions provide some support for process management and performance monitoring.
- Numerical Python
- This is required by Matplotlib.
- Matplotlib
- This library is used to plot the graphs of performance characteristics of Firefox during the page load test.
- PySyc
- This is used for reading/writing config files in YAML.
- Cygwin
- For some reason, we can't get output from Firefox unless it's wrapped in a Cygwin sh shell. So we use cygwin for that, and to sync between Firefox runs.
- msvcp71.dll
- Microsoft® C Runtime Library, v. 7.10.3077.0
- You need a copy of this file to run Python with all the libraries.
- If you have Visual Studio 2003 installed, you've already got it.
Framework Configuration
Configure the environment
- Make sure the prerequisites are installed.
- Make a directory for the framework:
$mkdir C:\mozilla
- Download entire framework (directories and all subdirectories) and put it in C:\mozilla
Configure the paths (paths.py)
- To setup the appropriate paths for the framework, open the paths.py file in c:\mozilla\testing\performance\win32\ and configure the paths:
CYGWIN
- Path to cygwin bash to get output from dump() in Firefox
- By default, the Cygwin you just installed is in your C:\ drive
SYNC
- Run sync between Ts runs
- By default, the Cygwin you just installed is in your C:\ drive
BASE_PROFILE_DIR
- Path to the base profile directory used for testing.
- Check that BASE_PROFILE_DIR is pointing to the base_profile directory
- If you didn't alter the dir structure, it would be: C:\mozilla\testing\performance\win32\base_profile
- Check there is content in the base_profile directory
- (Optional) For the page load test to work, modify the hostperm.1 file
- Set to allow urls with scheme:file to open in new windows
- Set the preference to open new windows in a tab should be off.
REPORTS_DIR
- Path to the directory that the generated reports will go into
- Create extension_perf_reports in C:\mozilla\testing\performance\win32\
OR
- Configure REPORT_DIR path to a directory where you want the reports to go into
- INIT_URL
- Path to the file url to load when initializing a new profile
- Check that initialize.html is in C:\mozilla\testing\performance\win32\
- Note: Don't change the local file url to a file path
INIT_URL = 'file:///c:/mozilla/testing/performance/win32/initialize.html'
- TS_URL
- The path to the file url to load for startup test (Ts)
- Check that startup_test.html is in C:\mozilla\testing\performance\win32\startup_test
- Note: Don't change the local file url to a file path
TS_URL = 'file:///c:/mozilla/testing/performance/win32/startup_test/startup_test.html?begin='
- TP_URL
- The path to the file url to load for page load test (Tp)
- Check that cycler.html is in C:\mozilla\testing\performance\win32\page_load_test
- Note: Don't change the local file url to a file path
TP_URL = 'file:///c:/mozilla/testing/performance/win32/page_load_test/cycler.html'
- Create a YAML config file (config.yaml) with info about the profiles you want to test. Copy and paste the following into config.yaml:
# Filename will be appended to the timestamp in the report filename. # Use letters and underscores only filename: yourfilename # The title of the report title: 1 GHz Celeron # Name of profile to test Test profile 1: # Path to Firefox to test firefox: C:\Program Files\Mozilla Firefox 2 Beta 1\firefox.exe # Preferences to set in the test (use "preferences : {}" for no prefs) preferences: javascript.options.showInConsole : true xpinstall.enabled : true # Extensions to install in test (use "extensions: {}" for none) extensions: # Need quotes around guid because of curly braces "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi foo@sample.com : c:\path\to\other\unzipped\xpi
- Note:
- Check that your firefox path is right
- Preferences and extensions can be left as none:
- use "preferences : {}" for no prefs
- use "extensions: {}" for none
- filename and title can be anything
$run_tests.py config.yaml
- You can use a space-separated list of config files, to generate a report of startup and page load times. The report will be placed in the REPORTS_DIR directory you specified in paths.py.
Directory Structure of Framework
page_load_test/
This directory contains the JavaScript files and html data files for the page load test. The page load test opens a new window and cycles through loading each html file, timing each load.
startup_test/
This directory contains the JavaScript to run the startup test. It measures how long it takes Firefox to start up.
base_profile/
This directory contains the base profile used for testing. A copy of this profile is made for each testing profile, and extensions or prefs are added according to the test_configs array in run_tests.py. For the page load test to run correctly, the hostperm.1 file must be set to allow scheme:file uris to open in new windows, and the pref to force a window to open in a tab must not be set. The dom.allow_scripts_to_close_windows pref should also be set to true. The browser.shell.checkDefaultBrowser pref should be set to false.
run_tests.py, paths.py
These files should be configured to run the test on different machines, with different extensions or preferences. See setup above.