Difference between revisions of "Firefox Performance Testing : A Python framework for Windows Documentation"
(→Directory Structure of Framework) |
(→Directory Structure of Framework) |
||
Line 130: | Line 130: | ||
*You can use a space-separated list of config files, to generate a report of startup and page load times. | *You can use a space-separated list of config files, to generate a report of startup and page load times. | ||
*The report will be placed in the '''REPORTS_DIR''' directory you specified in '''paths.py'''. | *The report will be placed in the '''REPORTS_DIR''' directory you specified in '''paths.py'''. | ||
+ | |||
=Directory Structure of Framework= | =Directory Structure of Framework= | ||
+ | |||
+ | ==Overview of Structure== | ||
+ | A glance at the Framework File Structure (CVS files not included): | ||
+ | |||
+ | <pre> | ||
+ | win32 | ||
+ | | | ||
+ | | | ||
+ | |__ base_profile (dir) | ||
+ | | | | ||
+ | | |__ bookmarkbackups (dir) | ||
+ | | | | | ||
+ | | | |__ .html files | ||
+ | | | | ||
+ | | |__ Cache (dir) | ||
+ | | | | ||
+ | | |__ .bak, .html, .ini, .dat, .txt, .js, .rdf, .mfl files | ||
+ | | | ||
+ | | | ||
+ | |__ page_load_test(dir) | ||
+ | | | | ||
+ | | |__ base(dir) | ||
+ | | | | | ||
+ | | | |__ other dirs and .html files | ||
+ | | | | ||
+ | | |__ cycler.html & report.html | ||
+ | | | ||
+ | | | ||
+ | |__ startup_test | ||
+ | | | | ||
+ | | |__ startup_test.html | ||
+ | | | ||
+ | | | ||
+ | |__ report dir | ||
+ | | | ||
+ | | | ||
+ | |__ run_tests.py, paths.py, config.yaml and other .py, .html files | ||
+ | |||
+ | </pre> | ||
+ | |||
+ | *NOTE: Content in '''base_profile''' dir may vary | ||
+ | |||
+ | The following is written by Annie Sullivan (annie.sullivan@gmail.com): | ||
+ | |||
==page_load_test/== | ==page_load_test/== | ||
− | This directory contains the JavaScript files and html data files for the page load test. The page load test opens a new window and cycles through loading each html file, timing each load. | + | *This directory contains the JavaScript files and html data files for the page load test. |
+ | *The page load test opens a new window and cycles through loading each html file, timing each load. | ||
==startup_test/== | ==startup_test/== | ||
− | This directory contains the JavaScript to run the startup test. It measures how long it takes Firefox to start up. | + | *This directory contains the JavaScript to run the startup test. |
+ | *It measures how long it takes Firefox to start up. | ||
==base_profile/== | ==base_profile/== | ||
− | This directory contains the base profile used for testing. A copy of this profile is made for each testing profile, and extensions or prefs are added according to the test_configs array in run_tests.py. For the page load test to run correctly, the hostperm.1 file must be set to allow scheme:file uris to open in new windows, and the pref to force a window to open in a tab must not be set. The dom.allow_scripts_to_close_windows pref should also be set to true. The browser.shell.checkDefaultBrowser pref should be set to false. | + | *This directory contains the base profile used for testing. |
+ | *A copy of this profile is made for each testing profile, and extensions or prefs are added according to the test_configs array in run_tests.py. | ||
+ | *For the page load test to run correctly, the hostperm.1 file must be set to allow scheme:file uris to open in new windows, and the pref to force a window to open in a tab must not be set. | ||
+ | *The dom.allow_scripts_to_close_windows pref should also be set to true. The browser.shell.checkDefaultBrowser pref should be set to false. | ||
==run_tests.py, paths.py== | ==run_tests.py, paths.py== | ||
− | These files should be configured to run the test on different machines, with different extensions or preferences. See setup above. | + | *These files should be configured to run the test on different machines, with different extensions or preferences. See setup above. |
Revision as of 00:35, 28 November 2006
Framework Prerequisites
You'll need to download these before running performance testing:
- Python 2.4
- The scripts all run from Python 2.4. You will need the windows version (not the cygwin version).
- Python Win32 Extensions
- These extensions provide some support for process management and performance monitoring.
- Numerical Python
- This is required by Matplotlib.
- Matplotlib
- This library is used to plot the graphs of performance characteristics of Firefox during the page load test.
- PySyc
- This is used for reading/writing config files in YAML.
- Cygwin
- For some reason, we can't get output from Firefox unless it's wrapped in a Cygwin sh shell. So we use cygwin for that, and to sync between Firefox runs.
- msvcp71.dll
- Microsoft® C Runtime Library, v. 7.10.3077.0
- You need a copy of this file to run Python with all the libraries.
- If you have Visual Studio 2003 installed, you've already got it.
Framework Configuration
Configure the environment
- Make sure the prerequisites are installed.
- Make a directory for the framework:
$mkdir C:\mozilla
- Download entire framework (directories and all subdirectories) and put it in C:\mozilla
Configure the paths (paths.py)
To setup the appropriate paths for the framework, open the paths.py file in c:\mozilla\testing\performance\win32\ and configure the paths:
CYGWIN
- Verify that the path is pointing to the desired cygwin directory
CYGWIN = r'c:\cygwin\bin\bash.exe -c'
- The path to cygwin bash to get output from dump() in Firefox
- By default, the Cygwin you just installed is in your C:\
SYNC
- Verify that the path is pointing to the desired cygwin directory
SYNC = r'c:\cygwin\bin\sync'
- Runs sync between Ts runs
- By default, the Cygwin you just installed is in C:\
BASE_PROFILE_DIR
- Verify that BASE_PROFILE_DIR is pointing to the base_profile directory
- If you didn't alter the directory structure, the base_profile path would be: C:\mozilla\testing\performance\win32\base_profile
BASE_PROFILE_DIR = r'C:\mozilla\testing\performance\win32\base_profile'
- IMPORTANT: Check that there is content in the base_profile directory
- (Optional) For the page load test to work, modify the hostperm.1 file
- Set to allow urls with scheme:file to open in new windows
- Set the preference to open new windows in a tab should be off.
REPORTS_DIR
- This is the path to the directory that the generated reports will go into
REPORTS_DIR = r'c:\C:\mozilla\testing\performance\win32\extension_perf_reports'
- Create extension_perf_reports in C:\mozilla\testing\performance\win32\
OR
- Set REPORT_DIR path to a directory where you want the reports to go into. Make sure that directory is created.
INIT_URL
- The path to the file url to load when initializing a new profile
INIT_URL = 'file:///c:/mozilla/testing/performance/win32/initialize.html'
- Check that initialize.html is in C:\mozilla\testing\performance\win32\
- Note: Don't change the local file url to a file path
TS_URL
- The path to the file url to load for startup test (Ts)
TS_URL = 'file:///c:/mozilla/testing/performance/win32/startup_test/startup_test.html?begin='
- Check that startup_test.html is in C:\mozilla\testing\performance\win32\startup_test
- Note: Don't change the local file url to a file path
TP_URL
- The path to the file url to load for page load test (Tp)
TP_URL = 'file:///c:/mozilla/testing/performance/win32/page_load_test/cycler.html'
- Check that cycler.html is in C:\mozilla\testing\performance\win32\page_load_test
- Note: Don't change the local file url to a file path
Config file
Config info
In C:\mozilla\testing\performance\win32\, create a YAML config file (config.yaml) with info about the profiles you want to test. Copy, paste and modify the following in config.yaml:
# Filename will be appended to the timestamp in the report filename. # Use letters and underscores only filename: yourfilename # The title of the report title: 1 GHz Celeron # Name of profile to test Test profile 1: # Path to Firefox to test firefox: C:\Program Files\Mozilla Firefox 2 Beta 1\firefox.exe # Preferences to set in the test (use "preferences : {}" for no prefs) preferences: javascript.options.showInConsole : true xpinstall.enabled : true # Extensions to install in test (use "extensions: {}" for none) extensions: # Need quotes around guid because of curly braces "{12345678-1234-1234-1234-abcd12345678}" : c:\path\to\unzipped\xpi foo@sample.com : c:\path\to\other\unzipped\xpi
Instructions:
1. Check that your firefox path is right
2. If you don't have preferences to set:
- change to:
preferences : {}
3. If you don't have extensions to install in test:
- change to:
extensions: {}
4. filename and title can be anything
5. Run run_tests.py with the name of your config file as an argument in c:\mozilla\testing\performance\win32\:$run_tests.py config.yaml
- You can use a space-separated list of config files, to generate a report of startup and page load times.
- The report will be placed in the REPORTS_DIR directory you specified in paths.py.
Directory Structure of Framework
Overview of Structure
A glance at the Framework File Structure (CVS files not included):
win32 | | |__ base_profile (dir) | | | |__ bookmarkbackups (dir) | | | | | |__ .html files | | | |__ Cache (dir) | | | |__ .bak, .html, .ini, .dat, .txt, .js, .rdf, .mfl files | | |__ page_load_test(dir) | | | |__ base(dir) | | | | | |__ other dirs and .html files | | | |__ cycler.html & report.html | | |__ startup_test | | | |__ startup_test.html | | |__ report dir | | |__ run_tests.py, paths.py, config.yaml and other .py, .html files
- NOTE: Content in base_profile dir may vary
The following is written by Annie Sullivan (annie.sullivan@gmail.com):
page_load_test/
- This directory contains the JavaScript files and html data files for the page load test.
- The page load test opens a new window and cycles through loading each html file, timing each load.
startup_test/
- This directory contains the JavaScript to run the startup test.
- It measures how long it takes Firefox to start up.
base_profile/
- This directory contains the base profile used for testing.
- A copy of this profile is made for each testing profile, and extensions or prefs are added according to the test_configs array in run_tests.py.
- For the page load test to run correctly, the hostperm.1 file must be set to allow scheme:file uris to open in new windows, and the pref to force a window to open in a tab must not be set.
- The dom.allow_scripts_to_close_windows pref should also be set to true. The browser.shell.checkDefaultBrowser pref should be set to false.
run_tests.py, paths.py
- These files should be configured to run the test on different machines, with different extensions or preferences. See setup above.