Changes

← Older edit

Firefox Performance Testing : A Python framework for Windows

21,605 bytes added, 14:22, 17 April 2013

m

Reverted edits by Saoagent (Talk) to last revision by Elichak

= Project Contributor(s) =

[[User:~~Bhearsum~~ dwwoodsi| ~~Ben Hearsum~~Dean Woodside]] (~~bhearsum~~dean)*~~Set up~~ Submitted an sh file that automates the ~~VM for~~ tedious performance testing*Helped with the debugging process for report.py, run_tests.py and ts.py ~~[[User:Themystic | Tom Aratyn]] (mystic)~~*Introduced Closures in Pythonframework configuration

Alice Nodelman

*Discussed on the things that need to be fixed to improve and strengthen the framework

*Gave suggestions on the new Performance Testing framework

[[User:Bhearsum | Ben Hearsum]] (bhearsum)

*Set up the VM for performance testing

*Helped get me started with the debugging process for report.py, run_tests.py and ts.py

[[User:Mylau | Michael Lau]] (mylau)

*Added comments on the documentation for setting up Performance Testing framework for Windows

*Tested the new and improved Performance Testing framework(version 1)*Gave constructive feedback on ~~new~~ documentation(version 1)

[[User:Eor | Eva Or]] (eor)

*Tested the new and improved Performance Testing framework(version 1)*Gave constructive feedback on new documentation(version 1)

[[User:djhamp-g | David Hamp Gonsalves ]] (inveigle)

*Gave pointers on flushing buffer

*Helped with some grammar and sentence structuring for documentation (version 1)*Tested and gave constructive feedback on the framework (version 1)*Looked into a batch file for automating configuration, gave pointers*Tested and commented on the new ~~documentation~~framework (version 2) [[User: robcee | Rob Campbell]] (robcee)*Gave me a number of python tips :) [[User:Themystic | Tom Aratyn]] (mystic)*Introduced Closures in Python

==In-Class Contributors==

* [[User:themystic| Tom Aratyn]] (mystic)

* [[User:djhamp-g | David Hamp]]

* [[User:dwwoodsi|Dean Woodside]] (dean)

= Project Details =

==Improved Documentation==

===Latest===

[[New_Firefox_Performance_Testing_Documentation | New Firefox Performance Testing Documentation]]

===First Attempt===

[[Firefox_Performance_Testing_:_A_Python_framework_for_Windows_Documentation | Performance Testing Setup Configuration Documentation]]

==Details==

*[http://wiki.mozilla.org/MozillaQualityAssurance:Performance_Testing Mozilla Quality Assurance Performance Testing]

==Submission==

Still in progress!<br />

While it's sizzling, try out the Firefox Performance Testing Framework (version 2) at [[New_Firefox_Performance_Testing_Documentation#Quick_start| New Firefox Performance Testing Documentation Quick Start section]] - 3 easy steps

<th style="width: 20%; background-color:#333333; color:#F3F3F3">Artifact</th>

<th style="width: 30%%; background-color:#333333; color:#F3F3F3">Details</th>

<th style="width: 20%; background-color:#333333; color:#F3F3F3">Links</th>

<tr>

<td valign="top">run_tests.py</td>

*Python file used to execute the Firefox Performance Testing

*Some of the things it does:

** Validates files and directories needed for Performance Testing

** Displays informative messages to allow users to fix any invalidity in files or directories

** Displays an informative progress/status bar that informs users how far into the testing they are

*Things that Alice and I discussed are reflected in this file:

** Configuration checker

*** yaml file validator

*** paths.py validator

**** checking the paths for existance

***** notify user if path doesnt exist and ask user if they want it created

**** checking if the directories have contents

</td>

[[run_tests.py]]

</td>

</tr>

<tr>

<td valign="top">perfconfig.sh</td>

<td valign="top">This script is used to automate the tedious Firefox Performance Testing Configuration

*Main Contributor

** [[User:dwwoodsi|Dean Woodside]] (dean)

*With this script:

** Configuration of python framework is eased

</td>

[[perfconfig.sh]]

</td>

</tr>

<tr>

<td valign="top">Progress bar class</td>

*Modified open source code to be used for this project

*Displays a progress/status bar so users will know at which stage of the Performance Testing they are at (by percentage%, sort of)

</td>

[[pb.py]]

</td>

</tr>

<tr>

<td valign="top">paths.py</td>

*This is the file that contains all the paths that are needed for the Performance Testing

*Fixed the paths and modified some documentation</td>

[[paths.py]]

</td>

</tr>

<tr>

<td valign="top">Firefox Performance Testing Documentation (Version 1)</td>

*This was the first documentation that was written to improve the documentation provided in the initial

Firefox Performance Testing Framework (Refer to [[Readme.txt]])

*It was deemed to be too elaborative and needed to be more concise

*Needed work

*Users didn't like the configuration steps they had to take

** For more ranting:[[#Comments_on_the_Firefox_Performance_Testing_Framework | DPS909 Student comments]]

</td>

[[Firefox_Performance_Testing_:_A_Python_framework_for_Windows_Documentation | Performance Testing Setup Configuration Documentation]]

</td>

</tr>

<tr>

<td valign="top">Firefox Performance Testing Documentation (Version 2)</td>

*<b>To try Firefox Performance Testing out, go to this documentation</b>

*This the new documentation that is improved with the [[#Comments_on_the_Firefox_Performance_Testing_Framework | student comments]]

*Building a user base.. some comments on this NEW framework

**[[New Firefox Performance Testing Framework comments]]

<b>Goals achieved:</b>

* Effective

**Easy-to-use

**Free from confusion

**Informative and helpful

*Efficient

**Takes almost no time to set up the framework

**Captures <b>most</b> user errors through validation and saves user time and frustration to fix an error while running the framework

***Informs user of the directory/file invalidity or if they ran an invalid config file etc.

Additional:

*Users don't have to know the whole framework to configure it

** Documentation helps

*Users are not left staring at the console wondering what is happening when they run the Performance Tests

** Progress bar and informative messages

*Users don't have to waste hours/days/weeks debugging the code to find out why they are having problems configuring the framework

** I spent 2 weeks configuring the <strike>bloody</strike> framework

</td>

[[New_Firefox_Performance_Testing_Documentation | New Firefox Performance Testing Documentation]]

</td>

</tr>

<tr>

<td valign="top">Status Documentation</td>

*Here are the progress charts for this project

*They are divided into 3:

** Performance Testing Framework First Version

** Performance Testing Framework Second Version

** More details of things to fix for the framework

</td>

*[[Things to fix for Performance Testing Framework | More details of things to fix for Performance Testing Framework]]

*[[#Extended_Progress_Chart_.28Version_2.29 | Version 2]]

*[[#Progress_.28Version_1.29 | Version 1]]

</td>

</tr>

<tr>

Firefox Performance Testing Framework Directory Structure

</td>

*Here is an overview of the Firefox Performance Testing Framework Directory Structure.

*The directory structure is revised

*There were too many extra libraries to load which caused too many dependencies

*Directory structure was difficult to keep track due to redundant directories and extra libraries

*All configuration is in one place!

** config.yaml

** paths.py

** constants

</td>

[[#Directory_Structure_of_Framework| Performance Testing Framework Directory Structure]]

</td>

</tr>

<tr>

Screen shots

</td>

*Here is a screen shot of the program when it is doing a performance testing:

[[Image:Perftesting running.png | Firefox Performance Testing running]]

</td>

Not Applicable

</td>

</tr>

<tr>

Next Steps

</td>

*Am I done? Not quite

</td>

[[#Next_Steps | Next Big Steps]]

</td>

</tr>

<tr>

Reflections

</td>

*Some thoughts

</td>

[[#Reflections_on_the_project | Reflections]]

</td>

</tr>

</table>

==Next Steps==

<th style="width: 30%; background-color:#333333; color:#F3F3F3">Details</th>

<th style="width: 10%; background-color:#333333; color:#F3F3F3">Priority</th>

<th style="width: 15%; background-color:#333333; color:#F3F3F3">Contributors</th>

<th style="width: 20%; background-color:#333333; color:#F3F3F3">Status</th>

<tr>

<td valign="top">Porting framework to other OSes</td>

* <b>What good is a performance testing framework if it only runs on Windows??</b>

* This has been a pending task for awhile.

* Since several students made a point that it should port to other OSes, I should look into that

* Students who use OS X, Unix Oses etc. couldn't test this framework

</td>

* [[User:elichak|Liz Chak]]

</td>

<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">0% completed</div>

</td>

</tr>

<tr>

*Will have a discussion with Alice Nodelman in regards to this

</td>

<td style="background-color: #E9DA73; text-align: center; font-weight: bold; color: #333333;" valign="top">Medium</td>

* [[User:elichak|Liz Chak]]

</td>

<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">0% completed</div>

</td>

</tr>

<tr>

<td valign="top">Create and execute new tests</td>

*New test case(s) for performance

*Will have a discussion with Alice Nodelman in regards to this

</td>

<td style="background-color: #E9DA73; text-align: center; font-weight: bold; color: #333333;" valign="top">Medium</td>

* [[User:elichak|Liz Chak]]

</td>

<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">0% completed</div>

</td>

</tr>

</table>

==Extended Progress Chart(Version 2)==

<th style="width: 30%; background-color:#333333; color:#F3F3F3">Details</th>

<th style="width: 10%; background-color:#333333; color:#F3F3F3">Priority</th>

<th style="width: 15%; background-color:#333333; color:#F3F3F3">Contributors</th>

<th style="width: 1520%; background-color:#333333; color:#F3F3F3">Status</th>

<tr>

<td valign="top">Read all student comments about framework and improve it</td>

* On Nov 29th, students of DPS909 contributed by testing the framework

* They have listed a constructive feedback and many great suggestions to help improve the setup of the framework

** [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#Comments_on_the_Firefox_Performance_Testing_Framework | Student comments about framework]]

</td>

* Nevertheless, all the comments are really REALLY helpful to facilitate in improving this framework.

<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">30100% completed</div></td></tr><tr><td valign="top">Drill and narrow down the student comments to improve framework</td><td valign="top">*Improve framework as per suggested by the students: [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#Comments_on_the_Firefox_Performance_Testing_Framework | Student comments about framework]] *Only pick the important ones*[[Things to fix for Performance Testing Framework | More details]]</td><td style="background-color: #990000; text-align: center; font-weight: bold; color: #ffffff;" valign="top">High</td><td valign="top">* [[User:elichak|Liz Chak]]</td><td valign="top">* Working on some of the comments that helps make the framework stronger and more robust* Some tasks might be redundant. Nonetheless, I'm clear about what should be fixed<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">In progress</div></td></tr><tr><td valign="top">Fix [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows_Documentation | Performance Testing Setup Configuration Documentation]] and framework configuration</td><td valign="top">* From the [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#Comments_on_the_Firefox_Performance_Testing_Framework | student comments about the framework]], I have gathered that the documentation have to be fixed as some parts are vague and some important areas are not highlighted. * Some Items ([[Things to fix for Performance Testing Framework | More details]]):** Make the ''Create extension_perf_reports directory''' part bold*** Maybe ask the user if they want it to be generated if it doesn't exist (will be done in framework)*** Should it be created from the sh script??** '''config.yaml''' file*** If user chooses to test with no preferences or extensions, should mention about commenting the lines beneath them.** Additional details regarding Framework Prerequisites would reduce confusion. *** Comment from student: There isn't any harm in "dumbing it down" a level. *[[Things to fix for Performance Testing Framework | More details]] </td><td style="background-color: #990000; text-align: center; font-weight: bold; color: #ffffff;" valign="top">High</td><td valign="top">* [[User:elichak|Liz Chak]]</td><td valign="top"><b>New status</b> [[New Firefox Performance Testing Documentation]] <b>Older status</b> Will start on this after drilling down the [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#Comments_on_the_Firefox_Performance_Testing_Framework | student comments]] <div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">In progress</div></td></tr><tr><td valign="top">Pre-requisites: Python, Cywin and .dll Installation Guidelines</td><td valign="top">* Order of installations* Cygwin**Students were confused as to whether they should keep the default [http://www.cygwin.com/setup.exe Cygwin] setup/packages as the documentation didn't outline that.**They inquired which Cygwin packages are required* Concerns about where the .[http://www.driverskit.com/dll/msvcp71.dll/2371.html msvcp71.dll] file should be installed after it's downloaded * A few students got the error: ImportError: No module named win32pdh ** missed out install the Python Win32 Extensions - sh script handles it now* Now that the sh script handles the pre-requisite installation, are the following necessary?** Some files seem to have dependencies on another. Need to point whether some files have to be installed in order.** Some files have the same installation GUI, which makes it hard to track down which have been installed and which have not. Some screenshots may help.** For '''Cygwin''' installation, should note that default package setting is sufficient. Remove all doubts.</td><td style="background-color: #E9DA73; text-align: center; font-weight: bold; color: #333333;" valign="top">Medium</td><td valign="top">* [[User:elichak|Liz Chak]]</td><td valign="top">*All these issues are mitigated by the [[perfconfig.sh]] script.*This script automates all the tedious steps you have to go through to run your performance testing which was, unfortunately, loathed by users. (Based on [[#Comments_on_the_Firefox_Performance_Testing_Framework | student comments]])*See it for yourself! Try it out: [[New_Firefox_Performance_Testing_Documentation#Quick_start| New Firefox Performance Testing Documentation Quick Start section]]*I'm also gathering user feedback: [[New Firefox Performance Testing Framework comments]]<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">100% done</div></td></tr><tr><td valign="top">Configuring Environment Specifications in documentation</td><td valign="top"><b>BASE_PROFILE_DIR</b>* No instructions on what exactly to place in the Hostperm.1 file.* Hostperm.1 file is autogenerated and warns against editing it.* Mention how to modify the Hostperm.1 file.</td><td style="background-color: #E9DA73; text-align: center; font-weight: bold; color: #333333;" valign="top">Medium</td><td valign="top">* [[User:elichak|Liz Chak]]</td><td valign="top">I have nothing to work with besides the [[Readme.txt | README.TXT]] file by Annie Sullivan. Therefore, I have to consult Alice Nodelman in this area.<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">In progress</div>

</td>

</tr>

<tr>

<td valign="top">~~Write batch script to automate the~~ Automate environment configuration ~~of [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows_Documentation#Framework_Prerequisites | pre-requisites]]~~</td>

*One of the [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#Comments_on_the_Firefox_Performance_Testing_Framework | student comments]] is to write a ~~batch~~ bat/sh file to automate the configurations of the pre-requisites for the framework

</td>

<td style="background-color: #E9DA73; text-align: center; font-weight: bold; color: #333333;" valign="top">Medium</td>

* [[User:dwwoodsi|Dean Woodside]] (dean)** Submitted sh script* [[User:djhamp-g | David Hamp-Gonsalves]](inveigle)** Looked into this, gave pointers

</td>

* ~~David Hamp~~ [[User:dwwoodsi|Dean Woodside]] has submitted a sh scripting for this. It works great. Needs to be touched up**[http://www.yorku.ca/abel/liz/mozilla/perfconfig.zip Sh script zip] * [[User:elichak |Liz Chak]] will fix up the sh script**Doesn't make a reports folder (should this be ~~doing this~~in the sh script??)**Still runs the whole script even if it couldn't download the pre-requisites, should terminate**Missed out the paths.py file**Requires touch ups, but everything else is good**Proper documentation Files:*[[perfconfig.sh]] - Source code*[http://www.yorku.ca/abel/liz/mozilla/perfconfig.sh Actual Script to configure Firefox Performance Testing Framework]<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">0100% completed</div>

</td>

</tr>

<tr>

<td valign="top">Porting framework to other OSes</td>

</table>

==Progress(Version 1)==

</td>

<b>Newer status</b>

The DPS909 class tested the documentation and framework out.

*Needed rework

<b>Older status</b>

*Improving the current documentation so that it's easier to follow

*Making sure that all the configuration documents are in one place

*This is done along with the code base work I'm doing

<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">90100% completed ~~- Needs to be reviewed and tested out~~</div>

</td>

</tr>

*I have fixed the program to check for the existence of the base_profile dir and it also checks if the bookmarkbackups, Cache dirs and perf.js dir exist.

<div style="width:100%;background-color:#F3F3F3">95100% completed</div>

</td>

80100% completed - needs to be reviewed</div>

</td>

</tr>

*Scroll down and look for dom.allow_scripts_to_close_windows

*Double click on it to set it to true

=Directory Structure of Framework=

==Overview of Structure==

A glance at the Framework File Structure (CVS files not included):

<pre>

win32

|

|__ base_profile (dir)

| |

| |__ bookmarkbackups (dir)

| | |

| | |__ .html files

| |

| |__ Cache (dir)

| |

| |__ .bak, .html, .ini, .dat, .txt, .js, .rdf, .mfl files

|

|__ page_load_test(dir)

| |

| |__ base(dir)

| | |

| | |__ other dirs and .html files

| |

| |__ cycler.html & report.html

|

|__ startup_test

| |

| |__ startup_test.html

|

|__ extension_perf_reports (dir for generated reports)

|

|__ run_tests.py, paths.py, config.yaml and other .py, .html files

</pre>

*NOTE: Content in '''base_profile''' dir may vary

The following is written by Annie Sullivan (annie.sullivan@gmail.com):

==base_profile/==

*This directory contains the base profile used for testing.

*A copy of this profile is made for each testing profile, and extensions or prefs are added according to the test_configs array in run_tests.py.

*For the page load test to run correctly, the hostperm.1 file must be set to allow scheme:file uris to open in new windows, and the pref to force a window to open in a tab must not be set.

*The dom.allow_scripts_to_close_windows pref should also be set to true. The browser.shell.checkDefaultBrowser pref should be set to false.

==page_load_test/==

*This directory contains the JavaScript files and html data files for the page load test.

*The page load test opens a new window and cycles through loading each html file, timing each load.

==startup_test/==

*This directory contains the JavaScript to run the startup test.

*It measures how long it takes Firefox to start up.

==extension_perf_report/==

*This directory is where the generated report will go into.

*You may specify another directory to substitute this directory, but make sure that the '''paths.py''' is changed to point to it.

==run_tests.py, paths.py==

*These files should be configured to run the test on different machines, with different extensions or preferences. See setup above.

= Project News =

==Tuesday, 21 Nov 2006==

Refer to progress chart.

[[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#~~Progress~~Progress_.28Version_1.29| Performance Testing Framework progress chart]] ==Wednesday, 29 Nov 2006==In-class Performance Testing Framework Configuration* DPS909 students contributed by testing the framework* Students have listed constructive feedback and many great suggestions to help improve the setup of the framework** [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#Comments_on_the_Firefox_Performance_Testing_Framework | Student comments about framework]] ==Sunday, 3 Dec 2006==* [[User:dwwoodsi|Dean Woodside]] has submitted a sh scripting for this. It works great. Needs to be touched up.**[http://www.yorku.ca/abel/mozilla/perfconfig.zip Sh script zip] * [[User:elichak |Liz Chak]] will fix up the sh script**Doesn't make a reports folder **Still runs the whole script even if it couldn't download the pre-requisites, should terminate**Missed out the paths.py file**Requires touch ups, but everything else is good ==Sunday, 10 Dec 2006==* Narrowing down the things that need to be done before 13 December 2006* Refer to [[#Extended_Progress_Chart_.28Version_2.29 | Extended Progress Chart]] ==Wednesday, 13 Dec 2006==* Created a [[New_Firefox_Performance_Testing_Documentation | new documentation]]* Finished my parts for the Performance Testing Framework** Refer to [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#Submission | submission]] The current Firefox Performance Testing Framework is effective and efficient. *<b>To try Firefox Performance Testing out, go to this documentation</b>*This the new documentation that is improved with the [[#Comments_on_the_Firefox_Performance_Testing_Framework | student comments]]*Building a user base.. some comments on this NEW framework**[[New Firefox Performance Testing Framework comments]] <b>Goals achieved:</b>* Effective**Easy-to-use**Free from confusion**Informative and helpful*Efficient**Takes almost no time to set up the framework**Captures <b>most</b> user errors through validation and saves user time and frustration to fix an error while running the framework***Informs user of the directory/file invalidity or if they ran an invalid config file etc. Things To-do (Immediately)*Clean up this page - almost done*Document source - almost done*Fix up the [[New_Firefox_Performance_Testing_Documentation | Performance Testing Documentation]]

=Project References=

*I recommend these sites to learn about python:

* [[davey_girl_comments| Erin Davey]]

* [[dwwoodsi perf comments| Dean Woodside]]

==Reflections on the project==

Now that the first phase of the project is done, I would like to sit back and reflect on some of the experiences I have had with this project. Not only did this project grow, I have been growing with it. I've learned endless amount of things by working on this project (not just technical stuff). And as the saying goes, "you only learn from experience"!

===Configuration frustration===

Words can't express how I was initially frustrated with the framework. I hit brick walls a countless number of times when I tried to set the framework up and running and I was on the verge of giving up. It was only through my own perseverance and determination that I got this to work.

Outcome: A list of things to fix on the framework to ease the configuration, strengthen the framework and better documentation!!

===First deliverable===

I asked the class to test my first deliverable. I had faith that my framework would bring delight to my testers, but I was proven otherwise. Even with the effort that I've put into my first deliverable, it still created a group of frustrated and agonized users. Many of them ranted on what could be improved on the framework, hence:

Outcome: A list of things to automate the framework and EASE THE CONFIGURATION

===Final deliverable===

I'm glad that my first tester was thrilled about my Performance Testing. There is nothing better than a happy user. What I've learned is that TESTING is the KEY to a successful application. Why do so many of us developers neglect that??

===Credits===

[[User:dwwoodsi|Dean Woodside]], Alice Nodelman, [[User:Bhearsum | Ben Hearsum]], [[User:Mylau | Michael Lau]], [[User:Eor | Eva Or]], [[User:djhamp-g | David Hamp Gonsalves ]], [[User:David.humphrey|Dave Humphrey]], [[User:robcee|Rob Campbell]] and of course, the DPS909 class who tested my framework. These are the individuals who played a significant part in making this framework headed toward success!

==Bon Echo Community Test Day==

Tiago

3

edits

CDOT Wiki β

Changes

Firefox Performance Testing : A Python framework for Windows

CDOT Wiki ^β