3
edits
Changes
m
To While it's sizzling, try out the Firefox Performance Testing Framework, go to (version 2) at [[New_Firefox_Performance_Testing_Documentation#Quick_start| New Firefox Performance Testing Documentation Quick Start section]] - 3 easy steps
[[New Firefox Performance Testing Documentation]]
80100% completed - needs to be reviewed</div>
[[User:Mylau | Michael Lau]] (mylau)
*Added comments on the documentation for setting up Performance Testing framework for Windows
*Tested the new and improved Performance Testing framework(version 1)*Gave constructive feedback on new documentation(version 1)
[[User:Eor | Eva Or]] (eor)
*Tested the new and improved Performance Testing framework(version 1)*Gave constructive feedback on new documentation(version 1)
[[User:djhamp-g | David Hamp Gonsalves ]] (inveigle)
*Gave pointers on flushing buffer
*Helped with some grammar and sentence structuring for documentation (version 1)*Tested and gave constructive feedback on the framework (version 1)*Looked into a batch file for automating configuration, gave pointers*Tested and commented on the new documentationframework (version 2) [[User: robcee | Rob Campbell]] (robcee)*Gave me a number of python tips :)
[[User:Themystic | Tom Aratyn]] (mystic)
==Submission==
Still in progress!<br />
<table style="width:100%;" border="1px" valign="top" class="standard-table" cellpadding=2 cellspacing=0>
<th style="width: 20%; background-color:#333333; color:#F3F3F3">Artifact</th>
<td valign="top">run_tests.py</td>
<td valign="top">
*This file is the Python file used to execute the Firefox Performance Testing*Needs proper source documentationSome of the things it does:*Need * Validates files and directories needed for Performance Testing** Displays informative messages to allow users to fix some of any invalidity in files or directories** Displays an informative progress/status bar that informs users how far into the testing they are*Things that Alice and I discussed are reflected in this file:** Configuration checker*** yaml file validator*** paths.py validator**** checking the paths for existance***** notify user if path doesnt exist and ask user if they want it created**** checking if the indentationdirectories have contents
</td>
<td valign="top">
*Main Contributor
** [[User:dwwoodsi|Dean Woodside]] (dean)
*With this script:
** Configuration of python framework is eased
</td>
<td valign="top">
<tr>
<td valign="top">Progress bar class</td>
<td valign="top">*Modified open source code to be used for this project*Displays a progress/status bar so users will know at which stage of the Performance Testing they are at (by percentage%, sort of)</td>
<td valign="top">
[[pb.py]]
<tr>
<td valign="top">paths.py</td>
<td valign="top">*This is the file that contains all the paths that are needed for the Performance Testing*Fixed the paths and modified some documentation</td>
<td valign="top">
[[paths.py]]
<td valign="top">
*This was the first documentation that was written to improve the documentation provided in the initial
Firefox Performance Testing Framework(Refer to [[Readme.txt]])
*It was deemed to be too elaborative and needed to be more concise
*Needed work
*Users didn't like the configuration steps they had to take
** For more ranting:[[#Comments_on_the_Firefox_Performance_Testing_Framework | DPS909 Student comments]]
</td>
<td valign="top">
<td valign="top">
*<b>To try Firefox Performance Testing out, go to this documentation</b>
*This the new documentation that is improved with the [[#Comments_on_the_Firefox_Performance_Testing_Framework | student comments]]*Building a user base.. some comments on this NEW framework**[[New Firefox Performance Testing Framework comments]] <b>Goals achieved:</b>* Effective**Easy-to-use**Free from confusion**Informative and helpful*Efficient**Takes almost no time to set up the framework**Captures <b>most</b> user errors through validation and saves user time and frustration to fix an error while running the framework***Informs user of the directory/file invalidity or if they ran an invalid config file etc. Additional:*Users don't have to know the whole framework to configure it** Documentation helps*Users are not left staring at the console wondering what is happening when they run the Performance Tests** Progress bar and informative messages*Users don't have to waste hours/days/weeks debugging the code to find out why they are having problems configuring the framework** I spent 2 weeks configuring the <strike>bloody</strike> framework
</td>
<td valign="top">
[[New_Firefox_Performance_Testing_Documentation | New Firefox Performance Testing Documentation]]
</td>
</tr>
<tr>
<td valign="top">Status Documentation</td>
<td valign="top">
*Here are the progress charts for this project
*They are divided into 3:
** Performance Testing Framework First Version
** Performance Testing Framework Second Version
** More details of things to fix for the framework
</td>
<td valign="top">
*[[Things to fix for Performance Testing Framework | More details of things to fix for Performance Testing Framework]]
*[[#Extended_Progress_Chart_.28Version_2.29 | Version 2]]
*[[#Progress_.28Version_1.29 | Version 1]]
</td>
</tr>
<tr>
<td valign="top">
Firefox Performance Testing Framework Directory Structure
</td>
<td valign="top">
*Here is an overview of the Firefox Performance Testing Framework Directory Structure.
*The directory structure is revised
*There were too many extra libraries to load which caused too many dependencies
*Directory structure was difficult to keep track due to redundant directories and extra libraries
*All configuration is in one place!
** config.yaml
** paths.py
** constants
</td>
<td valign="top">
[[#Directory_Structure_of_Framework| Performance Testing Framework Directory Structure]]
</td>
</tr>
<tr>
<td valign="top">
Screen shots
</td>
<td valign="top">
*Here is a screen shot of the program when it is doing a performance testing:
[[Image:Perftesting running.png | Firefox Performance Testing running]]
</td>
<td valign="top">
Not Applicable
</td>
</tr>
<tr>
<td valign="top">
Next Steps
</td>
<td valign="top">
*Am I done? Not quite
</td>
<td valign="top">
[[#Next_Steps | Next Big Steps]]
</td>
</tr>
<tr>
<td valign="top">
Reflections
</td>
<td valign="top">
*Some thoughts
</td>
<td valign="top">
[[#Reflections_on_the_project | Reflections]]
</td>
</tr>
</table>
==Next Steps==<table style="width:100%;" border="1px" valign="top" class="standard-table" cellpadding=2 cellspacing=0><th style="width: 15%; background-color:#333333; color:#F3F3F3">Task</th><th style="width: 30%; background-color:#333333; color:#F3F3F3">Details</th><th style="width: 10%; background-color:#333333; color:#F3F3F3">Priority</th><th style="width: 15%; background-color:#333333; color:#F3F3F3">Contributors</th><th style="width: 20%; background-color:#333333; color:#F3F3F3">Status</th><tr><td valign="top">Porting framework to other OSes</td><td valign="top">* <b>What good is a performance testing framework if it only runs on Windows??</b>* This has been a pending task for awhile. * Since several students made a point that it should port to other OSes, I should look into that* Students who use OS X, Unix Oses etc. couldn't test this framework</td><td style="background-color: #990000; text-align: center; font-weight: bold; color: #ffffff;" valign="top">High</td><td valign="top">* [[User:elichak|Liz Chak]]</td><td valign="top"> <div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">0% completed</div></td></tr><tr><td valign="top">Run both ts and tp at the same time</td><td valign="top">*Will have a discussion with Alice Nodelman in regards to this</td> <td style="background-color: #E9DA73; text-align: center; font-weight: bold; color: #333333;" valign="top">Medium</td><td valign="top">* [[User:elichak|Liz Chak]]</td><td valign="top"><div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">0% completed</div></td></tr> <tr><td valign="top">Create and execute new tests</td><td valign="top">*New test case(s) for performance*Will have a discussion with Alice Nodelman in regards to this</td><td style="background-color: #E9DA73; text-align: center; font-weight: bold; color: #333333;" valign="top">Medium</td><td valign="top">* [[User:elichak|Liz Chak]]</td><td valign="top"><div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">0% completed</div></td></tr> </table> ==Extended Progress Chart(Version 2)==
<table style="width:100%;" border="1px" valign="top" class="standard-table" cellpadding=2 cellspacing=0>
<th style="width: 15%; background-color:#333333; color:#F3F3F3">Task</th>
</tr>
<tr>
<td valign="top">Improve Drill and narrow down the student comments to improve framework as per commented by the students</td>
<td valign="top">
*Improve framework as per suggested by the students: [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#Comments_on_the_Firefox_Performance_Testing_Framework | Student comments about framework]]
*Only pick the important ones
*[[Things to fix for Performance Testing Framework | More details]]
</td>
<td style="background-color: #990000; text-align: center; font-weight: bold; color: #ffffff;" valign="top">High</td>
<td valign="top">
* From the [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#Comments_on_the_Firefox_Performance_Testing_Framework | student comments about the framework]], I have gathered that the documentation have to be fixed as some parts are vague and some important areas are not highlighted.
* Some Items([[Things to fix for Performance Testing Framework | More details]]):
** Make the ''Create extension_perf_reports directory''' part bold
*** Maybe ask the user if they want it to be generated if it doesn't exist (will be done in framework)
*** Should it be created from the sh script??
** '''config.yaml''' file
*** If uers user chooses to test with no preferences or extensions, should mention about commenting the lines beneath them.
** Additional details regarding Framework Prerequisites would reduce confusion.
*** Comment from student: There isn't any harm in "dumbing it down" a level.
*[[Things to fix for Performance Testing Framework | more..More details]]
</td>
</td>
<td valign="top">
<b>New status</b>
[[New Firefox Performance Testing Documentation]]
<b>Older status</b>
Will start on this after drilling down the [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#Comments_on_the_Firefox_Performance_Testing_Framework | student comments]]
<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">In progress</div>
</td>
<td valign="top">
*All these issues are mitigated by the [[perfconfig.sh]] script.
*This script automates all the tedious steps you have to go through to run your performance testing which was, unfortunately, loathed by users. (Based on [[#Comments_on_the_Firefox_Performance_Testing_Framework | student comments]])*See it for yourself! Try it out: [[New_Firefox_Performance_Testing_Documentation#Quick_start| New Firefox Performance Testing Documentation Quick Start section]]*I'm also gathering user feedback: [[New Firefox Performance Testing Framework comments]]
<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">100% done</div>
</td>
</td>
<td valign="top">
I have nothing to work with besides the [[Readme.txt | README.TXT]] file by Annie Sullivan. Therefore, I have to consult Alice Nodelman in this area.
<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">In progress</div>
</td>
**Proper documentation
Files:*[[perfconfig.sh]] - Source code*[http://www.yorku.ca/abel/liz/mozilla/perfconfig.sh Actual Script to configure Firefox Performance Testing Framework]
<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">100% completed</div>
</td>
</table>
==Progress(Version 1)==
<table style="width:100%;" border="1px" valign="top" class="standard-table" cellpadding=2 cellspacing=0>
</td>
<td valign="top">
<b>Newer status</b>
The DPS909 class tested the documentation and framework out.
*Needed rework
<b>Older status</b>
*Improving the current documentation so that it's easier to follow
*Making sure that all the configuration documents are in one place
*This is done along with the code base work I'm doing
<div style="width:100%;background-color:#F3F3F3;position:relative;margin-bottom:0px;">90100% completed - Needs to be reviewed and tested out</div>
</td>
</tr>
*I have fixed the program to check for the existence of the base_profile dir and it also checks if the bookmarkbackups, Cache dirs and perf.js dir exist.
<div style="width:100%;background-color:#F3F3F3">95100% completed</div>
</td>
</td>
</tr>
*Scroll down and look for dom.allow_scripts_to_close_windows
*Double click on it to set it to true
=Directory Structure of Framework=
==Overview of Structure==
A glance at the Framework File Structure (CVS files not included):
<pre>
win32
|
|
|__ base_profile (dir)
| |
| |__ bookmarkbackups (dir)
| | |
| | |__ .html files
| |
| |__ Cache (dir)
| |
| |__ .bak, .html, .ini, .dat, .txt, .js, .rdf, .mfl files
|
|
|__ page_load_test(dir)
| |
| |__ base(dir)
| | |
| | |__ other dirs and .html files
| |
| |__ cycler.html & report.html
|
|
|__ startup_test
| |
| |__ startup_test.html
|
|
|__ extension_perf_reports (dir for generated reports)
|
|
|__ run_tests.py, paths.py, config.yaml and other .py, .html files
</pre>
*NOTE: Content in '''base_profile''' dir may vary
The following is written by Annie Sullivan (annie.sullivan@gmail.com):
==base_profile/==
*This directory contains the base profile used for testing.
*A copy of this profile is made for each testing profile, and extensions or prefs are added according to the test_configs array in run_tests.py.
*For the page load test to run correctly, the hostperm.1 file must be set to allow scheme:file uris to open in new windows, and the pref to force a window to open in a tab must not be set.
*The dom.allow_scripts_to_close_windows pref should also be set to true. The browser.shell.checkDefaultBrowser pref should be set to false.
==page_load_test/==
*This directory contains the JavaScript files and html data files for the page load test.
*The page load test opens a new window and cycles through loading each html file, timing each load.
==startup_test/==
*This directory contains the JavaScript to run the startup test.
*It measures how long it takes Firefox to start up.
==extension_perf_report/==
*This directory is where the generated report will go into.
*You may specify another directory to substitute this directory, but make sure that the '''paths.py''' is changed to point to it.
==run_tests.py, paths.py==
*These files should be configured to run the test on different machines, with different extensions or preferences. See setup above.
= Project News =
==Tuesday, 21 Nov 2006==
Refer to progress chart.
[[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#ProgressProgress_.28Version_1.29| Performance Testing Framework progress chart]]
==Wednesday, 29 Nov 2006==
==Sunday, 10 Dec 2006==
* Narrowing down the things that need to be done before 13 December 2006
* Refer to [[#Extended_Progress_Chart Extended_Progress_Chart_.28Version_2.29 | Extended Progress Chart]]
==Wednesday, 13 Dec 2006==
* Created a [[New_Firefox_Performance_Testing_Documentation | new documentation]]
* Finished my parts for the Performance Testing Framework
** Refer to [[Firefox_Performance_Testing_:_A_Python_framework_for_Windows#Submission | submission]] The current Firefox Performance Testing Framework is effective and efficient. *<b>To try Firefox Performance Testing out, go to this documentation</b>*This the new documentation that is improved with the [[#Comments_on_the_Firefox_Performance_Testing_Framework | student comments]]*Building a user base.. some comments on this NEW framework**[[New Firefox Performance Testing Framework comments]] <b>Goals achieved:</b>* Effective**Easy-to-use**Free from confusion**Informative and helpful*Efficient**Takes almost no time to set up the framework**Captures <b>most</b> user errors through validation and saves user time and frustration to fix an error while running the framework***Informs user of the directory/file invalidity or if they ran an invalid config file etc.
Things To-do (Immediately)
*Clean up this page- almost done*Document source- almost done
*Fix up the [[New_Firefox_Performance_Testing_Documentation | Performance Testing Documentation]]
* [[davey_girl_comments| Erin Davey]]
* [[dwwoodsi perf comments| Dean Woodside]]
==Reflections on the project==
Now that the first phase of the project is done, I would like to sit back and reflect on some of the experiences I have had with this project. Not only did this project grow, I have been growing with it. I've learned endless amount of things by working on this project (not just technical stuff). And as the saying goes, "you only learn from experience"!
===Configuration frustration===
Words can't express how I was initially frustrated with the framework. I hit brick walls a countless number of times when I tried to set the framework up and running and I was on the verge of giving up. It was only through my own perseverance and determination that I got this to work.
Outcome: A list of things to fix on the framework to ease the configuration, strengthen the framework and better documentation!!
===First deliverable===
I asked the class to test my first deliverable. I had faith that my framework would bring delight to my testers, but I was proven otherwise. Even with the effort that I've put into my first deliverable, it still created a group of frustrated and agonized users. Many of them ranted on what could be improved on the framework, hence:
Outcome: A list of things to automate the framework and EASE THE CONFIGURATION
===Final deliverable===
I'm glad that my first tester was thrilled about my Performance Testing. There is nothing better than a happy user. What I've learned is that TESTING is the KEY to a successful application. Why do so many of us developers neglect that??
===Credits===
[[User:dwwoodsi|Dean Woodside]], Alice Nodelman, [[User:Bhearsum | Ben Hearsum]], [[User:Mylau | Michael Lau]], [[User:Eor | Eva Or]], [[User:djhamp-g | David Hamp Gonsalves ]], [[User:David.humphrey|Dave Humphrey]], [[User:robcee|Rob Campbell]] and of course, the DPS909 class who tested my framework. These are the individuals who played a significant part in making this framework headed toward success!
==Bon Echo Community Test Day==
