Difference between revisions of "BuildAPI"
Line 42: | Line 42: | ||
** Armen suggested that one student works on the first graph, and the other works on the second graph | ** Armen suggested that one student works on the first graph, and the other works on the second graph | ||
** We will also be working on the documentation | ** We will also be working on the documentation | ||
+ | |||
+ | ''Updated by Andrew Singh'' | ||
Line 58: | Line 60: | ||
* 0.3 | * 0.3 | ||
** Be able to generate graphs, charts and CSVs from one of the databases based on CPU load and infrastructure load. | ** Be able to generate graphs, charts and CSVs from one of the databases based on CPU load and infrastructure load. | ||
+ | |||
+ | ''Updated by Andrew Singh'' | ||
Line 64: | Line 68: | ||
* BuildAPI database structure | * BuildAPI database structure | ||
* How to integrate GoogleAPI or another charting system with Pylons | * How to integrate GoogleAPI or another charting system with Pylons | ||
+ | |||
+ | ''Updated by Andrew Singh'' | ||
Line 72: | Line 78: | ||
* Able to run in from both regularly and from a python virtual environment (used for testing) | * Able to run in from both regularly and from a python virtual environment (used for testing) | ||
* Created my own test script using a controller to display information from a selected table in the schedulerdb database | * Created my own test script using a controller to display information from a selected table in the schedulerdb database | ||
+ | |||
+ | ''Updated by Andrew Singh'' | ||
Line 79: | Line 87: | ||
* Will seek help | * Will seek help | ||
* '''I would also like to set up Pylons to be accessible on iraq.proximity.on.ca, which I understand can be done through several methods such as through http_proxy (Apache web server) - thus will also check with Chris Tyler and other regarding this''' | * '''I would also like to set up Pylons to be accessible on iraq.proximity.on.ca, which I understand can be done through several methods such as through http_proxy (Apache web server) - thus will also check with Chris Tyler and other regarding this''' | ||
+ | |||
+ | ''Updated by Andrew Singh'' | ||
Line 96: | Line 106: | ||
** The ScriptFactory group met today with Armen and Ctyler in a conference call | ** The ScriptFactory group met today with Armen and Ctyler in a conference call | ||
** We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs | ** We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs | ||
+ | |||
+ | ''Updated by Andrew Singh'' | ||
+ | |||
== Links == | == Links == | ||
Line 104: | Line 117: | ||
* BuildBot Architecture: http://buildbot.net/buildbot/docs/0.8.1/images/overview.png | * BuildBot Architecture: http://buildbot.net/buildbot/docs/0.8.1/images/overview.png | ||
* Database Snapshots: http://people.mozilla.org/~catlee/schedulerdbschema.png and http://people.mozilla.org/~catlee/schema.png | * Database Snapshots: http://people.mozilla.org/~catlee/schedulerdbschema.png and http://people.mozilla.org/~catlee/schema.png | ||
+ | |||
+ | ''Updated by Andrew Singh'' |
Revision as of 07:42, 5 November 2010
Contents
Project Name
BuildAPI
Project Description
The BuildAPI project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying a database for the information on the BuildBot jobs, which will then be used to generate the necessary reports.
Initial contacts: Armenzg
Project Leader(s)
Andrew Singh [1]
Project Contributor(s)
Project Details
- Updated Objectives (October 28th 2010) from Armenzg
- We will choose one of the following to do:
- 1) Generate graphs, charts, CSVs and CPU totals for infrastructure load blog posts
- a) This is very useful and could move us forward towards having this information being published publicly for consumption
- b) I highly encourage this one as understanding the mental model behind it is easier
- 2)Write a tool that analyzes our statusDB and figure out slaves that have been continually been burning jobs (sometimes it takes us several days to spot them)
- 1) Generate graphs, charts, CSVs and CPU totals for infrastructure load blog posts
* Objective: To analyze and create reports from BuildBot data for multiple purposes. These reports will generated from querying buildbot databases to get the necessary information about jobs.
- It will be used for purposes such as:
- identifying critical machines that should not be processing jobs
- discovering rogue slaves
- Generating reports on performance, usage, build times, downtime, and more
- This project uses Pylons framework:
- Pylons combines the very best ideas from the worlds of Ruby, Python and Perl, providing a structured but extremely flexible Python web framework.
- We will need a snapshot of the database to test our work. For examples of the "statusdb" see:
- We will be generating different graphs from:
- http://oduinn.com/blog/2010/09/09/infrastructure-load-for-august-2010/
- Armen suggested that one student works on the first graph, and the other works on the second graph
- We will also be working on the documentation
Updated by Andrew Singh
Project Plan
Goals for each release:
- 0.1
- Complete a local installation of Python, Pylons, MySQL (with databases snapshots loaded) and google virtualization
- Set up BuildAPI and be able to run the Paste server to use the pre-existing BuildAPI controllers
- Create my own script that can generate a simple graph or to be able to pull information from a database
- 0.11 - Complete above script to generate a simple graph. At the moment I am able to pull information from the database. Further milestones to follow before 0.2.
- 0.2
- To be updated shortly
- 0.3
- Be able to generate graphs, charts and CSVs from one of the databases based on CPU load and infrastructure load.
Updated by Andrew Singh
What Needs to be Learned
- Pylons Framework
- BuildAPI database structure
- How to integrate GoogleAPI or another charting system with Pylons
Updated by Andrew Singh
Accomplishments
- Python, Mysql-Python, Google Visualizations API, BuildAPI installed locally on a virtual machine on my laptop
- Databases installed from sql dump file
- Able to run Paste server and BuildAPI pre-existing scripts
- Able to run in from both regularly and from a python virtual environment (used for testing)
- Created my own test script using a controller to display information from a selected table in the schedulerdb database
Updated by Andrew Singh
Issues
- I tried to install the same on our Seneca CDOT server: iraq.proximity.on.ca so that I could work off of the server but I ran into errors
- Tried regular installation as well as using a python virtual environment
- Will seek help
- I would also like to set up Pylons to be accessible on iraq.proximity.on.ca, which I understand can be done through several methods such as through http_proxy (Apache web server) - thus will also check with Chris Tyler and other regarding this
Updated by Andrew Singh
Project News
- What has been done so far
- Please see my blog at: http://andrewasdesigned.blogspot.com/2010/10/beginning-previously-we-had-conference.html
- I have been moving forward and have installed BuildAPI locally on my machine to test it out
- Still need time to finish reading on Pylons!
- October 22nd, 2010
- New schedulerdb snapshot available.
- October 14, 2010:
- Meeting of previously named "ScriptFactory" broke into three seperate projects: Mozharness, BuildAPI and Release Simple Bugs -Mustafa
- We will follow up with Armen on details of the Release Simple Bugs project and more on BuildAPI
- Wednesday October 13, 2010:
- The ScriptFactory group met today with Armen and Ctyler in a conference call
- We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs
Updated by Andrew Singh
Links
- Andrew Singh: http://andrewasdesigned.blogspot.com/
- Jing Yang: http://jyang42.blogspot.com/
- Armen Zambrano Gasparnian: http://armenzg.blogspot.com/
- Planet CDOT: http://zenit.senecac.on.ca/wiki/index.php/Planet_CDOT
- BuildBot Architecture: http://buildbot.net/buildbot/docs/0.8.1/images/overview.png
- Database Snapshots: http://people.mozilla.org/~catlee/schedulerdbschema.png and http://people.mozilla.org/~catlee/schema.png
Updated by Andrew Singh