Difference between revisions of "BuildAPI"

From CDOT Wiki
Jump to: navigation, search
(→‎Project Details)
 
(36 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
== Project Name ==
 
== Project Name ==
  
−
BuildAPI
+
'''BuildAPI'''
  
 
== Project Description ==
 
== Project Description ==
 +
The BuildAPI project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying a database for the information on the BuildBot jobs, which will then be used to generate the necessary reports.
 +
 +
Initial contacts: [[User:Armenzg|Armenzg]]
  
−
Description should be no longer than a paragraph.  Include links to any relevant on-line resources.  For example, http://google.com or [http://developer.mozilla.org MDC].
 
  
 
== Project Leader(s) ==
 
== Project Leader(s) ==
−
 
 
Andrew Singh [http://zenit.senecac.on.ca/wiki/index.php/User:Asingh]
 
Andrew Singh [http://zenit.senecac.on.ca/wiki/index.php/User:Asingh]
  
−
Jing Yang [http://zenit.senecac.on.ca/wiki/index.php/User:Jing_yang]
 
  
 
== Project Contributor(s) ==
 
== Project Contributor(s) ==
 +
* [[User:Armenzg|Armenzg]]
 +
* Google Groups - Pylon-Discuss [http://groups.google.com/group/pylons-discuss]
 +
* IRC: aki, cthomas
 +
* [[User:ctyler|Chris Tyler]]
 +
 +
 +
== Project Details ==
 +
* '''Updated Objectives (October 28th 2010) from [[User:Armenzg|Armenzg]]'''
 +
* We will choose one of the following to do:
 +
** 1) Generate graphs, charts, CSVs and CPU totals for infrastructure load blog posts
 +
*** a) This is very useful and could move us forward towards having this information being published publicly for consumption
 +
*** b) I highly encourage this one as understanding the mental model behind it is easier
 +
** 2)Write a tool that analyzes our statusDB and figure out slaves that have been continually been burning jobs (sometimes it takes us several days to spot them)
  
−
Name(s) of people casually working on the project, or who have contributed significant help.  Include links to personal pages within wiki
 
  
−
NOTE: only Project Leader(s) should add names here. You '''can’t''' add your own name to the Contributor list.
+
''* Objective: To analyze and create reports from BuildBot data for multiple purposes. These reports will generated from querying buildbot databases to get the necessary information about jobs.
 +
* It will be used for purposes such as:
 +
:* identifying critical machines that should not be processing jobs
 +
:* discovering rogue slaves
 +
:* Generating reports on performance, usage, build times, downtime, and more''
  
−
== Project Details ==
+
* This project uses Pylons framework:
 +
:* Pylons combines the very best ideas from the worlds of Ruby, Python and Perl, providing a structured but extremely flexible Python web framework.
  
−
* '''Objective:''' Need to identify critical machines that should not be processing jobs
+
* We will need a snapshot of the database to test our work. For examples of the "statusdb" see:
 +
** http://anamariamoz.wordpress.com/
 +
** https://bugzilla.mozilla.org/show_bug.cgi?id=604088
  
−
* '''candidates:''' asingh,jing yang
+
* We will be generating different graphs from:
 +
** http://oduinn.com/blog/2010/09/09/infrastructure-load-for-august-2010/
 +
** Armen suggested that one student works on the first graph, and the other works on the second graph
 +
** We will also be working on the documentation
  
−
* Querying the database to get information about jobs
 
  
−
* From that, reports can be generated
 
−
** e.g., discover rogue slaves
 
  
−
* pylons project (Python web framework)
+
== Resources ==
 +
* Pylons-Discuss Group http://groups.google.com/group/pylons-discuss
 +
* Mozilla Software Engineering group (see BuildAPI wiki)
 +
* Channels such as #build, #pylons and #seneca or irc.mozilla.org and irc.freenode.net
  
−
* statusdb snapshot needed.  For examples see:
 
−
** http://anamariamoz.wordpress.com/
 
−
** https://bugzilla.mozilla.org/show_bug.cgi?id=604088
 
  
 
== Project Plan ==
 
== Project Plan ==
−
 
 
Goals for each release:
 
Goals for each release:
 
* 0.1
 
* 0.1
 +
** Complete a local installation of Python, Pylons, MySQL (with databases snapshots loaded) and google virtualization
 +
** Set up BuildAPI and be able to run the Paste server to use the pre-existing BuildAPI controllers
 +
** Create my '''own''' controller, models or views to test and understand Pylons concepts
 +
 
* 0.2
 
* 0.2
 +
** Pull information from the database and display it on a webpage USING the MVC model and integrating with the provided query.py script
 +
** Would like to generate a simple graph from the information that is pulled from the database
 +
** Note: Still need to understand how JSON and Java is being used with the existing scripts. For now I am comfortable with the MVC model and can apply it practically, for the most part.
 +
 
* 0.3
 
* 0.3
 +
** Full documentation provided on how to set up BuildAPI: http://asdesigned.ca/sbr/BuildAPI-How-To-Setup.pdf
 +
** Reflections on project and course posted on blog [http://andrewasdesigned.blogspot.com/]
 +
** Placed the data into JSON output
 +
 +
 +
 +
== Components to Learn/Get an idea of ==
 +
* Pylons Framework
 +
** Read chapters from The Definitive Guide to Pylons book
 +
* How to integrate GoogleAPI and JSON output with Pylons
 +
* Python
 +
* MVC (Model, View, Controller) Model
 +
 +
 +
 +
== Accomplishments ==
 +
* Created my own "project" using a controller, model and template.
 +
** The model uses the existing query.py file but I added my own queries.
 +
** The controller is called project.py, imports my queries from the query.py file and renders the template project.mako with the query results
 +
** I have removed the json because I do not fully understand it and I am taking it step by step
 +
* BuildAPI set up on http://iraq.proximity.on.ca:5000
 +
* Python, Mysql-Python, Google Visualizations API, BuildAPI installed locally on a virtual machine on my laptop
 +
* Databases installed from sql dump file
 +
* Able to run Paste server and BuildAPI pre-existing scripts
 +
* Able to run in from both regularly and from a python virtual environment (used for testing)
 +
* Created my own test script using a controller to display information from a selected table in the schedulerdb database
 +
 +
 +
 +
== Issues ==
 +
* '''Update: Resolved all below issues'''
 +
** I tried to install the same on our Seneca CDOT server: iraq.proximity.on.ca so that I could work off of the server but I ran into errors
 +
** Tried regular installation as well as using a python virtual environment
 +
** Serving content on iraq.proximity.on.ca
 +
 +
  
 
== Project News ==
 
== Project News ==
 +
* Completed Documentation for setting up BuildAPI
 +
** See Blog Post or here
 +
* November 18th, 2010
 +
** Updated some information on the page
 +
** Made substantial progress learning Pylons and MVC framework
 +
 +
* October 28th, 2010
 +
* What has been done so far
 +
** Please see my blog at: http://andrewasdesigned.blogspot.com/2010/10/beginning-previously-we-had-conference.html
 +
** I have been moving forward and have installed BuildAPI locally on my machine to test it out
 +
** Still need time to finish reading on Pylons!
  
 +
* October 22nd, 2010
 +
** New [http://people.mozilla.org/~catlee/schedulerdb-2010-10-22.sql.bz2 schedulerdb snapshot] available.
 
* October 14, 2010:
 
* October 14, 2010:
 
** Meeting of previously named "ScriptFactory" broke into three seperate projects: [[Mozharness]], [[BuildAPI]] and [[Release Simple Bugs]] -Mustafa
 
** Meeting of previously named "ScriptFactory" broke into three seperate projects: [[Mozharness]], [[BuildAPI]] and [[Release Simple Bugs]] -Mustafa
 +
** We will follow up with Armen on details of the Release Simple Bugs project and more on BuildAPI
 +
 +
* Wednesday October 13, 2010:
 +
** The ScriptFactory group met today with Armen and Ctyler in a conference call
 +
** We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs
 +
 +
 +
 +
== Links ==
 +
* Andrew Singh: http://andrewasdesigned.blogspot.com/
 +
* Armen Zambrano Gasparnian: http://armenzg.blogspot.com/
 +
* Planet CDOT: http://zenit.senecac.on.ca/wiki/index.php/Planet_CDOT
 +
* BuildBot Architecture: http://buildbot.net/buildbot/docs/0.8.1/images/overview.png
 +
* Database Snapshots: http://people.mozilla.org/~catlee/schedulerdbschema.png and http://people.mozilla.org/~catlee/schema.png
 +
 +
 +
 +
''Updated by Andrew Singh''

Latest revision as of 09:56, 17 December 2010

Project Name

BuildAPI

Project Description

The BuildAPI project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying a database for the information on the BuildBot jobs, which will then be used to generate the necessary reports.

Initial contacts: Armenzg


Project Leader(s)

Andrew Singh [1]


Project Contributor(s)


Project Details

  • Updated Objectives (October 28th 2010) from Armenzg
  • We will choose one of the following to do:
    • 1) Generate graphs, charts, CSVs and CPU totals for infrastructure load blog posts
      • a) This is very useful and could move us forward towards having this information being published publicly for consumption
      • b) I highly encourage this one as understanding the mental model behind it is easier
    • 2)Write a tool that analyzes our statusDB and figure out slaves that have been continually been burning jobs (sometimes it takes us several days to spot them)


* Objective: To analyze and create reports from BuildBot data for multiple purposes. These reports will generated from querying buildbot databases to get the necessary information about jobs.

  • It will be used for purposes such as:
  • identifying critical machines that should not be processing jobs
  • discovering rogue slaves
  • Generating reports on performance, usage, build times, downtime, and more
  • This project uses Pylons framework:
  • Pylons combines the very best ideas from the worlds of Ruby, Python and Perl, providing a structured but extremely flexible Python web framework.


Resources


Project Plan

Goals for each release:

  • 0.1
    • Complete a local installation of Python, Pylons, MySQL (with databases snapshots loaded) and google virtualization
    • Set up BuildAPI and be able to run the Paste server to use the pre-existing BuildAPI controllers
    • Create my own controller, models or views to test and understand Pylons concepts
  • 0.2
    • Pull information from the database and display it on a webpage USING the MVC model and integrating with the provided query.py script
    • Would like to generate a simple graph from the information that is pulled from the database
    • Note: Still need to understand how JSON and Java is being used with the existing scripts. For now I am comfortable with the MVC model and can apply it practically, for the most part.


Components to Learn/Get an idea of

  • Pylons Framework
    • Read chapters from The Definitive Guide to Pylons book
  • How to integrate GoogleAPI and JSON output with Pylons
  • Python
  • MVC (Model, View, Controller) Model


Accomplishments

  • Created my own "project" using a controller, model and template.
    • The model uses the existing query.py file but I added my own queries.
    • The controller is called project.py, imports my queries from the query.py file and renders the template project.mako with the query results
    • I have removed the json because I do not fully understand it and I am taking it step by step
  • BuildAPI set up on http://iraq.proximity.on.ca:5000
  • Python, Mysql-Python, Google Visualizations API, BuildAPI installed locally on a virtual machine on my laptop
  • Databases installed from sql dump file
  • Able to run Paste server and BuildAPI pre-existing scripts
  • Able to run in from both regularly and from a python virtual environment (used for testing)
  • Created my own test script using a controller to display information from a selected table in the schedulerdb database


Issues

  • Update: Resolved all below issues
    • I tried to install the same on our Seneca CDOT server: iraq.proximity.on.ca so that I could work off of the server but I ran into errors
    • Tried regular installation as well as using a python virtual environment
    • Serving content on iraq.proximity.on.ca


Project News

  • Completed Documentation for setting up BuildAPI
    • See Blog Post or here
  • November 18th, 2010
    • Updated some information on the page
    • Made substantial progress learning Pylons and MVC framework
  • October 22nd, 2010
  • October 14, 2010:
    • Meeting of previously named "ScriptFactory" broke into three seperate projects: Mozharness, BuildAPI and Release Simple Bugs -Mustafa
    • We will follow up with Armen on details of the Release Simple Bugs project and more on BuildAPI
  • Wednesday October 13, 2010:
    • The ScriptFactory group met today with Armen and Ctyler in a conference call
    • We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs


Links


Updated by Andrew Singh

Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=BuildAPI&oldid=53113"