Difference between revisions of "BuildAPI"

From CDOT Wiki
Jump to: navigation, search
 
(32 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
== Project Name ==
 
== Project Name ==
  
BuildAPI
+
'''BuildAPI'''
  
 
== Project Description ==
 
== Project Description ==
 
The BuildAPI project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying a database for the information on the BuildBot jobs, which will then be used to generate the necessary reports.
 
The BuildAPI project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying a database for the information on the BuildBot jobs, which will then be used to generate the necessary reports.
  
 +
Initial contacts: [[User:Armenzg|Armenzg]]
  
Initial contacts: [[User:Armenzg|Armenzg]]
 
  
 
== Project Leader(s) ==
 
== Project Leader(s) ==
 
 
Andrew Singh [http://zenit.senecac.on.ca/wiki/index.php/User:Asingh]
 
Andrew Singh [http://zenit.senecac.on.ca/wiki/index.php/User:Asingh]
  
Jing Yang [http://zenit.senecac.on.ca/wiki/index.php/User:Jing_yang]
 
  
 
== Project Contributor(s) ==
 
== Project Contributor(s) ==
 +
* [[User:Armenzg|Armenzg]]
 +
* Google Groups - Pylon-Discuss [http://groups.google.com/group/pylons-discuss]
 +
* IRC: aki, cthomas
 +
* [[User:ctyler|Chris Tyler]]
  
* [[User:Armenzg|Armenzg]]
 
  
 
== Project Details ==
 
== Project Details ==
 +
* '''Updated Objectives (October 28th 2010) from [[User:Armenzg|Armenzg]]'''
 +
* We will choose one of the following to do:
 +
** 1) Generate graphs, charts, CSVs and CPU totals for infrastructure load blog posts
 +
*** a) This is very useful and could move us forward towards having this information being published publicly for consumption
 +
*** b) I highly encourage this one as understanding the mental model behind it is easier
 +
** 2)Write a tool that analyzes our statusDB and figure out slaves that have been continually been burning jobs (sometimes it takes us several days to spot them)
 +
  
* Objective: To analyze and create reports from BuildBot data for multiple purposes. These reports will generated from querying buildbot databases to get the necessary information about jobs.
+
''* Objective: To analyze and create reports from BuildBot data for multiple purposes. These reports will generated from querying buildbot databases to get the necessary information about jobs.
 
* It will be used for purposes such as:
 
* It will be used for purposes such as:
 
:* identifying critical machines that should not be processing jobs
 
:* identifying critical machines that should not be processing jobs
 
:* discovering rogue slaves
 
:* discovering rogue slaves
:* Generating reports on performance, usage, build times, downtime, and more
+
:* Generating reports on performance, usage, build times, downtime, and more''
  
 
* This project uses Pylons framework:
 
* This project uses Pylons framework:
Line 38: Line 46:
 
** Armen suggested that one student works on the first graph, and the other works on the second graph
 
** Armen suggested that one student works on the first graph, and the other works on the second graph
 
** We will also be working on the documentation
 
** We will also be working on the documentation
 +
 +
 +
 +
== Resources ==
 +
* Pylons-Discuss Group http://groups.google.com/group/pylons-discuss
 +
* Mozilla Software Engineering group (see BuildAPI wiki)
 +
* Channels such as #build, #pylons and #seneca or irc.mozilla.org and irc.freenode.net
 +
  
 
== Project Plan ==
 
== Project Plan ==
 +
Goals for each release:
 +
* 0.1
 +
** Complete a local installation of Python, Pylons, MySQL (with databases snapshots loaded) and google virtualization
 +
** Set up BuildAPI and be able to run the Paste server to use the pre-existing BuildAPI controllers
 +
** Create my '''own''' controller, models or views to test and understand Pylons concepts
  
Goals for each release:
 
* 0.1 -- to be updated on October 15th, 2010
 
 
* 0.2
 
* 0.2
 +
** Pull information from the database and display it on a webpage USING the MVC model and integrating with the provided query.py script
 +
** Would like to generate a simple graph from the information that is pulled from the database
 +
** Note: Still need to understand how JSON and Java is being used with the existing scripts. For now I am comfortable with the MVC model and can apply it practically, for the most part.
 +
 
* 0.3
 
* 0.3
 +
** Full documentation provided on how to set up BuildAPI: http://asdesigned.ca/sbr/BuildAPI-How-To-Setup.pdf
 +
** Reflections on project and course posted on blog [http://andrewasdesigned.blogspot.com/]
 +
** Placed the data into JSON output
 +
 +
 +
 +
== Components to Learn/Get an idea of ==
 +
* Pylons Framework
 +
** Read chapters from The Definitive Guide to Pylons book
 +
* How to integrate GoogleAPI and JSON output with Pylons
 +
* Python
 +
* MVC (Model, View, Controller) Model
 +
 +
 +
 +
== Accomplishments ==
 +
* Created my own "project" using a controller, model and template.
 +
** The model uses the existing query.py file but I added my own queries.
 +
** The controller is called project.py, imports my queries from the query.py file and renders the template project.mako with the query results
 +
** I have removed the json because I do not fully understand it and I am taking it step by step
 +
* BuildAPI set up on http://iraq.proximity.on.ca:5000
 +
* Python, Mysql-Python, Google Visualizations API, BuildAPI installed locally on a virtual machine on my laptop
 +
* Databases installed from sql dump file
 +
* Able to run Paste server and BuildAPI pre-existing scripts
 +
* Able to run in from both regularly and from a python virtual environment (used for testing)
 +
* Created my own test script using a controller to display information from a selected table in the schedulerdb database
 +
 +
 +
 +
== Issues ==
 +
* '''Update: Resolved all below issues'''
 +
** I tried to install the same on our Seneca CDOT server: iraq.proximity.on.ca so that I could work off of the server but I ran into errors
 +
** Tried regular installation as well as using a python virtual environment
 +
** Serving content on iraq.proximity.on.ca
 +
 +
  
 
== Project News ==
 
== Project News ==
 +
* Completed Documentation for setting up BuildAPI
 +
** See Blog Post or here
 +
* November 18th, 2010
 +
** Updated some information on the page
 +
** Made substantial progress learning Pylons and MVC framework
 +
 +
* October 28th, 2010
 +
* What has been done so far
 +
** Please see my blog at: http://andrewasdesigned.blogspot.com/2010/10/beginning-previously-we-had-conference.html
 +
** I have been moving forward and have installed BuildAPI locally on my machine to test it out
 +
** Still need time to finish reading on Pylons!
 +
 +
* October 22nd, 2010
 +
** New [http://people.mozilla.org/~catlee/schedulerdb-2010-10-22.sql.bz2 schedulerdb snapshot] available.
 
* October 14, 2010:
 
* October 14, 2010:
 
** Meeting of previously named "ScriptFactory" broke into three seperate projects: [[Mozharness]], [[BuildAPI]] and [[Release Simple Bugs]] -Mustafa
 
** Meeting of previously named "ScriptFactory" broke into three seperate projects: [[Mozharness]], [[BuildAPI]] and [[Release Simple Bugs]] -Mustafa
Line 54: Line 127:
 
** The ScriptFactory group met today with Armen and Ctyler in a conference call
 
** The ScriptFactory group met today with Armen and Ctyler in a conference call
 
** We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs
 
** We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs
 +
  
  
 
== Links ==
 
== Links ==
 
* Andrew Singh: http://andrewasdesigned.blogspot.com/
 
* Andrew Singh: http://andrewasdesigned.blogspot.com/
* Jing Yang: http://jyang42.blogspot.com/
 
 
* Armen Zambrano Gasparnian: http://armenzg.blogspot.com/
 
* Armen Zambrano Gasparnian: http://armenzg.blogspot.com/
 
* Planet CDOT: http://zenit.senecac.on.ca/wiki/index.php/Planet_CDOT
 
* Planet CDOT: http://zenit.senecac.on.ca/wiki/index.php/Planet_CDOT
 +
* BuildBot Architecture: http://buildbot.net/buildbot/docs/0.8.1/images/overview.png
 +
* Database Snapshots: http://people.mozilla.org/~catlee/schedulerdbschema.png and http://people.mozilla.org/~catlee/schema.png
 +
 +
 +
 +
''Updated by Andrew Singh''

Latest revision as of 10:56, 17 December 2010

Project Name

BuildAPI

Project Description

The BuildAPI project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying a database for the information on the BuildBot jobs, which will then be used to generate the necessary reports.

Initial contacts: Armenzg


Project Leader(s)

Andrew Singh [1]


Project Contributor(s)


Project Details

  • Updated Objectives (October 28th 2010) from Armenzg
  • We will choose one of the following to do:
    • 1) Generate graphs, charts, CSVs and CPU totals for infrastructure load blog posts
      • a) This is very useful and could move us forward towards having this information being published publicly for consumption
      • b) I highly encourage this one as understanding the mental model behind it is easier
    • 2)Write a tool that analyzes our statusDB and figure out slaves that have been continually been burning jobs (sometimes it takes us several days to spot them)


* Objective: To analyze and create reports from BuildBot data for multiple purposes. These reports will generated from querying buildbot databases to get the necessary information about jobs.

  • It will be used for purposes such as:
  • identifying critical machines that should not be processing jobs
  • discovering rogue slaves
  • Generating reports on performance, usage, build times, downtime, and more
  • This project uses Pylons framework:
  • Pylons combines the very best ideas from the worlds of Ruby, Python and Perl, providing a structured but extremely flexible Python web framework.


Resources


Project Plan

Goals for each release:

  • 0.1
    • Complete a local installation of Python, Pylons, MySQL (with databases snapshots loaded) and google virtualization
    • Set up BuildAPI and be able to run the Paste server to use the pre-existing BuildAPI controllers
    • Create my own controller, models or views to test and understand Pylons concepts
  • 0.2
    • Pull information from the database and display it on a webpage USING the MVC model and integrating with the provided query.py script
    • Would like to generate a simple graph from the information that is pulled from the database
    • Note: Still need to understand how JSON and Java is being used with the existing scripts. For now I am comfortable with the MVC model and can apply it practically, for the most part.


Components to Learn/Get an idea of

  • Pylons Framework
    • Read chapters from The Definitive Guide to Pylons book
  • How to integrate GoogleAPI and JSON output with Pylons
  • Python
  • MVC (Model, View, Controller) Model


Accomplishments

  • Created my own "project" using a controller, model and template.
    • The model uses the existing query.py file but I added my own queries.
    • The controller is called project.py, imports my queries from the query.py file and renders the template project.mako with the query results
    • I have removed the json because I do not fully understand it and I am taking it step by step
  • BuildAPI set up on http://iraq.proximity.on.ca:5000
  • Python, Mysql-Python, Google Visualizations API, BuildAPI installed locally on a virtual machine on my laptop
  • Databases installed from sql dump file
  • Able to run Paste server and BuildAPI pre-existing scripts
  • Able to run in from both regularly and from a python virtual environment (used for testing)
  • Created my own test script using a controller to display information from a selected table in the schedulerdb database


Issues

  • Update: Resolved all below issues
    • I tried to install the same on our Seneca CDOT server: iraq.proximity.on.ca so that I could work off of the server but I ran into errors
    • Tried regular installation as well as using a python virtual environment
    • Serving content on iraq.proximity.on.ca


Project News

  • Completed Documentation for setting up BuildAPI
    • See Blog Post or here
  • November 18th, 2010
    • Updated some information on the page
    • Made substantial progress learning Pylons and MVC framework
  • October 22nd, 2010
  • October 14, 2010:
    • Meeting of previously named "ScriptFactory" broke into three seperate projects: Mozharness, BuildAPI and Release Simple Bugs -Mustafa
    • We will follow up with Armen on details of the Release Simple Bugs project and more on BuildAPI
  • Wednesday October 13, 2010:
    • The ScriptFactory group met today with Armen and Ctyler in a conference call
    • We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs


Links


Updated by Andrew Singh