Difference between revisions of "BuildAPI"

From CDOT Wiki
Jump to: navigation, search
Line 42: Line 42:
 
** Armen suggested that one student works on the first graph, and the other works on the second graph
 
** Armen suggested that one student works on the first graph, and the other works on the second graph
 
** We will also be working on the documentation
 
** We will also be working on the documentation
 +
 +
''Updated by Andrew Singh''
  
  
Line 58: Line 60:
 
* 0.3
 
* 0.3
 
** Be able to generate graphs, charts and CSVs from one of the databases based on CPU load and infrastructure load.
 
** Be able to generate graphs, charts and CSVs from one of the databases based on CPU load and infrastructure load.
 +
 +
''Updated by Andrew Singh''
  
  
Line 64: Line 68:
 
* BuildAPI database structure
 
* BuildAPI database structure
 
* How to integrate GoogleAPI or another charting system with Pylons
 
* How to integrate GoogleAPI or another charting system with Pylons
 +
 +
''Updated by Andrew Singh''
  
  
Line 72: Line 78:
 
* Able to run in from both regularly and from a python virtual environment (used for testing)
 
* Able to run in from both regularly and from a python virtual environment (used for testing)
 
* Created my own test script using a controller to display information from a selected table in the schedulerdb database
 
* Created my own test script using a controller to display information from a selected table in the schedulerdb database
 +
 +
''Updated by Andrew Singh''
  
  
Line 79: Line 87:
 
* Will seek help
 
* Will seek help
 
* '''I would also like to set up Pylons to be accessible on iraq.proximity.on.ca, which I understand can be done through several methods such as through http_proxy (Apache web server) - thus will also check with Chris Tyler and other regarding this'''
 
* '''I would also like to set up Pylons to be accessible on iraq.proximity.on.ca, which I understand can be done through several methods such as through http_proxy (Apache web server) - thus will also check with Chris Tyler and other regarding this'''
 +
 +
''Updated by Andrew Singh''
  
  
Line 96: Line 106:
 
** The ScriptFactory group met today with Armen and Ctyler in a conference call
 
** The ScriptFactory group met today with Armen and Ctyler in a conference call
 
** We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs
 
** We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs
 +
 +
''Updated by Andrew Singh''
 +
  
 
== Links ==
 
== Links ==
Line 104: Line 117:
 
* BuildBot Architecture: http://buildbot.net/buildbot/docs/0.8.1/images/overview.png
 
* BuildBot Architecture: http://buildbot.net/buildbot/docs/0.8.1/images/overview.png
 
* Database Snapshots: http://people.mozilla.org/~catlee/schedulerdbschema.png and http://people.mozilla.org/~catlee/schema.png
 
* Database Snapshots: http://people.mozilla.org/~catlee/schedulerdbschema.png and http://people.mozilla.org/~catlee/schema.png
 +
 +
''Updated by Andrew Singh''

Revision as of 07:42, 5 November 2010

Project Name

BuildAPI

Project Description

The BuildAPI project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying a database for the information on the BuildBot jobs, which will then be used to generate the necessary reports.

Initial contacts: Armenzg


Project Leader(s)

Andrew Singh [1]


Project Contributor(s)

Project Details

  • Updated Objectives (October 28th 2010) from Armenzg
  • We will choose one of the following to do:
    • 1) Generate graphs, charts, CSVs and CPU totals for infrastructure load blog posts
      • a) This is very useful and could move us forward towards having this information being published publicly for consumption
      • b) I highly encourage this one as understanding the mental model behind it is easier
    • 2)Write a tool that analyzes our statusDB and figure out slaves that have been continually been burning jobs (sometimes it takes us several days to spot them)


* Objective: To analyze and create reports from BuildBot data for multiple purposes. These reports will generated from querying buildbot databases to get the necessary information about jobs.

  • It will be used for purposes such as:
  • identifying critical machines that should not be processing jobs
  • discovering rogue slaves
  • Generating reports on performance, usage, build times, downtime, and more
  • This project uses Pylons framework:
  • Pylons combines the very best ideas from the worlds of Ruby, Python and Perl, providing a structured but extremely flexible Python web framework.

Updated by Andrew Singh


Project Plan

Goals for each release:

  • 0.1
    • Complete a local installation of Python, Pylons, MySQL (with databases snapshots loaded) and google virtualization
    • Set up BuildAPI and be able to run the Paste server to use the pre-existing BuildAPI controllers
    • Create my own script that can generate a simple graph or to be able to pull information from a database
    • 0.11 - Complete above script to generate a simple graph. At the moment I am able to pull information from the database. Further milestones to follow before 0.2.
  • 0.2
    • To be updated shortly
  • 0.3
    • Be able to generate graphs, charts and CSVs from one of the databases based on CPU load and infrastructure load.

Updated by Andrew Singh


What Needs to be Learned

  • Pylons Framework
  • BuildAPI database structure
  • How to integrate GoogleAPI or another charting system with Pylons

Updated by Andrew Singh


Accomplishments

  • Python, Mysql-Python, Google Visualizations API, BuildAPI installed locally on a virtual machine on my laptop
  • Databases installed from sql dump file
  • Able to run Paste server and BuildAPI pre-existing scripts
  • Able to run in from both regularly and from a python virtual environment (used for testing)
  • Created my own test script using a controller to display information from a selected table in the schedulerdb database

Updated by Andrew Singh


Issues

  • I tried to install the same on our Seneca CDOT server: iraq.proximity.on.ca so that I could work off of the server but I ran into errors
  • Tried regular installation as well as using a python virtual environment
  • Will seek help
  • I would also like to set up Pylons to be accessible on iraq.proximity.on.ca, which I understand can be done through several methods such as through http_proxy (Apache web server) - thus will also check with Chris Tyler and other regarding this

Updated by Andrew Singh


Project News

  • October 22nd, 2010
  • October 14, 2010:
    • Meeting of previously named "ScriptFactory" broke into three seperate projects: Mozharness, BuildAPI and Release Simple Bugs -Mustafa
    • We will follow up with Armen on details of the Release Simple Bugs project and more on BuildAPI
  • Wednesday October 13, 2010:
    • The ScriptFactory group met today with Armen and Ctyler in a conference call
    • We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs

Updated by Andrew Singh


Links

Updated by Andrew Singh