BuildAPI

Project Name

BuildAPI

Project Description

The BuildAPI project consists of generating analytic reports which can be used for a multitude of purposes such as performance and usage tests or discovering rogue buildbot slave machines. Moreover, it will require querying a database for the information on the BuildBot jobs, which will then be used to generate the necessary reports.

Initial contacts: Armenzg

Project Leader(s)

Andrew Singh [1]

Project Contributor(s)

• Armenzg
• Google Groups - Pylon-Discuss [2]
• IRC: aki, cthomas
• Chris Tyler

Project Details

• Updated Objectives (October 28th 2010) from Armenzg
• We will choose one of the following to do:
  • 1) Generate graphs, charts, CSVs and CPU totals for infrastructure load blog posts
    • a) This is very useful and could move us forward towards having this information being published publicly for consumption
    • b) I highly encourage this one as understanding the mental model behind it is easier
  • 2)Write a tool that analyzes our statusDB and figure out slaves that have been continually been burning jobs (sometimes it takes us several days to spot them)

* Objective: To analyze and create reports from BuildBot data for multiple purposes. These reports will generated from querying buildbot databases to get the necessary information about jobs.

• It will be used for purposes such as:

• identifying critical machines that should not be processing jobs
• discovering rogue slaves
• Generating reports on performance, usage, build times, downtime, and more

• This project uses Pylons framework:

• Pylons combines the very best ideas from the worlds of Ruby, Python and Perl, providing a structured but extremely flexible Python web framework.

• We will need a snapshot of the database to test our work. For examples of the "statusdb" see:
  • http://anamariamoz.wordpress.com/
  • https://bugzilla.mozilla.org/show_bug.cgi?id=604088

• We will be generating different graphs from:
  • http://oduinn.com/blog/2010/09/09/infrastructure-load-for-august-2010/
  • Armen suggested that one student works on the first graph, and the other works on the second graph
  • We will also be working on the documentation

Resources

• I have signed up for the mailing list at http://groups.google.com/group/pylons-discuss as there is a lot of archived information on Pylons and of course, I can post my own questions there
• Mozilla Software Engineering group (see BuildAPI wiki)
• Channels such as #build, #pylons and #seneca or irc.mozilla.org and irc.freenode.net

Project Plan

Goals for each release:

• 0.1
  • Complete a local installation of Python, Pylons, MySQL (with databases snapshots loaded) and google virtualization
  • Set up BuildAPI and be able to run the Paste server to use the pre-existing BuildAPI controllers
  • Create my own controller, models or views to test and understand Pylons concepts

• 0.2
  • Pull information from the database and display it on a webpage USING the MVC model and integrating with the provided query.py script
  • Would like to generate a simple graph from the information that is pulled from the database
  • Note: Still need to understand how JSON and Java is being used with the existing scripts. For now I am comfortable with the MVC model and can apply it practically, for the most part.

• 0.3
  • Documentation provided on how to set up BuildAPI
  • Reflections on project and course posted on blog
  • Placed the data into JSON output

What Needed to be Learned

• Pylons Framework
  • Learned some Pylons, read chapters from The Definitive Guide to Pylons book and also have a practical understanding of the MVC model
• How to integrate GoogleAPI or another charting system with Pylons
  • And also how json works with the templates

Accomplishments

• Created my own "project" using a controller, model and template.
  • The model uses the existing query.py file but I added my own queries.
  • The controller is called project.py, imports my queries from the query.py file and renders the template project.mako with the query results
  • I have removed the json because I do not fully understand it and I am taking it step by step
• BuildAPI set up on http://iraq.proximity.on.ca:5000
• Python, Mysql-Python, Google Visualizations API, BuildAPI installed locally on a virtual machine on my laptop
• Databases installed from sql dump file
• Able to run Paste server and BuildAPI pre-existing scripts
• Able to run in from both regularly and from a python virtual environment (used for testing)
• Created my own test script using a controller to display information from a selected table in the schedulerdb database

Issues

• Update: Resolved all below issues
  • I tried to install the same on our Seneca CDOT server: iraq.proximity.on.ca so that I could work off of the server but I ran into errors
  • Tried regular installation as well as using a python virtual environment
  • Serving content on iraq.proximity.on.ca

Project News

• November 18th, 2010
  • Updated some information on the page
  • Made substantial progress learning Pylons and MVC framework

• October 28th, 2010
• What has been done so far
  • Please see my blog at: http://andrewasdesigned.blogspot.com/2010/10/beginning-previously-we-had-conference.html
  • I have been moving forward and have installed BuildAPI locally on my machine to test it out
  • Still need time to finish reading on Pylons!

• October 22nd, 2010
  • New schedulerdb snapshot available.
• October 14, 2010:
  • Meeting of previously named "ScriptFactory" broke into three seperate projects: Mozharness, BuildAPI and Release Simple Bugs -Mustafa
  • We will follow up with Armen on details of the Release Simple Bugs project and more on BuildAPI

• Wednesday October 13, 2010:
  • The ScriptFactory group met today with Armen and Ctyler in a conference call
  • We will separate into two groups and tackle Mozharness, BuildAPI or Release Simple Bugs

Links

• Andrew Singh: http://andrewasdesigned.blogspot.com/
• Jing Yang: http://jyang42.blogspot.com/
• Armen Zambrano Gasparnian: http://armenzg.blogspot.com/
• Planet CDOT: http://zenit.senecac.on.ca/wiki/index.php/Planet_CDOT
• BuildBot Architecture: http://buildbot.net/buildbot/docs/0.8.1/images/overview.png
• Database Snapshots: http://people.mozilla.org/~catlee/schedulerdbschema.png and http://people.mozilla.org/~catlee/schema.png

Updated by Andrew Singh