Difference between revisions of "Buildbot and EC2"

From CDOT Wiki
Jump to: navigation, search
m (→‎Tasks)
(→‎Releases)
 
(56 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
Implements [[Sample Project| iSampleProject]]
 +
 
== Project Name ==
 
== Project Name ==
  
Line 14: Line 16:
  
 
== Project Contributor(s) ==
 
== Project Contributor(s) ==
 +
[[User:dominic|Dominic Baranski]]
  
−
Dominic Baranski
+
[[User:Armenzg|Armen Zambrano]]
  
−
== Project Details ==
+
[[User:david.humphrey|Dave Humphrey]]
  
−
===      Release 0.1 ===
+
== Releases ==
  
−
==== Goals ====
+
You can see more information about individual releases by following the links below.
−
* Add in a new build step to Buildbot
 
−
* Allow Buildbot to connect with Amazon's EC2 service
 
  
−
==== Downloads ====
+
* [[Buildbot and EC2/0.1]]
  
−
    [http://matrix.senecac.on.ca/~awdelyea/EC2_build_step_nokeys.py EC2_build_step_nokeys.py]
+
* [[Buildbot and EC2/0.2]]
  
−
    Note: You will need to install the boto library located [http://boto.googlecode.com/files/boto-0.9a.tar.gz here].
+
* [[Buildbot and EC2/0.3]]
  
 +
* [[Buildbot and EC2/0.4]]
  
−
==== About ====
+
* [[Buildbot and EC2/0.5]]
  
−
Basically, this first version attempts to connect to an EC2 account, and then display the connection information to show that it has connected.
+
* [[Buildbot and EC2/0.6 | Buildbot and EC2/0.6/0.7]]
  
−
It uses a library called "boto", and this library is used to connect to all of [http://www.amazon.com/gp/browse.html?node=3435361 Amazon's Web Services]
+
* [[Buildbot and Testing|Tangential Testing on Buildbot Release / 0.8]]
  
−
To use this in a Buildbot configuration file, simply call the connect_to_ec2() function, which takes in as paramaters, an Amazon id key, and an Amazon secret key.
+
* [[Buildbot and EC2/0.9]]
−
 
 
−
Example:
 
−
<pre>
 
−
    connection = connect_to_ec2("73ghf9h38fh", "75vo87rVO*&V%O7o75v7i%i75I&5li7%i75")
 
−
</pre>
 
−
 
 
−
----
 
−
 
 
−
===      Release 0.2 ===
 
−
 
 
−
==== Goals ====
 
−
 
 
−
* Start New Instances
 
−
* Add new instances to build slave list
 
−
* Start a build on an EC2 instance build slave
 
−
 
 
−
==== Downloads ====
 
−
&nbsp;&nbsp;&nbsp;&nbsp;[http://zenit.senecac.on.ca/wiki/imgs/EC2BuildSlave_nokey.zip EC2BuildSlave.py]
 
−
&nbsp;&nbsp;&nbsp;&nbsp;*Just put this file into your buildbot installation folder so python can access it
 
−
<br/>&nbsp;&nbsp;&nbsp;&nbsp;[http://zenit.senecac.on.ca/wiki/imgs/Master.zip Master.cfg]
 
−
 
 
−
==== About ====
 
−
 
 
−
To use the EC2 service with Buildbot, there needs to be some way for Buildbot to dynamically generate a list of build slaves. These build slaves (or just slaves) are hosted as a virtual machine on the EC2 service.
 
−
 
 
−
This will work by extending the BuildSlave class to add in some new functionality for starting a new EC2 instance. So, every time you add a new slave to the slave list, another instance will be spwaned, so that you can spawn as many instances as you think you will need to complete a build.
 
−
 
 
−
This new BuildSlave class will start a new instance, which will then proceed to send the connection information of the master to the new instance. How exactly, I'm not exactly sure, because it takes a minute for the virtual instance to start up.
 
−
 
 
−
The rest basically takes care of itself.
 
−
 
 
−
Still have to think about how to turn the new EC2 instances off after the build is done.
 
−
 
 
−
====0.2.1====
 
−
=====About=====
 
−
I have made a slight modification to the EC2BuildSlave.<br/>
 
−
It now requires some additional input, because I had been hardcoding some values in, and thats sloppy, so I corrected it to allow you to put any variable parameters in the config file.<br/>
 
−
Basically, you will lay out a new EC2BuildSlave like this:
 
−
 
 
−
    from buildbot.EC2BuildSlave import EC2BuildSlave
 
−
    c['slaves'] = [EC2BuildSlave("bot1name", "bot1passwd", "public key", "private key", "dns", "instance-id")]
 
−
    //Where dns = www.whatever.com:8898, or 88.123.55.456:8898
 
−
    //and public/private key is your public/private key from Amazon
 
−
    //and instance-id is the instance you would like to launch ("ami-87d530ee")
 
−
 
 
−
=====Downloads=====
 
−
* [[http://zenit.senecac.on.ca/wiki/imgs/Buildbot_EC2.zip Patch]]
 
−
    The EC2BuildSlave.py goes into your python's framework folder for Buildbot (note: on my mac, this is<br/>  /Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/buildbot/)
 
−
 
 
−
----
 
−
 
 
−
===      Release 0.3 ===
 
−
 
 
−
==== Goals ====
 
−
 
 
−
* Build Mozilla using an EC2 Instance
 
−
* Complete Documentation
 
−
* Generic EC2 instance for Buildbot Slaves
 
−
* Startup Scripts for making your own EC2 instances
 
−
 
 
−
==== Downloads ====
 
−
 
 
−
==== About ====
 
−
 
 
−
The main goal of this release is to make this project as useable and accessible to as many people as possible. Right now, the EC2 instance that is set up will only work for connecting to one server, because the build masters IP address is hard coded into the instance. There is a way of sending the build masters IP address to the instance at run time, so the instance can use the build masters IP address in the startup script.
 
−
 
 
−
I would also like to get a script written for people to use in their startup to allow the creation of custom EC2 instances that will have the same Buildbot functionality.
 
−
 
 
−
Also, can't forget the whole point to this project. Building Mozilla on EC2 Using Buildbot. Builds work now, so I don't see a reason why I wouldn't be able to use EC2 to build mozilla, but I guess we'll see.
 
  
 
== Tasks ==
 
== Tasks ==
  
 
<table style="width: 100%;" class="standard-table" cellborder=1 border=1 cellpadding=3 cellspacing=0>
 
<table style="width: 100%;" class="standard-table" cellborder=1 border=1 cellpadding=3 cellspacing=0>
−
     <th style="width: 15%;">Task</th>
+
     <th style=" width: 17%;"><br />'''''Task'''''<br /><br /></th>
−
     <th style="width: 45%>Details</th>
+
     <th style=" width: 35%;"><br />'''''Details'''''<br /><br /></th>
−
     <th style="width: 10%;">Priority</th>
+
     <th style=" width: 5%;"><br />'''''Priority'''''<br /><br /></th>
−
     <th style="width: 10%;">Contributors</th>
+
     <th style=" width: 10%;"><br />'''''Contributors'''''<br /><br /></th>
−
     <th style="width: 10%;">Status</th>
+
     <th style=" width: 15%;"><br />'''''Status'''''<br /><br /></th>
−
     <th style="width: 10%;">Link(s)</th>
+
    <th style=" width: 4%;"><br />'''''Target'''''<br /><br /></th>
 +
    <th style=" width: 4%;"><br />'''''Completed'''''<br /><br /></th>
 +
     <th style=" width: 10%;"><br />'''''Link'''''(s)<br /><br /></th>
  
 
     <tr>
 
     <tr>
−
         <td>Define new build step</td>
+
         <td>0.1 Release</td>
−
         <td>Add a new build step to the build master config file. The build step will be in an external file and imported into the build master config file.</td>
+
         <td>Covering the basics</td>
 +
        <td  style="background-color: red; text-align: center; font-weight: bold; font-size: bigger">High</td>
 +
        <td>[[User:Awdelyea|Adam Delyea]]</td>
 
         <td style="background-color: green; text-align: center; font-weight: bold; font-size: bigger">Done</td>
 
         <td style="background-color: green; text-align: center; font-weight: bold; font-size: bigger">Done</td>
−
        <td>Adam Delyea</td>
 
 
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
−
         <td>[http://buildbot.net/repos/release/docs/buildbot.html#Writing-New-BuildSteps Writing New Build Steps]</td>
+
         <td>&nbsp;</td>
 +
        <td>&nbsp;</td>
 
     </tr>
 
     </tr>
  
 
     <tr>
 
     <tr>
−
         <td>Connect To EC2 using Python</td>
+
         <td>0.2 Release</td>
−
         <td>Utilize the EC2 python library to connect to an account on the EC2 network. Once connected, display some information about the users account to prove it's actually connecting.</td>
+
         <td></td>
 +
        <td  style="background-color: red; text-align: center; font-weight: bold; font-size: bigger">High</td>
 +
        <td>[[User:Awdelyea|Adam Delyea]]</td>
 
         <td style="background-color: green; text-align: center; font-weight: bold; font-size: bigger">Done</td>
 
         <td style="background-color: green; text-align: center; font-weight: bold; font-size: bigger">Done</td>
−
        <td>Adam Delyea</td>
 
 
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
−
         <td><ul><li>[http://code.google.com/p/boto/ Boto Amazon Web Services Library in Python]</li><li>[http://jimmyg.org/2007/09/01/amazon-ec2-for-people-who-prefer-debian-and-python-over-fedora-and-java/ EC2 hearts Python]</li></ul></td>
+
         <td>&nbsp;</td>
 +
        <td>&nbsp;</td>
 
     </tr>
 
     </tr>
  
 
     <tr>
 
     <tr>
−
         <td>Start new instances</td>
+
         <td>0.3 Release</td>
−
         <td>Allow Buildbot to spawn an instance on demand. This could possibly be a function called right before the build slave list is populated, and would pass back an array of build slaves.</td>
+
         <td></td>
 +
        <td  style="background-color: red; text-align: center; font-weight: bold; font-size: bigger">High</td>
 +
        <td>[[User:Awdelyea|Adam Delyea]]</td>
 
         <td style="background-color: green; text-align: center; font-weight: bold; font-size: bigger">Done</td>
 
         <td style="background-color: green; text-align: center; font-weight: bold; font-size: bigger">Done</td>
−
         <td>Adam Delyea</td>
+
         <td>&nbsp;</td>
−
         <td></td>
+
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
 
     </tr>
 
     </tr>
  
 
     <tr>
 
     <tr>
−
         <td><del>Add new instances to build slave list</del></td>
+
         <td>0.4 Release</td>
−
        <td>Allow newly created EC2 instances to be added to the build slave list.
 
−
<div style="background-color: grey; text-align: center;">
 
−
After looking more closely at how the buildslave is created, this goal doesn't make any sense, since this is not really how the buildslave list works.
 
−
            </div>
 
−
        </td>
 
−
        <td style="background-color: white; text-align: center; font-size: bigger">Cut</td>
 
−
        <td>Adam Delyea</td>
 
 
         <td></td>
 
         <td></td>
 +
        <td  style="background-color: red; text-align: center; font-weight: bold; font-size: bigger">High</td>
 +
        <td>[[User:Awdelyea|Adam Delyea]]</td>
 +
        <td style="background-color: green; text-align: center; font-weight: bold; font-size: bigger">Done</td>
 +
        <td>14/1/08</td>
 +
        <td>14/1/08</td>
 
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
 
     </tr>
 
     </tr>
  
 
     <tr>
 
     <tr>
−
         <td>Start a new build on an EC2 instance</td>
+
         <td>0.5 Release</td>
−
        <td>Allow Buildbot to successfully start a build using a build slave on an EC2 instance</td>
 
−
        <td style="background-color: green; text-align: center; font-weight: bold; font-size: bigger">Done</td>
 
−
        <td>Adam Delyea</td>
 
 
         <td></td>
 
         <td></td>
 +
        <td  style="background-color: red; text-align: center; font-weight: bold; font-size: bigger">High</td>
 +
        <td>[[User:Awdelyea|Adam Delyea]]</td>
 +
        <td style="background-color: SkyBlue; text-align: center; font-weight: bold; font-size: bigger">In Progress</td>
 +
        <td>3/1/08</td>
 
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
−
    </tr>
 
−
 
−
    <tr>
 
−
        <td>Load a script on startup in Linux</td>
 
−
        <td>Make it possible to have a script load when Linux starts</td>
 
−
        <td style="background-color: green; text-align: center; font-weight: bold; font-size: bigger">Done</td>
 
−
        <td>Adam Delyea</td>
 
−
        <td></td>
 
 
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
 
     </tr>
 
     </tr>
Line 180: Line 112:
 
         <td>Testing</td>
 
         <td>Testing</td>
 
         <td>People needed to download the revisions and test them to make sure it works in different environments. Note, this will require you to have your own EC2 account. </td>
 
         <td>People needed to download the revisions and test them to make sure it works in different environments. Note, this will require you to have your own EC2 account. </td>
−
         <td style="background-color: lightblue; text-align: center; font-weight: bold; font-size: bigger">Low</td>
+
         <td style="background-color: grey; text-align: center; font-weight: bold; font-size: bigger">Expired</td>
 
         <td style="background-color: gold; text-align: center; font-weight: bold; font-size: bigger">NEEDED! ***<br />[http://zenit.senecac.on.ca/wiki/index.php/User:dominic Dominic Baranski]</td>
 
         <td style="background-color: gold; text-align: center; font-weight: bold; font-size: bigger">NEEDED! ***<br />[http://zenit.senecac.on.ca/wiki/index.php/User:dominic Dominic Baranski]</td>
−
         <td></td>
+
         <td style="background-color: grey; text-align: center; font-weight: bold; font-size: bigger">Expired</td>
 +
        <td>&nbsp;</td>
 +
        <td>&nbsp;</td>
 
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
 
     </tr>
 
     </tr>
  
 
     <tr>
 
     <tr>
−
         <td>Different Builds</td>
+
         <td><del>Different Builds</del></td>
−
         <td>Need different projects to try and build, to make sure it works to build projects other than Mozilla</td>
+
         <td><del>Need different projects to try and build, to make sure it works to build projects other than Mozilla</del><br />This is no longer a priority. Building with Mozilla is the focus of this project. The EC2 functionality can be extracted and used to build any program / system.</td>
−
         <td style="background-color: lightblue; text-align: center; font-weight: bold; font-size: bigger">Low</td>
+
         <td style="background-color: grey; text-align: center; font-weight: bold; font-size: bigger">Expired</td>
 
         <td style="background-color: gold; text-align: center; font-weight: bold; font-size: bigger">NEEDED! ***<br />[http://zenit.senecac.on.ca/wiki/index.php/User:dominic Dominic Baranski]</td>
 
         <td style="background-color: gold; text-align: center; font-weight: bold; font-size: bigger">NEEDED! ***<br />[http://zenit.senecac.on.ca/wiki/index.php/User:dominic Dominic Baranski]</td>
−
         <td></td>
+
         <td style="background-color: grey; text-align: center; font-weight: bold; font-size: bigger">Expired</td>
 +
        <td>&nbsp;</td>
 
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
−
    </tr>
 
−
 
−
    <tr>
 
−
        <td>Build Mozilla using an EC2 Instance</td>
 
−
        <td>Allow Buildbot to successfully build mozilla using a build slave on an EC2 instance</td>
 
−
        <td style="background-color: orange; text-align: center; font-weight: bold; font-size: bigger">In Progress</td>
 
−
        <td>Adam Delyea</td>
 
−
        <td></td>
 
 
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
 
     </tr>
 
     </tr>
Line 207: Line 134:
 
         <td>Complete Documentation</td>
 
         <td>Complete Documentation</td>
 
         <td></td>
 
         <td></td>
−
         <td style="background-color: orange; text-align: center; font-weight: bold; font-size: bigger">In Progress</td>
+
         <td style="background-color: orange; text-align: center; font-weight: bold; font-size: bigger">Medium</td>
−
         <td>Adam Delyea</td>
+
         <td>[[User:Awdelyea|Adam Delyea]]</td>
−
         <td></td>
+
         <td style="background-color: green; text-align: center; font-weight: bold; font-size: bigger">In Progress</td>
 
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
−
    </tr>
 
−
 
−
    <tr>
 
−
        <td>Generic EC2 instance for Buildbot Slaves</td>
 
−
        <td>Making a generic EC2 Instance for Buildbot</td>
 
−
        <td style="background-color: orange; text-align: center; font-weight: bold; font-size: bigger">In Progress</td>
 
−
        <td>Adam Delyea</td>
 
−
        <td></td>
 
 
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
−
    </tr>
 
−
 
−
    <tr>
 
−
        <td>Startup Scripts for making your own EC2 instances</td>
 
−
        <td></td>
 
−
        <td style="background-color: orange; text-align: center; font-weight: bold; font-size: bigger">In Progress</td>
 
−
        <td>Adam Delyea</td>
 
−
        <td></td>
 
 
         <td>&nbsp;</td>
 
         <td>&nbsp;</td>
 
     </tr>
 
     </tr>
Line 235: Line 146:
  
 
<br/>
 
<br/>
−
<div style="background-color: gold; text-align: center; font-weight: bold;">*** Would you like to contribute? Well here's an excellent opportunity for you to do just that! Feel free to e-mail me at <awdelyea@learn.senecac.on.ca></div>
+
<div style="background-color: gold; text-align: center;font-weight: bold;">&nbsp;<br />*** Would you like to contribute? Well here's an excellent opportunity for you to do just that! Feel free to e-mail me at <awdelyea@gmail.com><br />&nbsp;</div><br/>
−
 
 
  
 
== Project Resources ==
 
== Project Resources ==
Line 245: Line 155:
  
 
== Project News ==
 
== Project News ==
 +
 +
For a less formal, and more up-to-date view on whats happening, see the [http://www.awdelyea.wordpress.com BLOG].
  
 
2007-09-17: Project claimed!
 
2007-09-17: Project claimed!
Line 257: Line 169:
  
 
2007-11-21: Release v0.2.1 is complete and available for download: now to include variable EC2 users, and instances, and DNS'
 
2007-11-21: Release v0.2.1 is complete and available for download: now to include variable EC2 users, and instances, and DNS'
 +
 +
2007-12-13: Release v0.3 is complete and available for download: easier to install and setup, can use to build projects now
 +
 +
2008-01-14: Release v0.4 Defined. In the process of getting accounts and machines for Try Server Farm.
 +
 +
'''''2008-02-01: Task List and Release News is being moved to individual Release Pages. See individual Releases for more information.'''''

Latest revision as of 10:23, 26 March 2008

Implements iSampleProject

Project Name

Buildbot and EC2

Project Description

The BuildBot is software for automatically building and testing other software. EC2 is Amazon's "Elastic Compute Cloud", a service that lets you run Virtual Machines on Amazon's computing grid using a web API. It would be awesome if BuildBot had an EC2 module, and could spawn Virtual Machines on-demand to perform tasks. Conveniently, BuildBot is written in Python, and there's a Python library for interfacing with EC2.

References: Amazon EC2 Basics For Python Programmers

Project Leader(s)

Adam Delyea

Project Contributor(s)

Dominic Baranski

Armen Zambrano

Dave Humphrey

Releases

You can see more information about individual releases by following the links below.

Tasks


Task
Details
Priority
Contributors
Status
Target
Completed
Link(s)
0.1 Release Covering the basics High Adam Delyea Done      
0.2 Release High Adam Delyea Done      
0.3 Release High Adam Delyea Done      
0.4 Release High Adam Delyea Done 14/1/08 14/1/08  
0.5 Release High Adam Delyea In Progress 3/1/08    
Testing People needed to download the revisions and test them to make sure it works in different environments. Note, this will require you to have your own EC2 account. Expired NEEDED! ***
Dominic Baranski		 Expired      
Different Builds Need different projects to try and build, to make sure it works to build projects other than Mozilla
This is no longer a priority. Building with Mozilla is the focus of this project. The EC2 functionality can be extracted and used to build any program / system.		 Expired NEEDED! ***
Dominic Baranski		 Expired      
Complete Documentation Medium Adam Delyea In Progress      

Contributions


 
*** Would you like to contribute? Well here's an excellent opportunity for you to do just that! Feel free to e-mail me at <awdelyea@gmail.com>
 

Project Resources

Project News

For a less formal, and more up-to-date view on whats happening, see the BLOG.

2007-09-17: Project claimed!

2007-10-11: Found a python EC2 library that has all of the functions I will need for interacting with EC2. Just need to write a custom build step to run these library functions.

2007-10-12: Release v0.1 is complete and available for download.

2007-10-12: Release v0.2 loosely defined. More detail to follow.

2007-11-17: Release v0.2 is complete and available for download: can start an instance and start an EC2BuildSlave

2007-11-21: Release v0.2.1 is complete and available for download: now to include variable EC2 users, and instances, and DNS'

2007-12-13: Release v0.3 is complete and available for download: easier to install and setup, can use to build projects now

2008-01-14: Release v0.4 Defined. In the process of getting accounts and machines for Try Server Farm.

2008-02-01: Task List and Release News is being moved to individual Release Pages. See individual Releases for more information.

Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=Buildbot_and_EC2&oldid=15292"