Buildbot and EC2
Contents
Project Name
Buildbot and EC2
Project Description
The BuildBot is software for automatically building and testing other software. EC2 is Amazon's "Elastic Compute Cloud", a service that lets you run Virtual Machines on Amazon's computing grid using a web API. It would be awesome if BuildBot had an EC2 module, and could spawn Virtual Machines on-demand to perform tasks. Conveniently, BuildBot is written in Python, and there's a Python library for interfacing with EC2.
References: Amazon EC2 Basics For Python Programmers
Project Leader(s)
Project Contributor(s)
Project Details
Release 0.1
Goals
- Add in a new build step to Buildbot
- Allow Buildbot to connect with Amazon's EC2 service
Downloads
Note: You will need to install the boto library located here.
About
Basically, this first version attempts to connect to an EC2 account, and then display the connection information to show that it has connected.
It uses a library called "boto", and this library is used to connect to all of Amazon's Web Services
To use this in a Buildbot configuration file, simply call the connect_to_ec2() function, which takes in as paramaters, an Amazon id key, and an Amazon secret key.
Example:
connection = connect_to_ec2("73ghf9h38fh", "75vo87rVO*&V%O7o75v7i%i75I&5li7%i75")
Release 0.2
Goals
- Start New Instances
- Add new instances to build slave list
- Start a build on an EC2 instance build slave
Downloads
About
To use the EC2 service with Buildbot, there needs to be some way for Buildbot to dynamically generate a list of build slaves. These build slaves (or just slaves) are hosted as a virtual machine on the EC2 service.
Probably the best way to do this is to extend the BuildSlave class, and make a new class called EC2BuildSlave.
Theres a couple good reasons for extending this class. One big one I can think about is that the new class will have a different way of disconnecting, where it will have to send a command to the EC2 service to shut the instance down (thats if it's not being used by some other build master).
Making this new slave class will also be a much neater way to implement this new functionality, instead of having functions just randomly and haphazardly placed all over.
Lots more research and code reviewing is still on the horizon for this project, but this release should be ready in a months time.
Release 0.3
Goals
Downloads
About
Tasks
| Task | Details | Priority | Contributors | Status | Link(s) |
|---|---|---|---|---|---|
| Define new build step | Add a new build step to the build master config file. The build step will be in an external file and imported into the build master config file. | Done | Adam Delyea | Writing New Build Steps | |
| Connect To EC2 using Python | Utilize the EC2 python library to connect to an account on the EC2 network. Once connected, display some information about the users account to prove it's actually connecting. | Done | Adam Delyea | ||
| Start new instances | Allow Buildbot to spawn an instance on demand. This could possibly be a function called right before the build slave list is populated, and would pass back an array of build slaves. | High | Adam Delyea | In Progress, On Track | |
| Add new instances to build slave list | Allow newly created EC2 instances to be added to the build slave list. | High | Adam Delyea | In Progress, On Track | |
| Start a new build on an EC2 instance | Allow Buildbot to successfully start a build using a build slave on an EC2 instance | High | Adam Delyea | In Progress, On Track | |
| Load a script on startup in Linux | Make it possible to have a script load when Linux starts | Low | NEEDED! *** | Not Started | |
| Testing | People needed to download the revisions and test them to make sure it works in different environments. Note, this will require you to have your own EC2 account. | Low | NEEDED! *** | Not Started |
Contributions
Project Resources
- Mozilla Buildbot links: docs and code
- EC2 Firefox Extension
- Amazon-ec2-for-people-who-prefer-debian-and-python-over-fedora-and-java
Project News
2007-09-17: Project claimed!
2007-10-11: Found a python EC2 library that has all of the functions I will need for interacting with EC2. Just need to write a custom build step to run these library functions.
2007-10-12: Release v0.1 is complete and available for download.
2007-10-12: Release v0.2 loosely defined. More detail to follow.
