Fall 2009 SBR600 Weekly Schedule

Please note:

• The schedule here is tentative.
• Week-by-week details will be filled in as the course progresses.

Week 1 (September 8) - Introduction

Tuesday

• Welcome
• Introductions
• Intro to Build & Release
  • Brief overview of the process
    • Versioning & repository systems
    • Compilation
    • Testing
    • Packaging
    • Compositing
    • Release
    • Distribution
    • Mirroring
  • These steps vary according to the particular project/product. For example, when distributing software physically, "Release" means performing a RTM, where the final "gold disk" is sent to the duplicating house to be mass-produced; but when distributing software electronically, "Release" means sending the software to the online distribution system. The sequence of steps also varies between projects/products.
• Course Layout
  • Project-based course
  • Working with Open Source
  • Working with the Fedora Project
• Communication Tools
• Course Outline
• Visit the CDOT Area

Thursday

• Make
• Makefile Basics
  • Targets, Dependencies, and Commands
  • Implied rules (e.g., .o files)
  • Examples
• Building software from a source tarball using a makefile

Readings/Resources

• Two simple makefile examples

ToDo

Communication Lab: By Wednesday, September 9, Set up your accounts (wiki, IRC, FAS2).

• Create a blog post which will appear on the OpenSource@Seneca Planet, containing:
  • A portion of an IRC conversation you've had with someone on a Fedora or Seneca IRC channel.
  • A link to your User page on the Seneca wikis
  • A link to your User page on the Fedora wiki
  • Note: don't just dump this stuff in a blog post, add some introductory text as well!
• Add an entry to the Fall 2009 SBR600 Participants page

Register for:

• FSOSS
• FUDCon Toronto 2009

Lab 1: By Tuesday, September 15:

• Build 2 packages from Source
  • The NLED editor from http://cdot.senecac.on.ca
  • Any package that uses a configure script -- SourceForge might be a good place to look for such packages.
• Blog about the experience.

Week 2 (September 15) - Overview of the Build and Release Processs

Tuesday

• Discussion of issues related to building
  • Finding dependencies.
  • -jX flag to enable multiple jobs
  • Introduction to RPM
    • What is RPM?
    • Querying using the -q option

Thursday

• Telephone conference with Jesse Keating, Fedora Release Engineer
  • Audio recording of the call

ToDo

• Finish tasks from week 1 if not already completed.
  • Remember, marking in this course is done on the basis of blog posts which appear on the planet.
  • You should have two blog posts on the planet by now: One with a link to your Seneca and Fedora user pages plus a snippet of IRC conversation, and one with a reflection on your experience compiling software from source code.

Week 3 (September 22) - Creating RPM Packages I

RPM Packages

• Purpose
• What's in an RPM package file
  • Metadata
    • What the package provides
    • Dependencies
    • Packager, date, license, summary, description, ...
  • Digital signature
  • Software
  • Data
    • Fonts
    • Icons
    • Sample data
  • Documentation
  • Configuration files
  • Setup scripts
    • Pre-install
    • Post-install
    • Pre-uninstall
    • Post-uninstall
    • Triggers

The RPM Database

• Purpose of the database
• Querying the RPM database
  • rpm -q

Creating Packages

• Packaging scenarios
• Setting up a Packaging Environment
  • Needed packages
    • rpm-build
    • rpmdevtools
    • rpmlint
  • Setting up the RPM tree
    • run rpmdev-setuptree
• Taking a look at existing source RPMS (useful as examples)
  • Installing
    • yumdownloader --source nameofpackage
    • rpm -i nameofpackage.src.rpm
    • Source will be in ~/rpmbuild/SOURCES and specfile will be in ~/rpmbuild/SPECS
  • Examine the specfile
  • Rebuild on the local machine
    • rpmbuild --rebuild nameofpackage.src.rpm
  • Building from the spec file
    • cd ~/rpmbuild/SPECS; rpmbuild -ba nameofpackage.spec

Writing a specfile

• Run rpmdev-newspec packagename in ~/rpmbuild/SPECS
• Edit the skeleton specfile.
• Test it: rpmbuild -ba packagename.spec
  • If successful, output will be binary RPM(s) in ~/rpmbuild/RPMS and source RPM in ~/rpmbuild/SRPMS
    • Can install binary RPM with: rpm -i rpmname
  • If unsuccessful, read the error messages carefully.
• Check it with rpmlint: rpmlint packagename*
  • Remember to check the spec file as well as the binary and source RPMs.
  • Correct any errors found.

Layout of a specfile

• Basic Sections

1. preamble - basic metadata
2.  %prep - commands to prepare the package for building
3.  %build - commands to build the package
4.  %install - commands to install the built files
5.  %check - commands to check/test the built files (optional, often not included)
6.  %clean - commands to clean up the disk space
7.  %files - list of files to be included in the pacakge
8.  %changelog - record of the package's change-history

• Scriptlets
  •  %pre -- run before installation
  •  %post -- run after installation
  •  %preun -- run before uninstallation
  •  %postun -- run after uninstallation
    • Note that during upgrade, the installation of the new package is considered to happen before the removal of the old package.
• Macros
  •  %{_tmppath}
  •  %{buildroot}
  •  %{_bindir}
  •  %{_datadir}
  •  %{_mandir}
  •  %{_smp_flags}
  •  %setup
  •  %configure
  •  %makeinstall

Creating a Simple Package

• NLED
• Writing the specfile
• Testing the specfile
• Using rpmlint

Resources

• Fedora Package Maintainers page
  • Packaging How-To

See also "Fedora Linux" chapter 5 (see Seneca Library website > eBooks > View All > Safari > Fedora Linux).

ToDo

• Take the software you compiled last week and package it (not Nled!). Blog about the experience. Include a link to your source RPM (and optionally your binary RPM) from your blog. Please complete this by Monday, September 28.

Week 4 (September 29) - Creating RPM Packages II

Resources

• Using Koji

ToDo

• Test your RPM from last week with:
  • rpmlint
  • mock
  • koji

• Blog about your experience.

Week 5 (October 6) - Repositories/Distributing

Signing RPM packages

An RPM signature, like the digital signature used on many other software-signing systems, is a private key encryption of a checksum. RPM uses the GPG libraries for signing.

1. Create a GPG key: gpg --gen-key
2. Add the e-mail address associated with your gpg key to the %_gpg_name macro in ~/.rpmmacros -- the line will look like this: %_gpg_name "e-mail-address
3. Find (or make) some packages to put in your repository. Make sure that the epoch-version-release is higher than that of any package with the same name in the Fedora repositories.
4. Sign those packages with: rpm --addsign packagefile

Creating a YUM repository

A yum repository is just a directory of packages and some metadata.

1. Create a directory that can be served. The protocol used to serve that directory could be http, ftp, nfs, or something else (the files can be served by putting them on a DVD too!). For http, create the directory within /var/www/html
2. Put your signed packages in that directory.
3. Create the repository metadata for that directory: createrepo /name/of/directory

Notice that the repository metadata will be placed in a directory named repodata

Testing

1. Create a new repository file in /etc/yum.repos.d by copying and modifying an existing file in that directory. Keep gpgcheck=1 but comment out the gpgkey file.
2. Confirm that you cannot install from that repository using yum.
3. Uncomment the gpgkey line, and point it to a new file within /etc/pki/rpm-gpg/
4. Create that file by running (as your regular user): gpg --export --armour e-mail-address and saving the output
5. Confirm that you can now install from your repository. You should be asked whether you wish to import the key for your repo.

Guest Speakers: Ben Hearsum and Armen Zambrano

• Ben Hearsum and Armen Zambrano Gasparnian from the Mozilla build team will discuss what Build & Release means in the Mozilla context.

ToDo

1. Create an RPM package that will install your repository configuration file and the key.
2. Test it.
3. Blog about this lab, and include a link to your repository RPM package.

Week 6 (October 13) - Compositing

• Compositing (or Composing) is arranging media for distribution. These days, "Media" may be an image instead of physical media.

Creating a LiveCD/LiveDVD in Fedora

• Install the livecd-tools and example kickstart files: yum install livecd-tools spin-kickstarts

• Turn off SELinux temporarily: setenforce 0
• Run the livecd-creator with a specific kickstart file (this one uses an example from /usr/share/spin-kickstarts): livecd-creator --config=/usr/share/spin-kickstarts/fedora-livecd-desktop.ks --fslabel=Fedora-LiveCD --cache=/var/cache/live
• You should end up with an ISO image. You can test it with: qemu-kvm -m 512 -cdrom Fedora-LiveCD.iso

Creating Your Own LiveCD Image

• Create a modified kickstart file with these changes:
  • Replace the fedora-logos, fedora-release, and fedora-release-notes with the generic-logos, generic-release, and generic-release-notes packages.
  • Add your personal repository and the package which you created.
• Build the live disc and test it.

Resources

• Fedora Wiki:
  • LiveCDHowTo
  • Remix
  • Legal:Trademark Guidelines

ToDo

• Blog about your experiment finding the optimal %_smp_mflags value for a CDOT machine.
• Blog about the LiveCD image you created. Include a link to the kickstart file as well as to the ISO image itself.

Week 7 (October 20) - Server Farms I

Packaging ViewSource/DXR/Dehydra

Resources:

• This is what we're packaging (a web app): http://scotland.proximity.on.ca/dxr/viewsource/
• This is the script to build the modified GCC: http://scotland.proximity.on.ca/dxr/viewsource/build-tools.sh http://hg.mozilla.org/webtools/dxr/file/4e1eb442c826/viewsource/build-tools.sh
• This is DXR in its current form: http://dxr.proximity.on.ca/dxr/

ToDO

• Build the modified GCC before Thursday.
• On Thursday, benchmark the differences between the regular and modified GCC.

Study Week (October 27)

• FSOSS 2009
  • Please plan on attending at least the Friday sessions.
  • Register via the FSOSS web site (student registration is $15 in advance) -- or you can volunteer to get in for free.
• Toronto Open Source Week

Week 8 (November 3) - Server Farms II

DistCC

DistCC is a distributed C compiler. It provides a daemon (server, slave) which can be run on multiple machines, and a client (master) which is used to send jobs to daemons on other computers.

DistCC is installed as symbolic links named after the compilers present on the machine, targeting the 'distcc' binary. When invoked in this way, distcc will execute the compiler with the same name as the symbolic link, either on the local computer or on a slave.

Make is therefore unaware that it's running distcc -- it thinks it's running the regular C or C++ compiler (such as gcc or c++). In order to take advantage of the additional resources provided by the slave servers, it's necessary to increase make's -j value. If building with rpmbuild, this is done through the %_smp_mflags macro in ~/.rpmmacros

DistCC offers several different communication options, including a native (fairly insecure) protocol and ssh. The native protocol is often used despite being insecure because it is fast and most build farms are private networks that are not publicly accessible.

Resources

• http://distcc.org

Using DistCC

DistCC is packaged for Fedora (as "distcc" and "distcc-server").

Machine codes:

• [c] indicates client machine (where you are starting the build)
• [s] indicates server machine (running distccd)

Prerequisites for building with distcc:

• [c] distcc plus any requirements for a normal build (e.g., rpmbuild, cc, make, BuildRequires as specified in the spec file, etc)
• [s] distcc-server plus compilers (libraries etc. are not required)

Basic instructions on building with distcc:

1. [c] create ~/bin and populate it with symbolic links pointing to /usr/bin/distcc for each of the compilers your package might use -- gcc, cc, g++, c++
2. [c] add that directory to the start of your search path: PATH=${HOME}/bin:$PATH
3. [c] edit /etc/distcc/hosts to list the names of the hosts which will be servers (Recommendation: if your server and client are both on the GigE network -- scotland, ireland, china, india, or australia -- then append a "2" to the hostname to use the GigE interface (e.g., "scotland2")).
4. [s] on each server, allow access through the firewall for the distcc port (firewalls may be disabled on some of the machines alread)
5. [s] on each server, start distccd: distccd --daemon --allow IP-addresses-of-client-system
6. [c] start the build: time rpmbuild --rebuild package.src.rpm

ToDo

• Blog about the build times with and without distcc.

Week 9 (November 10) - Distributed Processing

Week 10 (November 17) - Virtualization

Week 11 (November 24) - Monitoring & Management

Week 12 (December 1) - Presentations

FUDCon (December 5-7)

• FUDCon Toronto 2009 consists of:
  • An unconference at SEQ on Saturday
    • A FUDPub social night at Dave & Busters on Saturday night
  • A hackfest at SEQ on Sunday
    • Skating at Nathan Phillips Square on Sunday night
  • A hackfest at TEL on Sunday
• Please plan on attending the Saturday Unconference.

Week 13 (December 8) - Wrap-Up

Exam Week (December 15)

• There is no exam in this course.