Difference between revisions of "Winter 2009 SBR600 Weekly Schedule"
Chris Tyler (talk | contribs) (→Week 8 (March 9) - Server Farms & Distributed Processing I) |
Chris Tyler (talk | contribs) |
||
(3 intermediate revisions by the same user not shown) | |||
Line 157: | Line 157: | ||
* Take the software you compiled last week and package it (not Nled!). Blog about the experience. Include a link to your source RPM (and optionally your binary RPM) from your blog. | * Take the software you compiled last week and package it (not Nled!). Blog about the experience. Include a link to your source RPM (and optionally your binary RPM) from your blog. | ||
− | = Week 4 (February 2) | + | = Week 4 (February 2) = |
* Free Software and Open Source | * Free Software and Open Source | ||
Line 215: | Line 215: | ||
# Test it with mock | # Test it with mock | ||
− | = Week 5 (February 9) | + | = Week 5 (February 9) = |
− | = Week 6 (February 16) | + | = Week 6 (February 16) = |
* We analyzed the [[Winter 2009 SBR 600 Fedora Mass Rebuild|mass rebuild]] results so far to find the slowest 15 packages: [[Winter 2009 SBR600 Packages of Interest]] | * We analyzed the [[Winter 2009 SBR 600 Fedora Mass Rebuild|mass rebuild]] results so far to find the slowest 15 packages: [[Winter 2009 SBR600 Packages of Interest]] | ||
Line 227: | Line 227: | ||
* Prepare one of the CDOT machines for a non-mock build of a slow package. See [[Winter_2009_SBR600_Packages_of_Interest#Preparing_to_test_distcc|Preparing to test distcc]] | * Prepare one of the CDOT machines for a non-mock build of a slow package. See [[Winter_2009_SBR600_Packages_of_Interest#Preparing_to_test_distcc|Preparing to test distcc]] | ||
− | = Week 7 (February 23) | + | = Week 7 (February 23) = |
= Study Week (March 2) = | = Study Week (March 2) = | ||
− | = Week 8 (March 9) | + | = Week 8 (March 9) = |
Machine codes: | Machine codes: | ||
Line 243: | Line 243: | ||
# [c] add that directory to the start of your search path: <code>PATH=${HOME}/bin:$PATH</code> | # [c] add that directory to the start of your search path: <code>PATH=${HOME}/bin:$PATH</code> | ||
# [c] edit /etc/distcc/hosts to list the names of the hosts which will be servers ('''Recommendation:''' if your server and client are both on the GigE network -- scotland, ireland, china, india, or australia -- then append a "2" to the hostname to use the GigE interface (e.g., "scotland2")). | # [c] edit /etc/distcc/hosts to list the names of the hosts which will be servers ('''Recommendation:''' if your server and client are both on the GigE network -- scotland, ireland, china, india, or australia -- then append a "2" to the hostname to use the GigE interface (e.g., "scotland2")). | ||
− | # [s] on each server, start distccd: | + | # [s] on each server, allow access through the firewall for the distcc port (firewalls may be disabled on some of the machines alread) |
+ | # [s] on each server, start distccd: <code>distccd --daemon --allow ''IP-addresses-of-client-system''</code> | ||
# [c] start the build: <code>time rpmbuild --rebuild ''package''.src.rpm</code> | # [c] start the build: <code>time rpmbuild --rebuild ''package''.src.rpm</code> | ||
''ToDo:'' | ''ToDo:'' | ||
− | * | + | * Blog about the build times with and without distcc. |
− | * Blog | + | * Blog an analysis of the /var/f10source/buildall.log file on Scotland. |
− | = Week 9 ( | + | = Week 9 = |
− | = Week 10 (March 23) | + | |
− | = Week 11 (March 30) | + | == Signing RPM packages == |
− | = Week 12 (April 6) - | + | |
− | = Week 13 (April 13) | + | An RPM signature, like the digital signature used on many other software-signing systems, is a private key encryption of a checksum. RPM uses the GPG libraries for signing. |
+ | |||
+ | # Create a GPG key: <code>gpg --gen-key</code> | ||
+ | # Add the e-mail address associated with your gpg key to the <code>%_gpg_name</code> macro in <code>~/.rpmmacros</code> -- the line will look like this: <code>%_gpg_name "<i>e-mail-address</i></code> | ||
+ | # Find (or make) some packages to put in your repository. Make sure that the epoch-version-release is higher than that of any package with the same name in the Fedora repositories. | ||
+ | # Sign those packages with: <code>rpm --addsign <i>packagefile</i></code> | ||
+ | |||
+ | == Creating a YUM repository == | ||
+ | |||
+ | A yum repository is just a directory of packages and some metadata. | ||
+ | |||
+ | # Create a directory that can be served. The protocol used to serve that directory could be http, ftp, nfs, or something else (the files can be served by putting them on a DVD too!). For http, create the directory within <code>/var/www/html</code> | ||
+ | # Put your signed packages in that directory. | ||
+ | # Create the repository metadata for that directory: <code>createrepo <i>/name/of/directory</i></code> | ||
+ | |||
+ | Notice that the repository metadata will be placed in a directory named <code>repodata</code> | ||
+ | |||
+ | == Testing == | ||
+ | |||
+ | # Create a new repository file in <code>/etc/yum.repos.d</code> by copying and modifying an existing file in that directory. Keep <code>gpgcheck=1</code> but comment out the <code>gpgkey</code> file. | ||
+ | # Confirm that you cannot install from that repository using yum. | ||
+ | # Uncomment the <code>gpgkey</code> line, and point it to a new file within <code>/etc/pki/rpm-gpg/</code> | ||
+ | # Create that file by running (as your regular user): <code>gpg --export --armour <i>e-mail-address</i></code> and saving the output | ||
+ | # Confirm that you can now install from your repository. You should be asked whether you wish to import the key for your repo. | ||
+ | |||
+ | ''ToDo:'' | ||
+ | |||
+ | # Create an RPM package that will install your repository configuration file and the key. | ||
+ | # Test it. | ||
+ | # Blog about this lab, and include a link to your repository RPM package. | ||
+ | |||
+ | = Week 10 (March 23) = | ||
+ | = Week 11 (March 30) = | ||
+ | = Week 12 (April 6) = | ||
+ | |||
+ | == Using git == | ||
+ | === Resources === | ||
+ | * git web site: http://git-scm.com/ | ||
+ | * git tutorial: <code>man gittutorial</code> or http://www.kernel.org/pub/software/scm/git/docs/gittutorial.html | ||
+ | * git documentation: http://git-scm.com/documentation | ||
+ | * Fedora packages: <code>git, gitk</code> | ||
+ | |||
+ | === Lab Steps === | ||
+ | # Select a CDOT system that no one else is using. | ||
+ | # Make a backup archive of the /etc directory: <code>tar cvzf ~/etc-backup.tgz /etc</code> | ||
+ | # Put the <code>/etc</code> directory under git control and do an initial commit. | ||
+ | # Experiment with making changes in /etc and committing those changes. | ||
+ | # Create an experimental branch and checkout that branch. | ||
+ | # Make radical changes in /etc. Experiment with switching back and forth between the master and experimental branches. | ||
+ | # When you're done, checkout the master branch and delete the experimental branch. | ||
+ | |||
+ | ''ToDo''' | ||
+ | * Blog about your using git -- what commands you used, what worked well, what was confusing, what you like/don't like. | ||
+ | |||
+ | = Week 13 (April 13) = | ||
= Exam Week (April 20) = | = Exam Week (April 20) = |
Latest revision as of 10:44, 9 April 2009
Please note:
- The schedule here is tentative.
- Week-by-week details will be filled in as the course progresses.
Contents
- 1 Week 1 (January 12) - Introduction
- 2 Week 2 (January 19) - Overview of the Build and Release Processs
- 3 Week 3 (January 26) - Creating RPM Packages
- 4 Week 4 (February 2)
- 5 Week 5 (February 9)
- 6 Week 6 (February 16)
- 7 Week 7 (February 23)
- 8 Study Week (March 2)
- 9 Week 8 (March 9)
- 10 Week 9
- 11 Week 10 (March 23)
- 12 Week 11 (March 30)
- 13 Week 12 (April 6)
- 14 Week 13 (April 13)
- 15 Exam Week (April 20)
Week 1 (January 12) - Introduction
Tuesday
- Welcome
- Introductions
- Intro to Build & Release
- Brief overview of the process
- Versioning & repository systems
- Compilation
- Testing
- Packaging
- Compositing
- Release
- Distribution
- Mirroring
- These steps vary according to the particular project/product. For example, when distributing software physically, "Release" means performing a RTM, where the final "gold disk" is sent to the duplicating house to be mass-produced; but when distributing software electronically, "Release" means sending the software to the online distribution system. The sequence of steps also varies between projects/products.
- Brief overview of the process
- Course Layout
- Project-based course
- Working with Open Source
- Working with the Fedora Project
- Communication Tools
- Course Outline
- Visit the CDOT Area
Thursday
- Makefile Basics
- Targets, Dependencies, and Commands
- Implied rules (e.g., .o files)
- Examples
Readings/Resources
- Two simple makefile examples
ToDo:
Communication Lab: By Thursday, January 15, Set up your accounts (wiki, IRC, FAS2).
- Create a blog post which will appear on the OpenSource@Seneca Planet, containing:
- A portion of an IRC conversation you've had with someone on a Fedora or Seneca IRC channel.
- A link to your User page on the Seneca wiki
- A link to your User page on the Fedora wiki
- Add an entry to the Winter 2009 SBR600 Participants page
Lab 1: By Wednesday, January 21:
- Build 2 packages from Source
- The NLED editor from http://cdot.senecac.on.ca
- Any package that uses autoconf -- SourceForge might be a good place to look for such packages.
- Blog about the experience.
Week 2 (January 19) - Overview of the Build and Release Processs
(Classes cancelled due to professor's illness)
- Complete ToDo items from week 1 if not already done.
Week 3 (January 26) - Creating RPM Packages
RPM Packages
- Purpose
- What's in an RPM package file
- Metadata
- What the package provides
- Dependencies
- Packager, date, license, summary, description, ...
- Digital signature
- Software
- Data
- Fonts
- Icons
- Sample data
- Documentation
- Configuration files
- Setup scripts
- Pre-install
- Post-install
- Pre-uninstall
- Post-uninstall
- Triggers
- Metadata
The RPM Database
- Purpose of the database
- Querying the RPM database
- rpm -q
Creating Packages
- Packaging scenarios
- Setting up a Packaging Environment
- Needed packages
- rpm-build
- rpmdevtools
- rpmlint
- Setting up the RPM tree
- run
rpmdev-setuptree
- run
- Needed packages
- Taking a look at existing source RPMS (useful as examples)
- Installing
-
yumdownloader --source nameofpackage
-
rpm -i nameofpackage.src.rpm
- Source will be in ~/rpmbuild/SOURCES and specfile will be in ~/rpmbuild/SPECS
-
- Examine the specfile
- Rebuild on the local machine
-
rpmbuild --rebuild nameofpackage.src.rpm
-
- Building from the spec file
-
cd ~/rpmbuild/SPECS; rpmbuild -ba nameofpackage.spec
-
- Installing
Writing a specfile
- Run
rpmdev-newspec packagename
in ~/rpmbuild/SPECS - Edit the skeleton specfile.
- Test it:
rpmbuild -ba packagename.spec
Layout of a specfile
- Basic Sections
- preamble - basic metadata
- %prep - commands to prepare the package for building
- %build - commands to build the package
- %install - commands to install the built files
- %check - commands to check/test the built files (optional, often not included)
- %clean - commands to clean up the disk space
- %files - list of files to be included in the pacakge
- %changelog - record of the package's change-history
- Scriptlets
- %pre
- %post
- %preun
- %postun
- Macros
- %{_tmppath}
- %{buildroot}
- %{_bindir}
- %{_datadir}
- %{_mandir}
- %{_smp_flags}
- %setup
- %configure
- %makeinstall
Creating a Simple Package
- NLED
- Writing the specfile
- Testing the specfile
- Using rpmlint
Resources
See also "Fedora Linux" chapter 5 (see Seneca Library website > eBooks > View All > Safari > Fedora Linux).
TODO:
- Take the software you compiled last week and package it (not Nled!). Blog about the experience. Include a link to your source RPM (and optionally your binary RPM) from your blog.
Week 4 (February 2)
- Free Software and Open Source
- Origins
- Early computing
- Public domain software
- Richard Stallman and the Free Software Foundation
- The Open Source Initiative
- Current state
- Becoming a dominant model in the industry
- Prominent projects: Linux, Apache, OpenOffice.org, Eclipse, ...
- What Open Source Means
- Working in Public
- Collaborating
- Thinking and working globally
- Reusing code, thinking smart, being nimble
- Open Source business models
- Support
- Customization
- Sponsorship
- Origins
- Release and Build on Open Source
- Looks like release and build in proprietary projects
- More developers
- More casual contributors
- Internet instead of private network
- Security issues
- The Fedora Project
- Red Hat Linux
- Fedora Core and Fedora Extras
- Fedora
- "Dedicated to the rapid advancement of Open Source"
- The 4 F's
- Features
- Friends
- Freedom
- First
- The Structure of Fedora
- Communication tools
- Funding
- How software gets into Fedora
- Packagers
- Sponsors
- Infrastructure
- Face to face
- FUDCons
- FADs
- Example from class: md5deep-3.2-1.fc10.src.rpm
ToDo:
- Create an RPM.
- Test it with rpmlint (spec, source RPM, binary RPM)
- Test it with mock
Week 5 (February 9)
Week 6 (February 16)
- We analyzed the mass rebuild results so far to find the slowest 15 packages: Winter 2009 SBR600 Packages of Interest
- Questions:
- Why are 2 books in the top-4 slowest packages to build?
- Why are there so many Java packages in the top 15? Are they big, or slow to compile, or ?
- How much can we speed up packages in the top 15 using distcc?
ToDo:
- Prepare one of the CDOT machines for a non-mock build of a slow package. See Preparing to test distcc
Week 7 (February 23)
Study Week (March 2)
Week 8 (March 9)
Machine codes:
- [c] indicates client machine (where you are starting the build)
- [s] indicates server machine (running distccd)
Prerequisites for building with distcc:
- [c] distcc plus any requirements for a normal build (e.g., rpmbuild, cc, make, BuildRequires as specified in the spec file, etc)
- [s] distcc-server plus compilers (libraries etc. are not required)
Basic instructions on building with distcc:
- [c] create ~/bin and populate it with symbolic links pointing to /usr/bin/distcc for each of the compilers your package might use -- gcc, cc, g++, c++
- [c] add that directory to the start of your search path:
PATH=${HOME}/bin:$PATH
- [c] edit /etc/distcc/hosts to list the names of the hosts which will be servers (Recommendation: if your server and client are both on the GigE network -- scotland, ireland, china, india, or australia -- then append a "2" to the hostname to use the GigE interface (e.g., "scotland2")).
- [s] on each server, allow access through the firewall for the distcc port (firewalls may be disabled on some of the machines alread)
- [s] on each server, start distccd:
distccd --daemon --allow IP-addresses-of-client-system
- [c] start the build:
time rpmbuild --rebuild package.src.rpm
ToDo:
- Blog about the build times with and without distcc.
- Blog an analysis of the /var/f10source/buildall.log file on Scotland.
Week 9
Signing RPM packages
An RPM signature, like the digital signature used on many other software-signing systems, is a private key encryption of a checksum. RPM uses the GPG libraries for signing.
- Create a GPG key:
gpg --gen-key
- Add the e-mail address associated with your gpg key to the
%_gpg_name
macro in~/.rpmmacros
-- the line will look like this:%_gpg_name "e-mail-address
- Find (or make) some packages to put in your repository. Make sure that the epoch-version-release is higher than that of any package with the same name in the Fedora repositories.
- Sign those packages with:
rpm --addsign packagefile
Creating a YUM repository
A yum repository is just a directory of packages and some metadata.
- Create a directory that can be served. The protocol used to serve that directory could be http, ftp, nfs, or something else (the files can be served by putting them on a DVD too!). For http, create the directory within
/var/www/html
- Put your signed packages in that directory.
- Create the repository metadata for that directory:
createrepo /name/of/directory
Notice that the repository metadata will be placed in a directory named repodata
Testing
- Create a new repository file in
/etc/yum.repos.d
by copying and modifying an existing file in that directory. Keepgpgcheck=1
but comment out thegpgkey
file. - Confirm that you cannot install from that repository using yum.
- Uncomment the
gpgkey
line, and point it to a new file within/etc/pki/rpm-gpg/
- Create that file by running (as your regular user):
gpg --export --armour e-mail-address
and saving the output - Confirm that you can now install from your repository. You should be asked whether you wish to import the key for your repo.
ToDo:
- Create an RPM package that will install your repository configuration file and the key.
- Test it.
- Blog about this lab, and include a link to your repository RPM package.
Week 10 (March 23)
Week 11 (March 30)
Week 12 (April 6)
Using git
Resources
- git web site: http://git-scm.com/
- git tutorial:
man gittutorial
or http://www.kernel.org/pub/software/scm/git/docs/gittutorial.html - git documentation: http://git-scm.com/documentation
- Fedora packages:
git, gitk
Lab Steps
- Select a CDOT system that no one else is using.
- Make a backup archive of the /etc directory:
tar cvzf ~/etc-backup.tgz /etc
- Put the
/etc
directory under git control and do an initial commit. - Experiment with making changes in /etc and committing those changes.
- Create an experimental branch and checkout that branch.
- Make radical changes in /etc. Experiment with switching back and forth between the master and experimental branches.
- When you're done, checkout the master branch and delete the experimental branch.
ToDo'
- Blog about your using git -- what commands you used, what worked well, what was confusing, what you like/don't like.