Difference between revisions of "Pidora SOP"
(→Vfat Reformating Script) |
(→Preparing Pidora Noobs distribution) |
||
Line 527: | Line 527: | ||
Pidora 2014 Noobs Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Noobs-Script | Pidora 2014 Noobs Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Noobs-Script | ||
+ | |||
+ | *Note this script uses PV (pipe viewer), if it's not already installed on the system please install it. | ||
Example Usage | Example Usage |
Revision as of 16:53, 18 August 2014
Contents
- 1 Intro
- 2 Building Packages
- 3 Sign, Mash, and Rsync Together!
- 4 Signing Packages
- 5 Creating Repositories
- 6 Composing Images
- 6.1 Example Livemedia-Creator Command
- 6.2 Example Kickstart File
- 6.3 Package List File
- 6.4 Example Post Script Execution
- 6.5 Preparing The Final Image
- 6.6 Vfat Reformating Script
- 6.7 Checking The Partition FStype
- 6.8 Shrinking The Image
- 6.9 Preparing Pidora Noobs distribution
- 6.10 Testing Pidora Noobs
Intro
This page covers how one can manage the various needed parts of a complete release process. This includes the other SOP pages containing file configurations and/or quick command line references for the following procedures:
- Building packages
- Signing packages
- Creating repositories
- Composing images
Building Packages
Koji-Follow Source Code and Configuration SOPs
How-To-Run
- ssh <your_username>@japan
- sudo kojiadmin
- screen -xr follow
- python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log
Post Commands
- Repo analysis:
repoclosure --arch=armv6hl --arch=noarch --repofrompath=v6,http://japan.proximity.on.ca/repos/f20-build/latest/armv6hl/ -r v6 | tee /tmp/reporeport.log
Sign, Mash, and Rsync Together!
Prerequisites
- Must setup passwordless logins from root@japan to all users in the script
- Must have a working mash configuration: See mash below
- Must have a configured sigul user with access to the signing key
- Make sure that any externally called scripts are in the correct locations
Running pidora-update.py
- ssh japan
- run as root: pidora-smr
- script location /root/pidora-update/pidora-update.py
Usage: pidora-smr [options] Options: -h, --help show this help message and exit -i, --info check machine status and configuration -a, --all sign, mash, rsync -s, --sign sign all packages in listed tag -m, --mash start a mash run -r, --rsync perform a rsync of the mash repos -f, --force can force some options -l, --list-unsigned list unsigned rpms --pidora=PIDORA specify version of pidora = 18, 19 --auto enables logging and emails logs --koji-tag=KOJITAG specify the koji tag to sign --email=email@senecacollege.ca specify the email to send logs to --sigul-user=USER specify the user for sigul --sigul-host=HOSTNAME specify the host for sigul --mash-user=USER specify the user for mash --mash-host=HOSTNAME specify the host for mash --rsync-user=USER specify the user for rsync --rsync-host=HOSTNAME specify the host for rsync --log-dir=/var/log/pidora-smr/ specify a logging directory --log-file=output specify a log file name
- All the configurations in the script can be changed with the command line options
- Can change the user to access: sigul, mash, and rsync
- Can change the host which runs: sigul, mash, and rsync
- Can change which koji tag will be used for signing
- Can change the version of pidora to ssign, mash, rsync
Examples runs
- Check which hosts can connect successfully
- Also shows default configurations
pidora-smr --info [Connection] sigulhost = england.proximity.on.ca siguluser = user mashhost = japan.proximity.on.ca mashuser = user rsynchost = pidora.proximity.on.ca rsyncuser = user [General] auto = False mashdir = /usr/local/bin/mash-pidora kojitags = ['f18-updates', 'f18-rpfr-updates', 'f18-updates-testing', 'f18-rpfr-updates-testing'] email = email@senecacollege.ca logdir = /var/log/pidora-smr/ logfile = /var/log/pidora-smr/output [Hosts] working hosts: ['japan.proximity.on.ca', 'england.proximity.on.ca', 'pidora.proximity.on.ca'] failed hosts: []
- Get a list of unsigned packages
- Can check a single tag with --koji-tag=<tag-name>
pidora-smr --list-unsigned --pidora 18 pidora-smr --list-unsigned --pidora 19 pidora-smr --list-unsigned --pidora 20
- Using the default configuration, start a sign, mash, rsync
pidora-smr --all --pidora 20
- Change sigul user
pidora-smr --all --sigul-user=oatley --pidora 20
- Run sign, mash, or sync separately, instead of together with the --all option
pidora-smr --sign --pidora 20 pidora-smr --mash --pidora 20 pidora-smr --rsync --pidora 20
- Changing the configurations with command line options
- Sign using a different user, on a single koji tag
pidora-smr --sign --koji-tag=f18-updates-testing --pidora 20
Changing Default Configurations
- Can view all default configurations with the following command:
pidora-smr --info
- Modify the script with a text editor
- Change any of the variables listed below:
sigulhost = england.proximity.on.ca siguluser = user mashhost = japan.proximity.on.ca mashuser = user rsynchost = pidora.proximity.on.ca rsyncuser = user [General] auto = False mashdir = /usr/local/bin/mash-pidora kojitags = ['f18-updates', 'f18-rpfr-updates', 'f18-updates-testing', 'f18-rpfr-updates-testing'] email = email@senecacollege.ca
Source Code
Signing Packages
Sigul Configuration SOPs
- http://zenit.senecac.on.ca/wiki/index.php/Sigul_Signing_Server_Setup
- http://zenit.senecac.on.ca/wiki/index.php/Cdot_sigul_sop
- http://fedoraproject.org/wiki/Create_release_signing_key
Creating Repositories
Mash File Configuration Details
/etc/mash/pidora-18.mash
# mash config file [pidora-18] rpm_path = %(arch)s/os/Packages repodata_path = %(arch)s/os/ source_path = source/SRPMS debuginfo = True multilib = False multilib_method = devel tag = f18-rpfr inherit = True # Unsigned is temporary strict_keys = True #keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9 #keys = CE8D31E3 keys = f1590cd5 repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/ repoviewtitle = "Fedora Branched - %(arch)s" arches = armhfp # armv6 doesn't have horsepower to make delta useful delta = False # Change distro_tags as fedora-release version gets bumped # The cpe URI should be fixed up distro_tags = cpe:/o:fedoraproject:fedora:18 rawhide hash_packages = True
/etc/mash/pidora-18-updates.mash
[pidora-18-updates] rpm_path = %(arch)s/ repodata_path = %(arch)s/ source_path = SRPMS debuginfo = True multilib = False multilib_method = devel tag = f18-updates inherit = False # Unsigned is temporary strict_keys = True #keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9 #keys = CE8D31E3 keys = f1590cd5 repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/ repoviewtitle = "Fedora Branched - %(arch)s" arches = armhfp # armv6 doesn't have horsepower to make delta useful delta = False # Change distro_tags as fedora-release version gets bumped # The cpe URI should be fixed up distro_tags = cpe:/o:fedoraproject:fedora:18 update hash_packages = True
/etc/mash/pidora-18-updates-testing.mash
[pidora-18-updates-testing] rpm_path = %(arch)s/ repodata_path = %(arch)s/ source_path = SRPMS debuginfo = True multilib = False multilib_method = devel tag = f18-updates-testing inherit = False # Unsigned is temporary strict_keys = True #keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9 #keys = CE8D31E3 keys = f1590cd5 repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/ repoviewtitle = "Fedora Branched - %(arch)s" arches = armhfp # armv6 doesn't have horsepower to make delta useful delta = False # Change distro_tags as fedora-release version gets bumped # The cpe URI should be fixed up distro_tags = cpe:/o:fedoraproject:fedora:18 update hash_packages = True
/etc/mash/pidora-18-rpfr-updates.mash
[pidora-18-rpfr-updates] rpm_path = %(arch)s/ repodata_path = %(arch)s/ source_path = SRPMS debuginfo = True multilib = False multilib_method = devel tag = f18-rpfr-updates inherit = False # Unsigned is temporary strict_keys = True #keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9 #keys = CE8D31E3 keys = f1590cd5 repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/ repoviewtitle = "Fedora Branched - %(arch)s" arches = armhfp # armv6 doesn't have horsepower to make delta useful delta = False # Change distro_tags as fedora-release version gets bumped # The cpe URI should be fixed up distro_tags = cpe:/o:fedoraproject:fedora:18 update hash_packages = True
/etc/mash/pidora-18-rpfr-updates-testing.mash
[pidora-18-rpfr-updates-testing] rpm_path = %(arch)s/ repodata_path = %(arch)s/ source_path = SRPMS debuginfo = True multilib = False multilib_method = devel tag = f18-rpfr-updates-testing inherit = False # Unsigned is temporary strict_keys = True #keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9 #keys = CE8D31E3 keys = f1590cd5 repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/ repoviewtitle = "Fedora Branched - %(arch)s" arches = armhfp # armv6 doesn't have horsepower to make delta useful delta = False # Change distro_tags as fedora-release version gets bumped # The cpe URI should be fixed up distro_tags = cpe:/o:fedoraproject:fedora:18 update hash_packages = True
/etc/mash/mash.conf
[defaults] configdir = /etc/mash buildhost = http://japan.proximity.on.ca/kojihub repodir = file:///mnt/koji use_sqlite = True use_repoview = False
/usr/local/bin/mashrun-pidora-18
#!/bin/bash # # mashrun-pidora18 :: Compose a Pidora 18 repository # DISTNAME=pidora RELEASE=18 # Default mash output dir is unique by dates (YYYYMMDD) BASEDIR=/mnt/koji/mash MASHDIR=${BASEDIR}/${DISTNAME}-${RELEASE}-$(date +%Y%m%d) # If there is already a compose output directory for today, # append -HHMMSS if [ -d ${MASHDIR} ] # If YYYYMMDD exists, add -HHMMSS then MASHDIR=${MASHDIR}-$(date +%H%M%S) fi mkdir -p ${MASHDIR} { exec 2>&1 echo "Composing to ${MASHDIR}..." time mash -o ${MASHDIR} ${DISTNAME}-${RELEASE} -f /usr/local/mash/comps-${DISTNAME}-${RELEASE}.xml.gz rm ${BASEDIR}/${DISTNAME}-${RELEASE}-latest ln -s ${MASHDIR} ${BASEDIR}/${DISTNAME}-${RELEASE}-latest echo echo "Warnings about unsigned packages are temporarily suppressed from email." echo "Refer to ${MASHDIR}/mash.log for warnings." }| cat -v | tee ${MASHDIR}/mash.log | fgrep -v '(signed with no key)' | mail -s "${DISTNAME}-${RELEASE} Mash Run" mashrun-alert # The 'cat -v' above is there to prevent an unprintable code from throwing off the # mail command.
Mash Command Line Execution
- /usr/local/bin/mashrun-pidora-18
Rsync to Pidora 18 Repositories
- ssh pidorapr@pidora.ca
- ~/bin/rsync-japan
Composing Images
Before you can attempt to run a compose you should check to make sure that the following packages are installed:
- anaconda
- lorax
Because livemedia-tools is not stable software for creating pidora images we have a modified version of livemedia-creator that works well however this software is currently out of date, so we have decided to keep using this version of livemedia to produce future images. You can accessing our custom version of livemedia which is located on the host machine cal-7-2 within a chroot log onto the host machine iraq.proximity.on.ca then ssh into the compose node host machine cal-7-2.
- ssh user@iraq.proximity.on.ca
- ssh root@cal-7-2
- Note if you restart or the node is shutdown you will need to bind mount /proc, /dev, /sys to f17v6/proc, f17v6/dev, f17v6/sys with the following command before you chroot.
- mount -o bind /proc/ f17v6/proc
- mount -o bind /dev/ f17v6/dev
- mount -o bind /sys/ f17v6/sys
chroot into the armv6hl directory on the arm device.
- chroot /root/f17v6
cd into the livemedia directory or where ever your kickstart file is located.
- cd /livemedia/f20
Run the provided Livemedia-Creator command provided below to start the compose process.
Example Livemedia-Creator Command
- livemedia-creator --make-disk --no-virt --image-only --keep-image --ks=pidora-f20.ks
Estimated time for compilation (15mins)
Example Kickstart File
Pidora 18 kickstart: http://zenit.senecac.on.ca/wiki/index.php/Pidora_kickstart Pidora 2014 kickstart: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-kickStart
Package List File
Pidora 2014 Package List kickstart: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Packagelist-kickStart
- command details
-h, --help For more info on commands
--make-disk Build a disk image
--no-virt Use Anaconda's image install instead of virt-install
--image-only Exit after creating disk image.
--ks KS Kickstart file defining the install.
Example Post Script Execution
You may have more than one %post section, which can be useful for cases where some post-installation scripts need to be run in the chroot and others that need access outside the chroot.
%post section is required to be closed with a corresponding %end
Post Section Options
- command details
--nochroot
Allows you to specify commands that you would like to run outside of the chroot environment.
--interpreter /usr/bin/python
Allows you to specify a different scripting language, such as Python. Replace /usr/bin/python with the scripting language of your choice.
--erroronfail
If the post-installation script fails, this option will cause an error dialog to be displayed and will halt installation. The error message will direct you to where the cause of the failure is logged.
--log=
Example Post Installation Script "Creating/Modifying the fstab"
cat << EOF > /etc/fstab
LABEL="rootfs" / ext4 defaults,noatime 1 1
LABEL="boot" /boot vfat noauto,comment=systemd.automount 1 2
EOF
Preparing The Final Image
When you have a successful disk image ready the boot partition needs to be reformatted with a vfat filesystem type, the Raspberry Pi will only boot with a vfat boot partition. This can be done manually or by using the script provided below. The final image's rootfs also needs to be resized to minimize the size of file system, there is also a script provided to do this.
- all the preparation scripts are located in /var/tmp/ directory along with the disk*.img file created by livemedia-creator.
Vfat Reformating Script
This script copies the boot partition to a temporary folder then reformat's that partition to vfat from ext4 it also removes the swap partition that is created by live-media-creator the swap partition is created during the firstboot process by the user.
Pidora 2014 Reformating Boot Partition Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Reformating-Boot-Script
Example usage
- revfat disk_image.img
Checking The Partition FStype
After executing the revfat script your partition layout should look similar to this:
- fdisk -l disk_image.img
Device Boot Start End Blocks Id System diskz9llIY.img1 * 2048 104447 51200 c W95 FAT32 (LBA) diskz9llIY.img2 104448 6248447 3072000 83 Linux
Shrinking The Image
Script to take an SD card image (such as for a Raspberry Pi) with a dos disklabel and two partitions (boot vfat and rootfs extX), and shrink it so that the rootfs (2nd partition) is as short as possible plus a small free space allowance.
Pidora 2014 Shrink Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Shrink-Script
Example usage
- shrink disk_image.img
Preparing Pidora Noobs distribution
This script extracts and compresses the boot and rootfs of a Pidora image. It also generates and updates all the necessary json files, images and release notes required by the Noobs distribution.
Pidora 2014 Noobs Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Noobs-Script
- Note this script uses PV (pipe viewer), if it's not already installed on the system please install it.
Example Usage
- makenoobs disk_image
Testing Pidora Noobs
Download the latest version of Noobs: http://www.raspberrypi.org/downloads/ (NOOBS Offline and network install) unzip the Noobs package then replace the files located in the os/Pidora directory with the files generated by the makenoobs script
NOOBs Pidora OS location: ./Noobs-Pidora/14-08-18 ./Noobs-Pidora/14-08-18 ├── boot.tar.xz ├── os.json ├── partition_setup.sh ├── partitions.json ├── Pidora.png ├── release-notes.txt ├── rootfs.tar.xz ├── slides │ ├── A.png │ ├── B.png │ ├── C.png │ ├── D.png │ └── E.png └── slides_vga ├── A.png ├── B.png ├── C.png ├── D.png ├── E.png └── Pidora.png 2 directories, 21 files