Difference between revisions of "Pidora SOP"

From CDOT Wiki
Jump to: navigation, search
(→‎Example Livemedia-Creator Command)
(→‎Setting up environment)
 
(82 intermediate revisions by 6 users not shown)
Line 1: Line 1:
 +
{{Pidora}}[[Category:Pidora 18 (Raspberry Pi Fedora Remix)]][[Category:OSTEP SOP]]
 +
 +
{{Admon/important|Seneca-Specific SOP|This SOP is specific to the environment at Seneca CDOT since it refers to specific hosts, configurations, and tools. It is intended solely as a reference for the OSTEP team at CDOT, but the content may be useful to readers in other contexts.}}
 +
 
= Intro =
 
= Intro =
  
 
This page covers how one can manage the various needed parts of a complete release process. This includes the other SOP pages containing file configurations and/or quick command line references for the following procedures:
 
This page covers how one can manage the various needed parts of a complete release process. This includes the other SOP pages containing file configurations and/or quick command line references for the following procedures:
  
 +
# Setting up environment
 
# Building packages
 
# Building packages
 
# Signing packages
 
# Signing packages
 
# Creating repositories
 
# Creating repositories
 
# Composing images
 
# Composing images
 +
 +
= Setting up environment =
 +
 +
Follow the instructions on Chris's blog post: http://blog.chris.tylers.info/index.php?/archives/272-Acessing-the-armv6hl-Koji-Buildsystem.html
 +
 +
For Fedora 21 or later, you need to enable MD5 verification before using koji commands (according to [https://bugzilla.redhat.com/show_bug.cgi?id=1157260 here]):
 +
<pre>
 +
export OPENSSL_ENABLE_MD5_VERIFY=1
 +
</pre>
  
 
= Building Packages =
 
= Building Packages =
−
== Koji-Follow Configuration SOPs ==
+
 
 +
== Koji-Follow Source Code and Configuration SOPs ==
 
* http://zenit.senecac.on.ca/wiki/index.php/Koji_Follow
 
* http://zenit.senecac.on.ca/wiki/index.php/Koji_Follow
 +
 
== How-To-Run ==
 
== How-To-Run ==
 
* ssh <your_username>@japan
 
* ssh <your_username>@japan
Line 16: Line 32:
 
* screen -xr follow
 
* screen -xr follow
 
* python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log
 
* python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log
 +
 +
== Post Commands ==
 +
* Repo analysis: <pre>repoclosure --arch=armv6hl --arch=noarch --repofrompath=v6,http://japan.proximity.on.ca/repos/f20-build/latest/armv6hl/ -r v6 | tee /tmp/reporeport.log</pre>
 +
 +
= Sign, Mash, and Rsync Together! =
 +
 +
== Prerequisites ==
 +
* Must setup passphraseless logins from root@japan to all users in the script
 +
* Must have a working mash configuration: [[Pidora_SOP#Creating_Repositories | See mash below]]
 +
* Must have a configured sigul user with access to the signing key
 +
* Make sure that any externally called scripts are in the correct locations
 +
 +
== Running pidora-update.py ==
 +
 +
* ssh japan
 +
* run as root: pidora-smr
 +
* script location /root/pidora-update/pidora-update.py
 +
 +
<pre>
 +
Usage: pidora-smr [options]
 +
 +
Options:
 +
  -h, --help            show this help message and exit
 +
  -i, --info            check machine status and configuration
 +
  -a, --all            sign, mash, rsync
 +
  -s, --sign            sign all packages in listed tag
 +
  -m, --mash            start a mash run
 +
  -r, --rsync          perform a rsync of the mash repos
 +
  -f, --force          can force some options
 +
  -l, --list-unsigned  list unsigned rpms
 +
  --pidora=PIDORA      specify version of pidora = 18, 19
 +
  --auto                enables logging and emails logs
 +
  --koji-tag=KOJITAG    specify the koji tag to sign
 +
  --email=email@senecacollege.ca
 +
                        specify the email to send logs to
 +
  --sigul-user=USER  specify the user for sigul
 +
  --sigul-host=HOSTNAME
 +
                        specify the host for sigul
 +
  --mash-user=USER      specify the user for mash
 +
  --mash-host=HOSTNAME
 +
                        specify the host for mash
 +
  --rsync-user=USER
 +
                        specify the user for rsync
 +
  --rsync-host=HOSTNAME
 +
                        specify the host for rsync
 +
  --log-dir=/var/log/pidora-smr/
 +
                        specify a logging directory
 +
  --log-file=output    specify a log file name
 +
</pre>
 +
 +
* All the configurations in the script can be changed with the command line options
 +
 +
** Can change the user to access: sigul, mash, and rsync
 +
 +
** Can change the host which runs: sigul, mash, and rsync
 +
 +
** Can change which koji tag will be used for signing
 +
 +
** Can change the version of pidora to ssign, mash, rsync
 +
 +
== Examples runs ==
 +
 +
* Check which hosts can connect successfully
 +
** Also shows default configurations
 +
 +
<pre>
 +
 +
pidora-smr --info
 +
 +
[Connection]
 +
sigulhost = england.proximity.on.ca
 +
siguluser = user
 +
mashhost = japan.proximity.on.ca
 +
mashuser = user
 +
rsynchost = pidora.proximity.on.ca
 +
rsyncuser = user
 +
 +
[General]
 +
auto = False
 +
mashdir = /usr/local/bin/mash-pidora
 +
kojitags = ['f18-updates', 'f18-rpfr-updates', 'f18-updates-testing', 'f18-rpfr-updates-testing']
 +
email = email@senecacollege.ca
 +
 +
logdir = /var/log/pidora-smr/
 +
logfile = /var/log/pidora-smr/output
 +
 +
[Hosts]
 +
working hosts: ['japan.proximity.on.ca', 'england.proximity.on.ca', 'pidora.proximity.on.ca']
 +
failed hosts: []
 +
</pre>
 +
 +
* Get a list of unsigned packages
 +
** Can check a single tag with --koji-tag=<tag-name>
 +
 +
<pre>
 +
 +
pidora-smr --list-unsigned --pidora 18
 +
pidora-smr --list-unsigned --pidora 19
 +
pidora-smr --list-unsigned --pidora 20
 +
 +
</pre>
 +
 +
* Using the default configuration, start a sign, mash, rsync
 +
 +
<pre>
 +
 +
pidora-smr --all --pidora 20
 +
 +
</pre>
 +
 +
* Change sigul user
 +
<pre>
 +
 +
pidora-smr --all --sigul-user=oatley --pidora 20
 +
 +
</pre>
 +
 +
* Run sign, mash, or sync separately, instead of together with the --all option
 +
 +
<pre>
 +
 +
pidora-smr --sign --pidora 20
 +
pidora-smr --mash --pidora 20
 +
pidora-smr --rsync --pidora 20
 +
 +
</pre>
 +
 +
 +
* Changing the configurations with command line options
 +
** Sign using a different user, on a single koji tag
 +
 +
<pre>
 +
 +
pidora-smr --sign --koji-tag=f18-updates-testing --pidora 20
 +
 +
</pre>
 +
 +
== Changing Default Configurations ==
 +
 +
* Can view all default configurations with the following command:
 +
 +
<pre>
 +
 +
pidora-smr --info
 +
 +
</pre>
 +
 +
* Modify the script with a text editor
 +
 +
** Change any of the variables listed below:
 +
 +
<pre>
 +
sigulhost = england.proximity.on.ca
 +
siguluser = user
 +
mashhost = japan.proximity.on.ca
 +
mashuser = user
 +
rsynchost = pidora.proximity.on.ca
 +
rsyncuser = user
 +
 +
[General]
 +
auto = False
 +
mashdir = /usr/local/bin/mash-pidora
 +
kojitags = ['f18-updates', 'f18-rpfr-updates', 'f18-updates-testing', 'f18-rpfr-updates-testing']
 +
email = email@senecacollege.ca
 +
</pre>
 +
 +
== Source Code ==
 +
 +
[https://github.com/oatley/pidora-smr/tree/master Pidora-smr source code]
  
 
= Signing Packages =
 
= Signing Packages =
Line 24: Line 209:
  
 
= Creating Repositories =
 
= Creating Repositories =
−
== Mash Config Files ==
+
 
−
* config file here
+
== Mash File Configuration Details ==
−
== Mash Command Line Details ==
+
 
−
* user/host, crontab lines, script execution, config files, etc
+
=== /etc/mash/pidora-18.mash ===
 +
 
 +
<pre>
 +
# mash config file
 +
 
 +
[pidora-18]
 +
rpm_path = %(arch)s/os/Packages
 +
repodata_path = %(arch)s/os/
 +
source_path = source/SRPMS
 +
debuginfo = True
 +
multilib = False
 +
multilib_method = devel
 +
tag = f18-rpfr
 +
inherit = True
 +
 
 +
# Unsigned is temporary
 +
strict_keys = True
 +
#keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
 +
#keys = CE8D31E3
 +
keys = f1590cd5
 +
 
 +
repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/
 +
repoviewtitle = "Fedora Branched - %(arch)s"
 +
arches = armhfp
 +
# armv6 doesn't have horsepower to make delta useful
 +
delta = False
 +
# Change distro_tags as fedora-release version gets bumped
 +
# The cpe URI should be fixed up
 +
distro_tags = cpe:/o:fedoraproject:fedora:18 rawhide
 +
hash_packages = True
 +
</pre>
 +
 
 +
=== /etc/mash/pidora-18-updates.mash ===
 +
 
 +
<pre>
 +
[pidora-18-updates]
 +
rpm_path = %(arch)s/
 +
repodata_path = %(arch)s/
 +
source_path = SRPMS
 +
debuginfo = True
 +
multilib = False
 +
multilib_method = devel
 +
tag = f18-updates
 +
inherit = False
 +
 
 +
# Unsigned is temporary
 +
strict_keys = True
 +
#keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
 +
#keys = CE8D31E3
 +
keys = f1590cd5
 +
 
 +
repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/
 +
repoviewtitle = "Fedora Branched - %(arch)s"
 +
arches = armhfp
 +
# armv6 doesn't have horsepower to make delta useful
 +
delta = False
 +
# Change distro_tags as fedora-release version gets bumped
 +
# The cpe URI should be fixed up
 +
distro_tags = cpe:/o:fedoraproject:fedora:18 update
 +
hash_packages = True
 +
</pre>
 +
 
 +
=== /etc/mash/pidora-18-updates-testing.mash ===
 +
 
 +
<pre>
 +
[pidora-18-updates-testing]
 +
rpm_path = %(arch)s/
 +
repodata_path = %(arch)s/
 +
source_path = SRPMS
 +
debuginfo = True
 +
multilib = False
 +
multilib_method = devel
 +
tag = f18-updates-testing
 +
inherit = False
 +
 
 +
# Unsigned is temporary
 +
strict_keys = True
 +
#keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
 +
#keys = CE8D31E3
 +
keys = f1590cd5
 +
 
 +
repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/
 +
repoviewtitle = "Fedora Branched - %(arch)s"
 +
arches = armhfp
 +
# armv6 doesn't have horsepower to make delta useful
 +
delta = False
 +
# Change distro_tags as fedora-release version gets bumped
 +
# The cpe URI should be fixed up
 +
distro_tags = cpe:/o:fedoraproject:fedora:18 update
 +
hash_packages = True
 +
</pre>
 +
 
 +
=== /etc/mash/pidora-18-rpfr-updates.mash ===
 +
 
 +
<pre>
 +
[pidora-18-rpfr-updates]
 +
rpm_path = %(arch)s/
 +
repodata_path = %(arch)s/
 +
source_path = SRPMS
 +
debuginfo = True
 +
multilib = False
 +
multilib_method = devel
 +
tag = f18-rpfr-updates
 +
inherit = False
 +
 
 +
# Unsigned is temporary
 +
strict_keys = True
 +
#keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
 +
#keys = CE8D31E3
 +
keys = f1590cd5
 +
 
 +
repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/
 +
repoviewtitle = "Fedora Branched - %(arch)s"
 +
arches = armhfp
 +
# armv6 doesn't have horsepower to make delta useful
 +
delta = False
 +
# Change distro_tags as fedora-release version gets bumped
 +
# The cpe URI should be fixed up
 +
distro_tags = cpe:/o:fedoraproject:fedora:18 update
 +
hash_packages = True
 +
</pre>
 +
 
 +
=== /etc/mash/pidora-18-rpfr-updates-testing.mash ===
 +
 
 +
<pre>
 +
[pidora-18-rpfr-updates-testing]
 +
rpm_path = %(arch)s/
 +
repodata_path = %(arch)s/
 +
source_path = SRPMS
 +
debuginfo = True
 +
multilib = False
 +
multilib_method = devel
 +
tag = f18-rpfr-updates-testing
 +
inherit = False
 +
 
 +
# Unsigned is temporary
 +
strict_keys = True
 +
#keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
 +
#keys = CE8D31E3
 +
keys = f1590cd5
 +
 
 +
repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/
 +
repoviewtitle = "Fedora Branched - %(arch)s"
 +
arches = armhfp
 +
# armv6 doesn't have horsepower to make delta useful
 +
delta = False
 +
# Change distro_tags as fedora-release version gets bumped
 +
# The cpe URI should be fixed up
 +
distro_tags = cpe:/o:fedoraproject:fedora:18 update
 +
hash_packages = True
 +
</pre>
 +
 
 +
=== /etc/mash/mash.conf ===
 +
 
 +
<pre>
 +
[defaults]
 +
configdir = /etc/mash
 +
buildhost = http://japan.proximity.on.ca/kojihub
 +
repodir = file:///mnt/koji
 +
use_sqlite = True
 +
use_repoview = False
 +
</pre>
 +
 
 +
=== /usr/local/bin/mashrun-pidora-18 ===
 +
 
 +
<pre>
 +
#!/bin/bash
 +
#
 +
# mashrun-pidora18 :: Compose a Pidora 18 repository
 +
#
 +
 
 +
DISTNAME=pidora
 +
RELEASE=18
 +
 
 +
# Default mash output dir is unique by dates (YYYYMMDD)
 +
BASEDIR=/mnt/koji/mash
 +
MASHDIR=${BASEDIR}/${DISTNAME}-${RELEASE}-$(date +%Y%m%d)
 +
 
 +
# If there is already a compose output directory for today,
 +
# append -HHMMSS
 +
if [ -d ${MASHDIR} ] # If YYYYMMDD exists, add -HHMMSS
 +
then
 +
    MASHDIR=${MASHDIR}-$(date +%H%M%S)
 +
fi
 +
mkdir -p ${MASHDIR}
 +
 
 +
{
 +
 
 +
exec 2>&1
 +
 
 +
echo "Composing to ${MASHDIR}..."
 +
 
 +
time mash -o ${MASHDIR} ${DISTNAME}-${RELEASE} -f /usr/local/mash/comps-${DISTNAME}-${RELEASE}.xml.gz
 +
 
 +
rm ${BASEDIR}/${DISTNAME}-${RELEASE}-latest
 +
ln -s ${MASHDIR} ${BASEDIR}/${DISTNAME}-${RELEASE}-latest
 +
 
 +
echo
 +
echo "Warnings about unsigned packages are temporarily suppressed from email."
 +
echo "Refer to ${MASHDIR}/mash.log for warnings."
 +
 
 +
}| cat -v | tee ${MASHDIR}/mash.log | fgrep -v '(signed with no key)' | mail -s "${DISTNAME}-${RELEASE} Mash Run" mashrun-alert
 +
 
 +
# The 'cat -v' above is there to prevent an unprintable code from throwing off the
 +
# mail command.
 +
</pre>
 +
 
 +
== Mash Command Line Execution ==
 +
* /usr/local/bin/mashrun-pidora-18
 +
 
 +
== Rsync to Pidora 18 Repositories ==
 +
* ssh pidorapr@pidora.ca
 +
* ~/bin/rsync-japan
  
 
= Composing Images =
 
= Composing Images =
−
== Example Kickstart File ==
+
 
−
http://scotland.proximity.on.ca/raspberrypi/test-releases/rpfr18v6/latest/pidora-18.ks
+
<s>Before you can attempt to run a compose you should check to make sure that the following packages are installed:
 +
* anaconda
 +
* lorax</s>
 +
 
 +
Because livemedia-tools is not stable software for creating pidora images we have a modified version of livemedia-creator that works well however this software is currently out of date, so we have decided to keep using this version of livemedia to produce future images. You can accessing our custom version of livemedia which is located on the host machine cal-7-2  within a chroot log onto the host machine iraq.proximity.on.ca then ssh into the compose node host machine cal-7-2.
 +
*ssh user@iraq.proximity.on.ca
 +
*ssh root@cal-7-2
 +
 
 +
*Note if you restart or the node is shutdown you will need to bind mount /proc, /dev, /sys to f17v6/proc, f17v6/dev, f17v6/sys with the following command before you chroot.
 +
*mount -o bind /proc/ f17v6/proc
 +
*mount -o bind /dev/ f17v6/dev
 +
*mount -o bind /sys/ f17v6/sys
 +
 
 +
 
 +
chroot into the armv6hl directory on the arm device.
 +
* chroot /root/f17v6
 +
 
 +
cd into the livemedia directory or where ever your kickstart file is located.
 +
* cd /livemedia/f20
 +
 
 +
Run the provided Livemedia-Creator command provided below to start the compose process.
  
 
== Example Livemedia-Creator Command ==
 
== Example Livemedia-Creator Command ==
−
livemedia-creator --make-disk --no-virt --image-only --keep-image --ks=rpfr-18.ks
+
* livemedia-creator --make-disk --no-virt --image-only --keep-image --ks=pidora-f20.ks
 +
 
 +
Estimated time for compilation (15mins)
 +
 
 +
== Example Kickstart File ==
 +
Pidora 18 kickstart: http://zenit.senecac.on.ca/wiki/index.php/Pidora_kickstart
 +
Pidora 2014 kickstart: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-kickStart
 +
== Package List File ==
 +
Pidora 2014 Package List kickstart: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Packagelist-kickStart
  
 
* command details
 
* command details
−
-h, --help           show this help message and exit
+
-h, --help             For more info on commands
−
--make-disk           Build a disk image
+
<br>--make-disk       Build a disk image
−
--no-virt             Use Anaconda's image install instead of virt-install
+
<br>--no-virt         Use Anaconda's image install instead of virt-install
−
--image-only         Exit after creating disk image.
+
<br>--image-only       Exit after creating disk image.
−
--ks KS               Kickstart file defining the install.
+
<br>--ks KS           Kickstart file defining the install.
  
 
== Example Post Script Execution ==
 
== Example Post Script Execution ==
−
* post script details
+
 
 +
You may have more than one %post section, which can be useful for cases where some post-installation scripts need to be run in the chroot and others that need access outside the chroot.
 +
 
 +
%post
 +
section is required to be closed with a corresponding
 +
%end
 +
 
 +
Post Section Options
 +
* command details
 +
 
 +
--nochroot
 +
<br>Allows you to specify commands that you would like to run outside of the chroot environment.
 +
 
 +
--interpreter /usr/bin/python
 +
<br>Allows you to specify a different scripting language, such as Python. Replace /usr/bin/python with the scripting language of your choice.
 +
 
 +
--erroronfail
 +
<br>If the post-installation script fails, this option will cause an error dialog to be displayed and will halt installation. The error message will direct you to where the cause of the failure is logged.
 +
 
 +
--log=
 +
 
 +
 
 +
Example Post Installation Script "Creating/Modifying the fstab"
 +
 
 +
cat << EOF > /etc/fstab
 +
<br>LABEL="rootfs"          /                      ext4    defaults,noatime                1 1
 +
<br>LABEL="boot"            /boot                  vfat    noauto,comment=systemd.automount 1 2
 +
<br>EOF
 +
 
 +
 
 +
== Preparing The Final Image ==
 +
 
 +
When you have a successful disk image ready the boot partition needs to be reformatted with a vfat filesystem type, the Raspberry Pi will only boot with a vfat boot partition. This can be done manually or by using the script provided below. The final image's rootfs also needs to be resized to minimize the size of file system, there is also a script provided to do this.
 +
 
 +
* all the preparation scripts are located in /var/tmp/ directory along with the disk*.img file created by livemedia-creator.
 +
* <s>Note these scripts work best if you copy them to your host machine and run them locally.</s> If you choose todo so you will need to scp the disk_image to your host machine.
 +
 
 +
== Vfat Reformating Script ==
 +
This script copies the boot partition to a temporary folder then reformat's that partition to vfat from ext4 it also removes the swap partition that is created by live-media-creator the swap partition is created during the firstboot process by the user.
 +
 
 +
Pidora 2014 Reformating Boot Partition Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Reformating-Boot-Script
 +
 
 +
Example usage
 +
* revfat disk_image.img
 +
 
 +
== Checking The Partition FStype ==
 +
After executing the revfat script your partition layout should look similar to this:
 +
 
 +
*fdisk -l disk_image.img
 +
<pre>
 +
        Device Boot      Start        End      Blocks  Id  System
 +
diskz9llIY.img1  *        2048      104447      51200    c  W95 FAT32 (LBA)
 +
diskz9llIY.img2          104448    6248447    3072000  83  Linux
 +
</pre>
 +
 
 +
== Shrinking The Image ==
 +
Script to take an SD card image (such as for a Raspberry Pi) with a dos disklabel and two partitions (boot vfat and rootfs extX), and shrink it so that the rootfs (2nd partition) is as short as possible plus a small free space allowance.
 +
Pidora 2014 Shrink Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Shrink-Script
 +
 
 +
Example usage
 +
* shrink disk_image.img
 +
 
 +
*Note sometimes this script does not unmount the tmp directory it mounts /tmp/tmp.v9ADZEWeP2.
 +
 
 +
script output
 +
<pre>
 +
Shrinking /var/tmp/diskTLTF4t.img to minimum size plus 220 MB.
 +
Filesystem          Size  Used Avail Use% Mounted on
 +
/dev/mapper/loop0p2  2.9G  1.8G  1.1G  63% /tmp/tmp.v9ADZEWeP2
 +
220+0 records in
 +
220+0 records out
 +
230686720 bytes (231 MB) copied, 2.30747 s, 100 MB/s
 +
Filesystem          Size  Used Avail Use% Mounted on
 +
/dev/mapper/loop0p2  2.9G  2.0G  832M  71% /tmp/tmp.v9ADZEWeP2
 +
e2fsck 1.42 (29-Nov-2011)
 +
Pass 1: Checking inodes, blocks, and sizes
 +
Pass 2: Checking directory structure
 +
Pass 3: Checking directory connectivity
 +
Pass 4: Checking reference counts
 +
Pass 5: Checking group summary information
 +
rootfs: 77485/192000 files (0.1% non-contiguous), 516702/768000 blocks
 +
 
 +
 
 +
Disk /var/tmp/diskTLTF4t.img: 4294 MB, 4294967296 bytes
 +
255 heads, 63 sectors/track, 522 cylinders, total 8388608 sectors
 +
Units = sectors of 1 * 512 = 512 bytes
 +
Sector size (logical/physical): 512 bytes / 512 bytes
 +
I/O size (minimum/optimal): 512 bytes / 512 bytes
 +
Disk identifier: 0x0009c1ec
 +
 
 +
                  Device Boot      Start        End      Blocks  Id  System
 +
/var/tmp/diskTLTF4t.img1  *        2048      104447      51200    c  W95 FAT32 (LBA)
 +
/var/tmp/diskTLTF4t.img2          104448    4234064    2064808+  83  Linux
 +
Filesystem          Size  Used Avail Use% Mounted on
 +
/dev/mapper/loop0p2  2.0G  2.0G    0 100% /tmp/tmp.v9ADZEWeP2
 +
removed `/tmp/tmp.v9ADZEWeP2/resize-reserve'
 +
Filesystem          Size  Used Avail Use% Mounted on
 +
/dev/mapper/loop0p2  2.0G  1.8G  133M  93% /tmp/tmp.v9ADZEWeP2
 +
Image shrink completed.
 +
</pre>
 +
 
 +
== Preparing Pidora Noobs distribution ==
 +
This script extracts and compresses the boot and rootfs of a Pidora image. It also generates and updates all the necessary json files, images and release notes required by the Noobs distribution.
 +
 +
Pidora 2014 Noobs Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Noobs-Script
 +
 
 +
*Note this script uses PV (pipe viewer), if it's not already installed on the system please install it.
 +
 
 +
Example Usage
 +
* makenoobs disk_image
 +
 
 +
== Testing Pidora Noobs ==
 +
 
 +
Download the latest version of Noobs: http://www.raspberrypi.org/downloads/ (NOOBS Offline and network install)
 +
unzip the Noobs package then replace the files located in the os/Pidora directory with the files generated by the makenoobs script
 +
 
 +
  <pre>NOOBs Pidora OS location: ./Noobs-Pidora/14-08-18
 +
./Noobs-Pidora/14-08-18
 +
├── boot.tar.xz
 +
├── os.json
 +
├── partition_setup.sh
 +
├── partitions.json
 +
├── Pidora.png
 +
├── release-notes.txt
 +
├── rootfs.tar.xz
 +
├── slides
 +
│  ├── A.png
 +
│  ├── B.png
 +
│  ├── C.png
 +
│  ├── D.png
 +
│  └── E.png
 +
└── slides_vga
 +
    ├── A.png
 +
    ├── B.png
 +
    ├── C.png
 +
    ├── D.png
 +
    ├── E.png
 +
    └── Pidora.png
 +
 
 +
2 directories, 21 files
 +
</pre>

Latest revision as of 20:00, 6 February 2015

Pidora-horizontal.png
Important.png
Seneca-Specific SOP
This SOP is specific to the environment at Seneca CDOT since it refers to specific hosts, configurations, and tools. It is intended solely as a reference for the OSTEP team at CDOT, but the content may be useful to readers in other contexts.

Intro

This page covers how one can manage the various needed parts of a complete release process. This includes the other SOP pages containing file configurations and/or quick command line references for the following procedures:

  1. Setting up environment
  2. Building packages
  3. Signing packages
  4. Creating repositories
  5. Composing images

Setting up environment

Follow the instructions on Chris's blog post: http://blog.chris.tylers.info/index.php?/archives/272-Acessing-the-armv6hl-Koji-Buildsystem.html

For Fedora 21 or later, you need to enable MD5 verification before using koji commands (according to here): 

export OPENSSL_ENABLE_MD5_VERIFY=1

Building Packages

Koji-Follow Source Code and Configuration SOPs

How-To-Run

  • ssh <your_username>@japan
  • sudo kojiadmin
  • screen -xr follow
  • python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log

Post Commands

  • Repo analysis: 
    repoclosure --arch=armv6hl --arch=noarch --repofrompath=v6,http://japan.proximity.on.ca/repos/f20-build/latest/armv6hl/ -r v6 | tee /tmp/reporeport.log

Sign, Mash, and Rsync Together!

Prerequisites

  • Must setup passphraseless logins from root@japan to all users in the script
  • Must have a working mash configuration: See mash below
  • Must have a configured sigul user with access to the signing key
  • Make sure that any externally called scripts are in the correct locations

Running pidora-update.py

  • ssh japan
  • run as root: pidora-smr
  • script location /root/pidora-update/pidora-update.py
Usage: pidora-smr [options]

Options:
  -h, --help            show this help message and exit
  -i, --info            check machine status and configuration
  -a, --all             sign, mash, rsync
  -s, --sign            sign all packages in listed tag
  -m, --mash            start a mash run
  -r, --rsync           perform a rsync of the mash repos
  -f, --force           can force some options
  -l, --list-unsigned   list unsigned rpms
  --pidora=PIDORA       specify version of pidora = 18, 19
  --auto                enables logging and emails logs
  --koji-tag=KOJITAG    specify the koji tag to sign
  --email=email@senecacollege.ca
                        specify the email to send logs to
  --sigul-user=USER  specify the user for sigul
  --sigul-host=HOSTNAME
                        specify the host for sigul
  --mash-user=USER      specify the user for mash
  --mash-host=HOSTNAME
                        specify the host for mash
  --rsync-user=USER
                        specify the user for rsync
  --rsync-host=HOSTNAME
                        specify the host for rsync
  --log-dir=/var/log/pidora-smr/
                        specify a logging directory
  --log-file=output     specify a log file name
  • All the configurations in the script can be changed with the command line options
    • Can change the user to access: sigul, mash, and rsync
    • Can change the host which runs: sigul, mash, and rsync
    • Can change which koji tag will be used for signing
    • Can change the version of pidora to ssign, mash, rsync

Examples runs

  • Check which hosts can connect successfully
    • Also shows default configurations

pidora-smr --info

[Connection]
sigulhost = england.proximity.on.ca
siguluser = user
mashhost = japan.proximity.on.ca
mashuser = user
rsynchost = pidora.proximity.on.ca
rsyncuser = user

[General]
auto = False
mashdir = /usr/local/bin/mash-pidora
kojitags = ['f18-updates', 'f18-rpfr-updates', 'f18-updates-testing', 'f18-rpfr-updates-testing']
email = email@senecacollege.ca

logdir = /var/log/pidora-smr/
logfile = /var/log/pidora-smr/output

[Hosts]
working hosts: ['japan.proximity.on.ca', 'england.proximity.on.ca', 'pidora.proximity.on.ca']
failed hosts: []
  • Get a list of unsigned packages
    • Can check a single tag with --koji-tag=<tag-name>

pidora-smr --list-unsigned --pidora 18
pidora-smr --list-unsigned --pidora 19
pidora-smr --list-unsigned --pidora 20
  • Using the default configuration, start a sign, mash, rsync

pidora-smr --all --pidora 20
  • Change sigul user

pidora-smr --all --sigul-user=oatley --pidora 20
  • Run sign, mash, or sync separately, instead of together with the --all option

pidora-smr --sign --pidora 20
pidora-smr --mash --pidora 20
pidora-smr --rsync --pidora 20


  • Changing the configurations with command line options
    • Sign using a different user, on a single koji tag

pidora-smr --sign --koji-tag=f18-updates-testing --pidora 20

Changing Default Configurations

  • Can view all default configurations with the following command:

pidora-smr --info
  • Modify the script with a text editor
    • Change any of the variables listed below:
sigulhost = england.proximity.on.ca
siguluser = user
mashhost = japan.proximity.on.ca
mashuser = user
rsynchost = pidora.proximity.on.ca
rsyncuser = user

[General]
auto = False
mashdir = /usr/local/bin/mash-pidora
kojitags = ['f18-updates', 'f18-rpfr-updates', 'f18-updates-testing', 'f18-rpfr-updates-testing']
email = email@senecacollege.ca

Source Code

Pidora-smr source code

Signing Packages

Sigul Configuration SOPs

Creating Repositories

Mash File Configuration Details

/etc/mash/pidora-18.mash

# mash config file

[pidora-18]
rpm_path = %(arch)s/os/Packages
repodata_path = %(arch)s/os/
source_path = source/SRPMS
debuginfo = True
multilib = False
multilib_method = devel
tag = f18-rpfr
inherit = True

# Unsigned is temporary
strict_keys = True
#keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
#keys = CE8D31E3
keys = f1590cd5

repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/
repoviewtitle = "Fedora Branched - %(arch)s"
arches = armhfp
# armv6 doesn't have horsepower to make delta useful
delta = False
# Change distro_tags as fedora-release version gets bumped
# The cpe URI should be fixed up
distro_tags = cpe:/o:fedoraproject:fedora:18 rawhide
hash_packages = True

/etc/mash/pidora-18-updates.mash

[pidora-18-updates]
rpm_path = %(arch)s/
repodata_path = %(arch)s/
source_path = SRPMS
debuginfo = True
multilib = False
multilib_method = devel
tag = f18-updates
inherit = False

# Unsigned is temporary
strict_keys = True
#keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
#keys = CE8D31E3
keys = f1590cd5

repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/
repoviewtitle = "Fedora Branched - %(arch)s"
arches = armhfp
# armv6 doesn't have horsepower to make delta useful
delta = False
# Change distro_tags as fedora-release version gets bumped
# The cpe URI should be fixed up
distro_tags = cpe:/o:fedoraproject:fedora:18 update
hash_packages = True

/etc/mash/pidora-18-updates-testing.mash

[pidora-18-updates-testing]
rpm_path = %(arch)s/
repodata_path = %(arch)s/
source_path = SRPMS
debuginfo = True
multilib = False
multilib_method = devel
tag = f18-updates-testing
inherit = False

# Unsigned is temporary
strict_keys = True
#keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
#keys = CE8D31E3
keys = f1590cd5

repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/
repoviewtitle = "Fedora Branched - %(arch)s"
arches = armhfp
# armv6 doesn't have horsepower to make delta useful
delta = False
# Change distro_tags as fedora-release version gets bumped
# The cpe URI should be fixed up
distro_tags = cpe:/o:fedoraproject:fedora:18 update
hash_packages = True

/etc/mash/pidora-18-rpfr-updates.mash

[pidora-18-rpfr-updates]
rpm_path = %(arch)s/
repodata_path = %(arch)s/
source_path = SRPMS
debuginfo = True
multilib = False
multilib_method = devel
tag = f18-rpfr-updates
inherit = False

# Unsigned is temporary
strict_keys = True
#keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
#keys = CE8D31E3
keys = f1590cd5

repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/
repoviewtitle = "Fedora Branched - %(arch)s"
arches = armhfp
# armv6 doesn't have horsepower to make delta useful
delta = False
# Change distro_tags as fedora-release version gets bumped
# The cpe URI should be fixed up
distro_tags = cpe:/o:fedoraproject:fedora:18 update
hash_packages = True

/etc/mash/pidora-18-rpfr-updates-testing.mash

[pidora-18-rpfr-updates-testing]
rpm_path = %(arch)s/
repodata_path = %(arch)s/
source_path = SRPMS
debuginfo = True
multilib = False
multilib_method = devel
tag = f18-rpfr-updates-testing
inherit = False

# Unsigned is temporary
strict_keys = True
#keys = 34E166FA, F8DF67E6, A82BA4B7, 069C8460, 97A1071F, E8E40FDE, 57BBCCBA, D22E77F2, 4EBFC273, 0B86274E, 6DF2196F, DF9B0AE9
#keys = CE8D31E3
keys = f1590cd5

repoviewurl = http://download.fedoraproject.org/pub/fedora-secondary/development/18/%(arch)s/os/
repoviewtitle = "Fedora Branched - %(arch)s"
arches = armhfp
# armv6 doesn't have horsepower to make delta useful
delta = False
# Change distro_tags as fedora-release version gets bumped
# The cpe URI should be fixed up
distro_tags = cpe:/o:fedoraproject:fedora:18 update
hash_packages = True

/etc/mash/mash.conf

[defaults]
configdir = /etc/mash
buildhost = http://japan.proximity.on.ca/kojihub
repodir = file:///mnt/koji
use_sqlite = True
use_repoview = False

/usr/local/bin/mashrun-pidora-18

#!/bin/bash
#
# mashrun-pidora18 :: Compose a Pidora 18 repository
#

DISTNAME=pidora
RELEASE=18

# Default mash output dir is unique by dates (YYYYMMDD)
BASEDIR=/mnt/koji/mash
MASHDIR=${BASEDIR}/${DISTNAME}-${RELEASE}-$(date +%Y%m%d)

# If there is already a compose output directory for today,
# append -HHMMSS
if [ -d ${MASHDIR} ] # If YYYYMMDD exists, add -HHMMSS
then
    MASHDIR=${MASHDIR}-$(date +%H%M%S)
fi
mkdir -p ${MASHDIR}

{

exec 2>&1

echo "Composing to ${MASHDIR}..."

time mash -o ${MASHDIR} ${DISTNAME}-${RELEASE} -f /usr/local/mash/comps-${DISTNAME}-${RELEASE}.xml.gz

rm ${BASEDIR}/${DISTNAME}-${RELEASE}-latest
ln -s ${MASHDIR} ${BASEDIR}/${DISTNAME}-${RELEASE}-latest

echo
echo "Warnings about unsigned packages are temporarily suppressed from email."
echo "Refer to ${MASHDIR}/mash.log for warnings."

}| cat -v | tee ${MASHDIR}/mash.log | fgrep -v '(signed with no key)' | mail -s "${DISTNAME}-${RELEASE} Mash Run" mashrun-alert

# The 'cat -v' above is there to prevent an unprintable code from throwing off the
# mail command.

Mash Command Line Execution

  • /usr/local/bin/mashrun-pidora-18

Rsync to Pidora 18 Repositories

  • ssh pidorapr@pidora.ca
  • ~/bin/rsync-japan

Composing Images

Before you can attempt to run a compose you should check to make sure that the following packages are installed:

  • anaconda
  • lorax

Because livemedia-tools is not stable software for creating pidora images we have a modified version of livemedia-creator that works well however this software is currently out of date, so we have decided to keep using this version of livemedia to produce future images. You can accessing our custom version of livemedia which is located on the host machine cal-7-2 within a chroot log onto the host machine iraq.proximity.on.ca then ssh into the compose node host machine cal-7-2.

  • ssh user@iraq.proximity.on.ca
  • ssh root@cal-7-2
  • Note if you restart or the node is shutdown you will need to bind mount /proc, /dev, /sys to f17v6/proc, f17v6/dev, f17v6/sys with the following command before you chroot.
  • mount -o bind /proc/ f17v6/proc
  • mount -o bind /dev/ f17v6/dev
  • mount -o bind /sys/ f17v6/sys


chroot into the armv6hl directory on the arm device.

  • chroot /root/f17v6

cd into the livemedia directory or where ever your kickstart file is located.

  • cd /livemedia/f20

Run the provided Livemedia-Creator command provided below to start the compose process.

Example Livemedia-Creator Command

  • livemedia-creator --make-disk --no-virt --image-only --keep-image --ks=pidora-f20.ks

Estimated time for compilation (15mins)

Example Kickstart File

Pidora 18 kickstart: http://zenit.senecac.on.ca/wiki/index.php/Pidora_kickstart
Pidora 2014 kickstart: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-kickStart

Package List File

Pidora 2014 Package List kickstart: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Packagelist-kickStart
  • command details

-h, --help For more info on commands
--make-disk Build a disk image
--no-virt Use Anaconda's image install instead of virt-install
--image-only Exit after creating disk image.
--ks KS Kickstart file defining the install.

Example Post Script Execution

You may have more than one %post section, which can be useful for cases where some post-installation scripts need to be run in the chroot and others that need access outside the chroot.

%post section is required to be closed with a corresponding %end

Post Section Options

  • command details

--nochroot
Allows you to specify commands that you would like to run outside of the chroot environment.

--interpreter /usr/bin/python
Allows you to specify a different scripting language, such as Python. Replace /usr/bin/python with the scripting language of your choice.

--erroronfail
If the post-installation script fails, this option will cause an error dialog to be displayed and will halt installation. The error message will direct you to where the cause of the failure is logged.

--log=


Example Post Installation Script "Creating/Modifying the fstab"

cat << EOF > /etc/fstab
LABEL="rootfs" / ext4 defaults,noatime 1 1
LABEL="boot" /boot vfat noauto,comment=systemd.automount 1 2
EOF


Preparing The Final Image

When you have a successful disk image ready the boot partition needs to be reformatted with a vfat filesystem type, the Raspberry Pi will only boot with a vfat boot partition. This can be done manually or by using the script provided below. The final image's rootfs also needs to be resized to minimize the size of file system, there is also a script provided to do this.

  • all the preparation scripts are located in /var/tmp/ directory along with the disk*.img file created by livemedia-creator.
  • Note these scripts work best if you copy them to your host machine and run them locally. If you choose todo so you will need to scp the disk_image to your host machine.

Vfat Reformating Script

This script copies the boot partition to a temporary folder then reformat's that partition to vfat from ext4 it also removes the swap partition that is created by live-media-creator the swap partition is created during the firstboot process by the user. 

Pidora 2014 Reformating Boot Partition Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Reformating-Boot-Script

Example usage

  • revfat disk_image.img

Checking The Partition FStype

After executing the revfat script your partition layout should look similar to this:

  • fdisk -l disk_image.img
         Device Boot      Start         End      Blocks   Id  System
diskz9llIY.img1   *        2048      104447       51200    c  W95 FAT32 (LBA)
diskz9llIY.img2          104448     6248447     3072000   83  Linux

Shrinking The Image

Script to take an SD card image (such as for a Raspberry Pi) with a dos disklabel and two partitions (boot vfat and rootfs extX), and shrink it so that the rootfs (2nd partition) is as short as possible plus a small free space allowance. 

Pidora 2014 Shrink Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Shrink-Script

Example usage

  • shrink disk_image.img
  • Note sometimes this script does not unmount the tmp directory it mounts /tmp/tmp.v9ADZEWeP2.

script output 

Shrinking /var/tmp/diskTLTF4t.img to minimum size plus 220 MB.
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/loop0p2  2.9G  1.8G  1.1G  63% /tmp/tmp.v9ADZEWeP2
220+0 records in
220+0 records out
230686720 bytes (231 MB) copied, 2.30747 s, 100 MB/s
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/loop0p2  2.9G  2.0G  832M  71% /tmp/tmp.v9ADZEWeP2
e2fsck 1.42 (29-Nov-2011)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
rootfs: 77485/192000 files (0.1% non-contiguous), 516702/768000 blocks


Disk /var/tmp/diskTLTF4t.img: 4294 MB, 4294967296 bytes
255 heads, 63 sectors/track, 522 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x0009c1ec

                  Device Boot      Start         End      Blocks   Id  System
/var/tmp/diskTLTF4t.img1   *        2048      104447       51200    c  W95 FAT32 (LBA)
/var/tmp/diskTLTF4t.img2          104448     4234064     2064808+  83  Linux
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/loop0p2  2.0G  2.0G     0 100% /tmp/tmp.v9ADZEWeP2
removed `/tmp/tmp.v9ADZEWeP2/resize-reserve'
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/loop0p2  2.0G  1.8G  133M  93% /tmp/tmp.v9ADZEWeP2
Image shrink completed.

Preparing Pidora Noobs distribution

This script extracts and compresses the boot and rootfs of a Pidora image. It also generates and updates all the necessary json files, images and release notes required by the Noobs distribution. 

Pidora 2014 Noobs Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Noobs-Script
  • Note this script uses PV (pipe viewer), if it's not already installed on the system please install it.

Example Usage

  • makenoobs disk_image

Testing Pidora Noobs

Download the latest version of Noobs: http://www.raspberrypi.org/downloads/ (NOOBS Offline and network install) unzip the Noobs package then replace the files located in the os/Pidora directory with the files generated by the makenoobs script 

NOOBs Pidora OS location: ./Noobs-Pidora/14-08-18
./Noobs-Pidora/14-08-18
├── boot.tar.xz
├── os.json
├── partition_setup.sh
├── partitions.json
├── Pidora.png
├── release-notes.txt
├── rootfs.tar.xz
├── slides
│   ├── A.png
│   ├── B.png
│   ├── C.png
│   ├── D.png
│   └── E.png
└── slides_vga
    ├── A.png
    ├── B.png
    ├── C.png
    ├── D.png
    ├── E.png
    └── Pidora.png

2 directories, 21 files
Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=Pidora_SOP&oldid=108396"