Changes

Jump to: navigation, search

Pidora SOP

2,245 bytes added, 20:00, 6 February 2015
→‎Setting up environment
{{Pidora}}[[Category:Pidora 18 (Raspberry Pi Fedora Remix)]][[Category:OSTEP SOP]]
 
{{Admon/important|Seneca-Specific SOP|This SOP is specific to the environment at Seneca CDOT since it refers to specific hosts, configurations, and tools. It is intended solely as a reference for the OSTEP team at CDOT, but the content may be useful to readers in other contexts.}}
 
= Intro =
This page covers how one can manage the various needed parts of a complete release process. This includes the other SOP pages containing file configurations and/or quick command line references for the following procedures:
# Setting up environment
# Building packages
# Signing packages
# Creating repositories
# Composing images
 
= Setting up environment =
 
Follow the instructions on Chris's blog post: http://blog.chris.tylers.info/index.php?/archives/272-Acessing-the-armv6hl-Koji-Buildsystem.html
 
For Fedora 21 or later, you need to enable MD5 verification before using koji commands (according to [https://bugzilla.redhat.com/show_bug.cgi?id=1157260 here]):
<pre>
export OPENSSL_ENABLE_MD5_VERIFY=1
</pre>
= Building Packages =
== Post Commands ==
* Repo analysis: <pre>repoclosure --arch=armv6hl --arch=noarch --repofrompath=v6,http://japan.proximity.on.ca/repos/f18f20-build/latest/armv6hl/ -r v6 | tee /tmp/reporeport.log</pre>
= Sign, Mash, and Rsync Together! =
== Prerequisites ==
* Must setup passwordless passphraseless logins from root@japan to all users in the script
* Must have a working mash configuration: [[Pidora_SOP#Creating_Repositories | See mash below]]
* Must have a configured sigul user with access to the signing key
* Make sure that any externally called scripts are in the correct locations
 
== Running pidora-update.py ==
* ssh japan
* run as root : pidora-smr* script location /root/pidora-update/pidora-update.py
<pre>
-m, --mash start a mash run
-r, --rsync perform a rsync of the mash repos
-f, --force can force some options
-l, --list-unsigned list unsigned rpms
--pidora=PIDORA specify version of pidora = 18, 19
--auto enables logging and emails logs
--koji-tag=KOJITAG specify the koji tag to sign
--email=andrew.oatley-willisemail@senecacollege.ca
specify the email to send logs to
--sigul-user=agreene USER specify the user for sigul --sigul-host=england.proximity.on.caHOSTNAME
specify the host for sigul
--mash-user=root USER specify the user for mash --mash-host=japan.proximity.on.caHOSTNAME
specify the host for mash
--rsync-user=pidoraprUSER
specify the user for rsync
--rsync-host=pidora.proximity.on.caHOSTNAME
specify the host for rsync
--log-dir=/var/log/pidora-smr/
** Can change which koji tag will be used for signing
 
** Can change the version of pidora to ssign, mash, rsync
== Examples runs ==
* Check which hosts can connect successfully
 
** Also shows default configurations
<pre>
./pidora-update.py smr --info
[Connection]sigulhost = england.proximity.on.casiguluser = usermashhost = japan.proximity.on.camashuser = userrsynchost = pidora.proximity.on.casiguluser rsyncuser = agreeneusermashuser = rootrsyncuser [General]auto = pidoraprFalse
mashdir = /usr/local/bin/mash-pidora
kojitags = ['f18-updates', 'f18-rpfr-updates', 'f18-updates-testing', 'f18-rpfr-updates-testing']email = email@senecacollege.ca
logdir = /var/log/pidora-smr/logfile = /var/log/pidora-smr/output [Hosts]working hosts: ['japan.proximity.on.ca', 'england.proximity.on.ca', 'pidora.proximity.on.ca']failed hosts: []=
</pre>
* Get a list of unsigned packages
 
** Can check a single tag with --koji-tag=<tag-name>
<pre>
./pidora-update.py smr --list-unsigned--pidora 18pidora-smr --list-unsigned --pidora 19pidora-smr --list-unsigned --pidora 20
</pre>
<pre>
./pidora-update.py smr --all--pidora 20
</pre>
<pre>
./pidora-update.py smr --all --sigul-user=oatley--pidora 20
</pre>
<pre>
./pidora-update.py smr --sign--pidora 20./pidora-update.py smr --mash--pidora 20./pidora-update.py smr --rsync--pidora 20
</pre>
* Changing the configurations with command line options
 
** Sign using a different user, on a single koji tag
<pre>
./pidora-update.py smr --sign --koji-tag=f18-updates-testing--pidora 20
</pre>
<pre>
./pidora-update.py smr --info
</pre>
<pre>
sigulhost = england.proximity.on.casiguluser = usermashhost = japan.proximity.on.camashuser = userrsynchost = pidora.proximity.on.casiguluser rsyncuser = agreeneusermashuser = rootrsyncuser [General]auto = pidoraprFalse
mashdir = /usr/local/bin/mash-pidora
kojitags = ['f18-updates', 'f18-rpfr-updates', 'f18-updates-testing', 'f18-rpfr-updates-testing']email = email@senecacollege.ca
</pre>
== Source Code ==
[[https://github.com/oatley/pidora-smr/tree/master Pidora-Update-Source | pidora-update.py]smr source code]
= Signing Packages =
= Composing Images =
<s>Before you can attempt to run a compose you should check to make sure that the following packages are installed:
* anaconda
* lorax</s> Because livemedia-tools is not stable software for creating pidora images we have a modified version of livemedia-creator that works well however this software is currently out of date, so we have decided to keep using this version of livemedia to produce future images. You can accessing our custom version of livemedia which is located on the host machine cal-7-2 within a chroot log onto the host machine iraq.proximity.on.ca then ssh into the compose node host machine cal-7-2.*ssh user@iraq.proximity.on.ca*ssh root@cal-7-2 *Note if you restart or the node is shutdown you will need to bind mount /proc, /dev, /sys to f17v6/proc, f17v6/dev, f17v6/sys with the following command before you chroot.*mount -o bind /proc/ f17v6/proc*mount -o bind /dev/ f17v6/dev*mount -o bind /sys/ f17v6/sys 
SSH to the arm device you want to run the compose on.
chroot into the armv6hl directory on the arm device.
* chroot /root/f17v6
cd into the livemedia directory or where ever your kickstart file is located.
* cd /livemedia/f20
Run the provided Livemedia-Creator command provided below to start the compose process.
== Example Livemedia-Creator Command ==
* livemedia-creator --make-disk --no-virt --image-only --keep-image --ks=pidora-f20.ks
 
Estimated time for compilation (15mins)
== Example Kickstart File ==
Pidora 18 kickstart: http://scotlandzenit.proximitysenecac.on.ca/raspberrypiwiki/index.php/Pidora_kickstart Pidora 2014 kickstart: http://test-releaseszenit.senecac.on.ca/rpfr18v6wiki/latestindex.php/pidoraPidora-18.ks2014-kickStart== Example Livemedia-Creator Command Package List File ==* livemedia-creator Pidora 2014 Package List kickstart: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-makePackagelist-disk --no-virt --image-only --keep-image --ks=rpfr-18.ks Estimated time for compilation (27mins)kickStart
* command details
<br>EOF
= Ansible Builder Configuration Management =
== Details About Ansible Preparing The Final Image ==
Ansible allows for remotely managing When you have a successful disk image ready the configuration of all builders in boot partition needs to be reformatted with a vfat filesystem type, the Raspberry Pi will only boot with a simple and efficient wayvfat boot partition. Ansible works This can be done manually or by running a playbookusing the script provided below. The final image's rootfs also needs to be resized to minimize the size of file system, a playbook there is also a way script provided to organize plays and run plays. A play is a set of ansible "command" or "modules" that are used on each builder, these modules can: copy files, change permissions, modify files, run commands, run scripts, and much moredo this.
host = japandirectory = * all the preparation scripts are located in /var/etctmp/ansiblehosts directory along with the disk*.img file = /etc/ansible/ansible_hostscreated by livemedia-creator.ansible config = /etc/ansible/ansible* <s>Note these scripts work best if you copy them to your host machine and run them locally.cfgplaybook = </etc/ansible/install_builders> If you choose todo so you will need to scp the disk_image to your host machine.ymlplays = /etc/ansible/builders_tasks/builders files = /etc/ansible/builders/
== How To Use Ansible Vfat Reformating Script ==This script copies the boot partition to a temporary folder then reformat's that partition to vfat from ext4 it also removes the swap partition that is created by live-media-creator the swap partition is created during the firstboot process by the user.
* Log in to japan as root<pre>ssh japan< Pidora 2014 Reformating Boot Partition Script: http:/pre>/zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Reformating-Boot-Script
Example usage* Change to the ansible directory<pre>cd /etc/ansible</pre>revfat disk_image.img
* Check == Checking The Partition FStype ==After executing the status of all hosts connected revfat script your partition layout should look similar to ansible** The word builders in the command below is specifying an ansible group<pre>ansible -m ping builders</pre>this:
* Copy over all configurations required and start the koji servicefdisk -l disk_image.img
<pre>
ansible-playbook install_builders Device Boot Start End Blocks Id Systemdiskz9llIY.yml --verboseimg1 * 2048 104447 51200 c W95 FAT32 (LBA)diskz9llIY.img2 104448 6248447 3072000 83 Linux
</pre>
== Change Builder Configurations Shrinking The Image ==Script to take an SD card image (such as for a Raspberry Pi) with a dos disklabel and two partitions (boot vfat and rootfs extX), and shrink it so that the rootfs (2nd partition) is as short as possible plus a small free space allowance. Pidora 2014 Shrink Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Shrink-Script
The best way to edit a play in ansible is to find the ansible module that is needed and read about it. Ansible modules have great documentation and there are tons of them, so many that there is one for every task that needs to be completedExample usage* shrink disk_image.img
* The ansible modules can be found here: [http:Note sometimes this script does not unmount the tmp directory it mounts /tmp/wwwtmp.ansibleworksv9ADZEWeP2.com/docs/modules.html/ Ansible Modules]
* All builder plays can be found inside script output<pre>Shrinking /etcvar/ansibletmp/builders_playsdiskTLTF4t.img to minimum size plus 220 MB.Filesystem Size Used Avail Use% Mounted on/ on japandev/mapper/loop0p2 2.9G 1.8G 1.1G 63% /tmp/tmp. v9ADZEWeP2220+0 records in220+0 records out* Make sure that if a new play is created230686720 bytes (231 MB) copied, it is added into the playbook at 2.30747 s, 100 MB/sFilesystem Size Used Avail Use% Mounted on/dev/etcmapper/ansibleloop0p2 2.9G 2.0G 832M 71% /install_builderstmp/tmp.yml on japanv9ADZEWeP2e2fsck 1.42 (29-Nov-2011)Pass 1: Checking inodes, blocks, and sizesPass 2: Checking directory structurePass 3: Checking directory connectivityPass 4: Checking reference countsPass 5: Checking group summary informationrootfs: 77485/192000 files (0.1% non-contiguous), 516702/768000 blocks
== How To Set Up A New Builder ==
 
Before adding a builder to ansible, there are a few things that need to be completed.
 
=== Network ===
* Add a hostname to the /etc/hosts file on japan
* Add a hostname to the /etc/ansible/builders/config_files/hosts file on japan
 
* If it uses DHCP, then link the hostname to a host in /etc/dhcp/dhcpd.conf by specifying the mac address and host name
or
* If the builder has a changing mac address and can't use DHCP, get access to the builder and set the ip manually
<pre>
ifconfig <interface> <ipaddr> netmask 255.255.255.0 up
route add default gw 192.168.1.254
</pre>
=== Services ===* Initially change services on the builderDisk /var/tmp/diskTLTF4t.img: 4294 MB, since ansible needs to gain access to the builder there are a few things that need to be done.4294967296 bytes* NetworkManager - If it is a static address255 heads, stop this service63 sectors/track, or if you have already setup DHCP on japan522 cylinders, start network managertotal 8388608 sectorsUnits = sectors of 1 * 512 = 512 bytes<pre>systemctl start NetworkManager<Sector size (logical/physical): 512 bytes /pre>or512 bytes<pre>systemctl stop NetworkManager<I/pre>* sshd - Start this service<pre>systemctl start sshd<O size (minimum/pre>* firewalld - Stop this service<pre>systemctl stop firewalld<optimal): 512 bytes /pre>512 bytes* selinux - Stop selinux for now as it interferes with ansible ssh<pre>setenforce 0</pre>Disk identifier: 0x0009c1ec
=== SSHD === Device Boot Start End Blocks Id System* Copy the file /etcvar/ansibletmp/builderdiskTLTF4t.img1 * 2048 104447 51200 c W95 FAT32 (LBA)/config_filesvar/authorized_keys from japan to the buildertmp/diskTLTF4t.img2 104448 4234064 2064808+ 83 Linux** This file contains public keys for users and ansibleFilesystem Size Used Avail Use% Mounted on<pre>scp /etcdev/ansiblemapper/builderloop0p2 2.0G 2.0G 0 100% /config_filestmp/authorized_keys root@builder:<tmp.v9ADZEWeP2removed `/pre>tmp/tmp.v9ADZEWeP2/resize-reserve'* Login to the builderFilesystem Size Used Avail Use% Mounted on<pre>ssh root@builder</pre>* Setup ssh and authorized keys<pre>mkdir dev/mapper/loop0p2 2.sshmv authorized_keys 0G 1.ssh8G 133M 93% /tmp/chmod 700 tmp.ssh/v9ADZEWeP2chmod 600 Image shrink completed.ssh/authorized_keys
</pre>
* Ansible should now have access to this builder
== Ansible Groups Preparing Pidora Noobs distribution ==This script extracts and compresses the boot and rootfs of a Pidora image. It also generates and updates all the necessary json files, images and release notes required by the Noobs distribution. Pidora 2014 Noobs Script: http://zenit.senecac.on.ca/wiki/index.php/Pidora-2014-Noobs-Script
The following ansible groups are used to change the type of configuration that each builder receives. Once each builder has been added to *Note this script uses PV (pipe viewer), if it's not already installed on the groups they should be in, run ansible and each group will get treated slightly differently, configuring all builderssystem please install it.
=== Group Structure ===Example Usage* makenoobs disk_image
The following is a structure of groups, this shows parent groups with child groups. == Testing Pidora Noobs ==
* buildersDownload the latest version of Noobs: http://www.raspberrypi.org/downloads/ (NOOBS Offline and network install)** builders_default*** trimslices*** arndales*** cubies*** specials** builders_nfs** builders_swap*** trimslices** builders_staticip***arndalesunzip the Noobs package then replace the files located in the os/Pidora directory with the files generated by the makenoobs script
The child groups link back to a list of hostnames <pre>NOOBs Pidora OS location: ./Noobs-Pidora/14-08-18./Noobs-Pidora/14-08-18├── boot.tar.xz├── os.json├── partition_setup.sh├── partitions.json├── Pidora.png├── release-notes.txt├── rootfs.tar.xz├── slides│ ├── A.png│ ├── B.png│ ├── C.png│ ├── D.png│ └── E.png└── slides_vga ├── A.png ├── B.png ├── C.png ├── D.png ├── E.png └── Pidora.png
* trimslices** tri-1-1** tri-1-2** tri-1-3** tri-1-4* cubies** cub-2-1directories, 21 files** cub-2-2* arndales** arn-3-1** arn-3-2* specials** arm-4-1** arm-4-2** arm-4-3** arm-4-4 === builders_default === This group is a default group to for all builders. All builders should be in this group. === builders_nfs === This group is used for nfs configuration. This was previously used on older builders that did not have hard drives and required more building space and speed. === builder_swap === This group will allow for ansible to generate a 4GB swap file on the builders and turn that swap file on. This is primarily used for builders that require more swap than is set up on their swap partitions. === builder_staticip === This group should be used for all builders that require static ip addresses. It will setup the custom ip address based on the resolved hostname inside the /etc/ansible</ansible_hosts file.pre>
Retrieved from "https://wiki.cdot.senecapolytechnic.ca/wiki/Special:MobileDiff/97174...108396"

Navigation menu