Koji Follow

From CDOT Wiki
Revision as of 16:14, 9 April 2013 by Fossjon (talk | contribs) (→‎Example Output)
Jump to: navigation, search

Koji Follow Que Script

Notes

  • This script does not automatically solve circular dependency issues (which there are a lot of, especially packages like sub perl modules)
  • This script does not automatically solve bootstrapping or library.so name bump requires issues (which happens often during a completely new tag rebuild)
  • This script does not attempt to solve any package build failure issues
  • This script can be run in 2 modes: "mass-rebuild" and "latest-follow" and ques builds in order based on their "creation_ts" value
    • mass-rebuild mode will attempt to compile a complete list of packages that were built for a given tag and then que them in the given order
    • latest-follow mode will perform a "list-tagged --latest" and que each unbuilt package in the given order

Common Repository Breakage Scenarios due to basic circular dependencies

All of the following examples take place after this series of events

  • Start of initial-build
  1. libA-1.0-1.fc17 is built (Provides: libA.so.1)
  2. packageB-1.0-1.fc17 (BuildRequires: libA,packageC) is temporarily built with its packageC requirement removed (Requires: libA.so.1)
  3. packageC-1.0-1.fc17 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-1.0-1.fc17 and is built (Requires: libA.so.1)
  4. packageB-1.0-2.fc17 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)

Building a new library first and the packages that depend on it later

  • Start of mass-rebuild
  1. libA-2.0-1.fc18 is built (Provides: libA.so.2)
    • Attempting to build packageB or packageC will BuildRequire each other and which both will then Require libA.so.1 which will then only find libA.so.2 in the current repodata

Building new packages first and then the following libraries they depend on after

  • Start of mass-rebuild
  1. packageB-2.0-2.fc18 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)
  2. packageC-2.0-1.fc18 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-2.0-2.fc18 and is built (Requires: libA.so.1)
  3. libA-2.0-1.fc18 is built (Provides: libA.so.2)
    • All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageB or packageC will require libA.so.1 which is not being provided for at all since the updated libA only provides libA.so.2

Downloads

Source

Example Config and Execution Scenarios

mass-rebuild mode

~/koji-follow.conf 

primary_url = "http://koji.fedoraproject.org"
primary_arch = "x86_64"
#secondary_url = "http://arm.koji.fedoraproject.org"
#secondary_arch = "armhfp"

#primary_url = "http://arm.koji.fedoraproject.org"
#primary_arch = "armhfp"
secondary_url = "http://japan.proximity.on.ca"
secondary_arch = "armv6hl"

client_cert = os.path.expanduser("~/.fedora.cert")
server_cert = os.path.expanduser("~/.fedora-server-ca.cert")

target_arch = "arm"
tag_name = "f18"
que_limit = 80

excl_list = ["glibc","kernel","koji","redhat-rpm-config","rpm","yum"]
check_tag = "all"
retry_build = "all"

python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log

latest-follow mode

~/koji-follow.conf 

primary_url = "http://koji.fedoraproject.org"
primary_arch = "x86_64"
#secondary_url = "http://arm.koji.fedoraproject.org"
#secondary_arch = "armhfp"

#primary_url = "http://arm.koji.fedoraproject.org"
#primary_arch = "armhfp"
secondary_url = "http://japan.proximity.on.ca"
secondary_arch = "armv6hl"

client_cert = os.path.expanduser("~/.fedora.cert")
server_cert = os.path.expanduser("~/.fedora-server-ca.cert")

target_arch = "arm"
tag_name = "f18"
que_limit = 80

excl_list = ["glibc","kernel","koji","redhat-rpm-config","rpm","yum"]
check_tag = "latest"
retry_build = "failed"

Execution Example

python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log

Example Log Output

STDOUT Messages

[info] pkg_process: [1/6604]: {'owner_name': 'jkeating', 'package_name': 'email2trac', 'task_id': 1503441, 'creation_event_id': 1793404, 'creation_time': '2009-07-25 14:59:59.646649', 'epoch': None, 'nvr': 'email2trac-0.13-6.fc12', 'name': 'email2trac', 'completion_time': '2009-07-25 15:35:37.427707', 'state': 1, 'version': '0.13', 'package_id': 6215, 'volume_name': 'DEFAULT', 'release': '6.fc12', 'creation_ts': 1248533999.64665, 'completion_ts': 1248536137.42771, 'id': 117394, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [email2trac-0.13-6.fc12.src.rpm] -> [cli-build/1362515245.29142.RuqZKocF]
[info] pkg_process: [2/6604]: {'owner_name': 'jkeating', 'package_name': 'powerpc-utils-papr', 'task_id': 1525540, 'creation_event_id': 1839140, 'creation_time': '2009-07-28 04:43:55.459368', 'epoch': None, 'nvr': 'powerpc-utils-papr-1.1.6-2.fc12', 'name': 'powerpc-utils-papr', 'completion_time': '2009-07-28 05:47:09.877729', 'state': 1, 'version': '1.1.6', 'package_id': 5369, 'volume_name': 'DEFAULT', 'release': '2.fc12', 'creation_ts': 1248756235.45937, 'completion_ts': 1248760029.87773, 'id': 122145, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [powerpc-utils-papr-1.1.6-2.fc12.src.rpm] -> [cli-build/1362515248.044621.imzInFfA]
[info] pkg_process: [3/6604]: {'owner_name': 'jkeating', 'package_name': 'ppc64-utils', 'task_id': 1525552, 'creation_event_id': 1839195, 'creation_time': '2009-07-28 04:46:54.025335', 'epoch': None, 'nvr': 'ppc64-utils-0.14-5.fc12', 'name': 'ppc64-utils', 'completion_time': '2009-07-28 05:51:18.860184', 'state': 1, 'version': '0.14', 'package_id': 131, 'volume_name': 'DEFAULT', 'release': '5.fc12', 'creation_ts': 1248756414.02534, 'completion_ts': 1248760278.86018, 'id': 122152, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [ppc64-utils-0.14-5.fc12.src.rpm] -> [cli-build/1362515249.622192.vcbTPtpj]
[info] pkg_process: [4/6604]: {'owner_name': 'rrakus', 'package_name': 'iprutils', 'task_id': 1787705, 'creation_event_id': 2291747, 'creation_time': '2009-11-04 05:55:36.885871', 'epoch': None, 'nvr': 'iprutils-2.2.18-1.fc12', 'name': 'iprutils', 'completion_time': '2009-11-04 05:58:25.827132', 'state': 1, 'version': '2.2.18', 'package_id': 722, 'volume_name': 'DEFAULT', 'release': '1.fc12', 'creation_ts': 1257314136.88587, 'completion_ts': 1257314305.82713, 'id': 139717, 'volume_id': 0, 'owner_id': 613}
        [info] que_build: [iprutils-2.2.18-1.fc12.src.rpm] -> [cli-build/1362515252.116506.kQNgDlWL]
[info] pkg_process: [5/6604]: {'owner_name': 'sharkcz', 'package_name': 'gtksourceview2-sharp', 'task_id': 3035160, 'creation_event_id': 3682022, 'creation_time': '2011-04-29 07:07:04.359159', 'epoch': None, 'nvr': 'gtksourceview2-sharp-1.0-10.svn89788.fc16', 'name': 'gtksourceview2-sharp', 'completion_time': '2011-04-29 07:09:55.860531', 'state': 1, 'version': '1.0', 'package_id': 6142, 'volume_name': 'DEFAULT', 'release': '10.svn89788.fc16', 'creation_ts': 1304060824.35916, 'completion_ts': 1304060995.86053, 'id': 241357, 'volume_id': 0, 'owner_id': 244}
        [info] que_build: [gtksourceview2-sharp-1.0-10.svn89788.fc16.src.rpm] -> [cli-build/1362515254.528944.WTNbsaqJ]
...

STDERR Messages

[info] primary_tag: [1188/1188/1584]: rwho
[info] primary_tag: [396/396/792]: jlatexmath
[info] primary_tag: [0/0/396]: fprintd
[info] primary_tag: [1584/1584/1980]: objectweb-asm4
[info] primary_tag: [1980/1980/2376]: ktp-approver
[info] primary_tag: [792/792/1188]: nginx
[info] primary_tag: [3168/3168/3564]: audex
[info] primary_tag: [2376/2376/2772]: python-sh
[info] primary_tag: [3564/3564/3960]: python-cmd2
[info] primary_tag: [2772/2772/3168]: perl-IPC-Run-SafeHandles
...
        [info] file_delete: aplus-fsf-4.22.4-21.fc16.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/aplus-fsf/4.22.4/21.fc16/src/aplus-fsf-4.22.4-21.fc16.src.rpm] -> [aplus-fsf-4.22.4-21.fc16.src.rpm]
        [info] file_delete: tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/tritonus/0.3.7/0.13.20101108cvs.fc17/src/tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm] -> [tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm]
        [info] file_delete: jffi-1.0.10-1.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/jffi/1.0.10/1.fc17/src/jffi-1.0.10-1.fc17.src.rpm] -> [jffi-1.0.10-1.fc17.src.rpm]
        [info] file_delete: nautilus-sendto-trac-0.4-2.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/nautilus-sendto-trac/0.4/2.fc17/src/nautilus-sendto-trac-0.4-2.fc17.src.rpm] -> [nautilus-sendto-trac-0.4-2.fc17.src.rpm]
        [info] file_delete: sweep-0.9.3-9.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/sweep/0.9.3/9.fc17/src/sweep-0.9.3-9.fc17.src.rpm] -> [sweep-0.9.3-9.fc17.src.rpm]
...
Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=Koji_Follow&oldid=95110"