Open main menu

CDOT Wiki β

Koji Follow

Revision as of 18:51, 5 March 2013 by Fossjon (talk | contribs) (→‎Possible Breakage Scenarios)

Koji Follow Que Script

Notes

  • This script does not automatically solve circular dependency issues (which there are a lot of, especially packages like sub perl modules)
  • This script does not automatically solve bootstrapping or library.so name bump requires issues (which happens often during a completely new tag rebuild)
  • This script does not attempt to solve any package build failure issues
  • This script can be run in 2 modes: "mass-rebuild" and "latest-follow" and ques builds in order based on their "creation_ts" value
    • mass-rebuild mode will attempt to compile a complete list of packages that were built for a given tag and then que them in the given order
    • latest-follow mode will perform a "list-tagged --latest" and que each unbuilt package in the given order

Possible Breakage Scenarios

Scenario 1

  1. libA-1.0-1.fc17 is built (Provides: libA.so.1)
  2. packageB-1.0-1.fc17 has a BuildRequires: packageC but is bootstrapped without it and built
  3. packageC-1.0-1.fc17 has a BuildRequires: libA,packageB and is built with libA-1.0-1.fc17,packageB-1.0-1.fc17 (Requires: libA.so.1)
  4. packageB-1.0-2.fc17 has a BuildRequires: packageC and is built with packageC-1.0-1.fc17,libA-1.0-1.fc17,packageB-1.0-1.fc17
  5. libA-2.0-1.fc18 is built (Provides: libA.so.2)
    1. Attempting to build the full version of packageB-2.0-1.fc18 will depend on packageC-1.0-1.fc17 which requires libA.so.1 which only find libA.so.2 with the current repodata (this package needs to be bootstrapped again)
    2. Attempting to build packageC-2.0-1.fc18 will depend on packageB-1.0-2.fc17 which will depend on packageC-1.0-1.fc17 which brings us to the same breakage error above

Scenario 2

  1. libA-1.0-1.fc17 is built (Provides: libA.so.1)
  2. packageB-1.0-1.fc17 has a BuildRequires: packageC but is bootstrapped without it and built
  3. packageC-1.0-1.fc17 has a BuildRequires: libA,packageB and is built with libA-1.0-1.fc17,packageB-1.0-1.fc17 (Requires: libA.so.1)
  4. packageB-1.0-2.fc17 has a BuildRequires: packageC and is built with packageC-1.0-1.fc17,libA-1.0-1.fc17,packageB-1.0-1.fc17
  5. packageB-2.0-1.fc18 has a BuildRequires: packageC and is built with packageC-1.0-1.fc17,libA-1.0-1.fc17,packageB-1.0-2.fc17
  6. packageC-2.0-1.fc18 has a BuildRequires: libA,packageB and is built with libA-1.0-1.fc17,packageB-2.0-1.fc18,packageC-1.0-1.fc17 (Requires: libA.so.1)
  7. libA-2.0-1.fc18 is built (Provides: libA.so.2)
    1. All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageC will require libA.so.1 which is not being provided for since the updated libA only provides libA.so.2

Scenario 3

  1. libA-1.0-1.fc17 is built (Provides: libA.so.1)
  2. packageB-1.0-1.fc17 has a BuildRequires: packageC but is bootstrapped without it and built
  3. packageC-1.0-1.fc17 has a BuildRequires: libA,packageB and is built with libA-1.0-1.fc17,packageB-1.0-1.fc17 (Requires: libA.so.1)
  4. packageC-2.0-1.fc18 has a BuildRequires: libA,packageB and is built with libA-1.0-1.fc17,packageB-1.0-1.fc17 (Requires: libA.so.1)
  5. libA-2.0-1.fc18 is built (Provides: libA.so.2)
    1. packageB-2.0-1.fc18 depends on packageC which requires packageC-2.0-1.fc18,libA-2.0-1.fc18,packageB-1.0-1.fc17 which requires libA.so.1 which only finds libA.so.2 given the current repodata

Scenario 4

  1. libA-1.0-1.fc17 is built (Provides: libA.so.1)
  2. packageB-1.0-1.fc17 has a BuildRequires: packageC but is bootstrapped without it and built
  3. packageC-1.0-1.fc17 has a BuildRequires: libA,packageB and is built with libA-1.0-1.fc17,packageB-1.0-1.fc17 (Requires: libA.so.1)
  4. packageB-2.0-1.fc18 has a BuildRequires: packageC and is built with packageC-1.0-1.fc17,libA-1.0-1.fc17,packageB-1.0-1.fc17
  5. libA-2.0-1.fc18 is built (Provides: libA.so.2)
    1. packageC-2.0-1.fc18 has a BuildRequires: libA,packageB which finds libA-2.0-1.fc18,packageB-2.0-1.fc18,packageC-1.0-1.fc17 which requires libA.so.1 which only finds the latest libA.so.2 in the repodata

Downloads

Example Config and Execution Scenarios

mass-rebuild mode

~/koji-follow.conf 

primary_url = "http://koji.fedoraproject.org"
primary_arch = "x86_64"
#secondary_url = "http://arm.koji.fedoraproject.org"
#secondary_arch = "armhfp"

#primary_url = "http://arm.koji.fedoraproject.org"
#primary_arch = "armhfp"
secondary_url = "http://japan.proximity.on.ca"
secondary_arch = "armv6hl"

client_cert = os.path.expanduser("~/.fedora.cert")
server_cert = os.path.expanduser("~/.fedora-server-ca.cert")

target_arch = "arm"
tag_name = "f18"
que_limit = 80

excl_list = ["glibc","kernel","koji","redhat-rpm-config","rpm","yum"]
check_tag = "all"
retry_build = "all"

python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log

latest-follow mode

~/koji-follow.conf 

primary_url = "http://koji.fedoraproject.org"
primary_arch = "x86_64"
#secondary_url = "http://arm.koji.fedoraproject.org"
#secondary_arch = "armhfp"

#primary_url = "http://arm.koji.fedoraproject.org"
#primary_arch = "armhfp"
secondary_url = "http://japan.proximity.on.ca"
secondary_arch = "armv6hl"

client_cert = os.path.expanduser("~/.fedora.cert")
server_cert = os.path.expanduser("~/.fedora-server-ca.cert")

target_arch = "arm"
tag_name = "f18"
que_limit = 80

excl_list = ["glibc","kernel","koji","redhat-rpm-config","rpm","yum"]
check_tag = "latest"
retry_build = "failed"

python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log

Example Output

STDOUT Messages

[info] pkg_process: [1/6604]: {'owner_name': 'jkeating', 'package_name': 'email2trac', 'task_id': 1503441, 'creation_event_id': 1793404, 'creation_time': '2009-07-25 14:59:59.646649', 'epoch': None, 'nvr': 'email2trac-0.13-6.fc12', 'name': 'email2trac', 'completion_time': '2009-07-25 15:35:37.427707', 'state': 1, 'version': '0.13', 'package_id': 6215, 'volume_name': 'DEFAULT', 'release': '6.fc12', 'creation_ts': 1248533999.64665, 'completion_ts': 1248536137.42771, 'id': 117394, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [email2trac-0.13-6.fc12.src.rpm] -> [cli-build/1362515245.29142.RuqZKocF]
[info] pkg_process: [2/6604]: {'owner_name': 'jkeating', 'package_name': 'powerpc-utils-papr', 'task_id': 1525540, 'creation_event_id': 1839140, 'creation_time': '2009-07-28 04:43:55.459368', 'epoch': None, 'nvr': 'powerpc-utils-papr-1.1.6-2.fc12', 'name': 'powerpc-utils-papr', 'completion_time': '2009-07-28 05:47:09.877729', 'state': 1, 'version': '1.1.6', 'package_id': 5369, 'volume_name': 'DEFAULT', 'release': '2.fc12', 'creation_ts': 1248756235.45937, 'completion_ts': 1248760029.87773, 'id': 122145, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [powerpc-utils-papr-1.1.6-2.fc12.src.rpm] -> [cli-build/1362515248.044621.imzInFfA]
[info] pkg_process: [3/6604]: {'owner_name': 'jkeating', 'package_name': 'ppc64-utils', 'task_id': 1525552, 'creation_event_id': 1839195, 'creation_time': '2009-07-28 04:46:54.025335', 'epoch': None, 'nvr': 'ppc64-utils-0.14-5.fc12', 'name': 'ppc64-utils', 'completion_time': '2009-07-28 05:51:18.860184', 'state': 1, 'version': '0.14', 'package_id': 131, 'volume_name': 'DEFAULT', 'release': '5.fc12', 'creation_ts': 1248756414.02534, 'completion_ts': 1248760278.86018, 'id': 122152, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [ppc64-utils-0.14-5.fc12.src.rpm] -> [cli-build/1362515249.622192.vcbTPtpj]
[info] pkg_process: [4/6604]: {'owner_name': 'rrakus', 'package_name': 'iprutils', 'task_id': 1787705, 'creation_event_id': 2291747, 'creation_time': '2009-11-04 05:55:36.885871', 'epoch': None, 'nvr': 'iprutils-2.2.18-1.fc12', 'name': 'iprutils', 'completion_time': '2009-11-04 05:58:25.827132', 'state': 1, 'version': '2.2.18', 'package_id': 722, 'volume_name': 'DEFAULT', 'release': '1.fc12', 'creation_ts': 1257314136.88587, 'completion_ts': 1257314305.82713, 'id': 139717, 'volume_id': 0, 'owner_id': 613}
        [info] que_build: [iprutils-2.2.18-1.fc12.src.rpm] -> [cli-build/1362515252.116506.kQNgDlWL]
[info] pkg_process: [5/6604]: {'owner_name': 'sharkcz', 'package_name': 'gtksourceview2-sharp', 'task_id': 3035160, 'creation_event_id': 3682022, 'creation_time': '2011-04-29 07:07:04.359159', 'epoch': None, 'nvr': 'gtksourceview2-sharp-1.0-10.svn89788.fc16', 'name': 'gtksourceview2-sharp', 'completion_time': '2011-04-29 07:09:55.860531', 'state': 1, 'version': '1.0', 'package_id': 6142, 'volume_name': 'DEFAULT', 'release': '10.svn89788.fc16', 'creation_ts': 1304060824.35916, 'completion_ts': 1304060995.86053, 'id': 241357, 'volume_id': 0, 'owner_id': 244}
        [info] que_build: [gtksourceview2-sharp-1.0-10.svn89788.fc16.src.rpm] -> [cli-build/1362515254.528944.WTNbsaqJ]
...

STDERR Messages

[info] primary_tag: [1188/1188/1584]: rwho
[info] primary_tag: [396/396/792]: jlatexmath
[info] primary_tag: [0/0/396]: fprintd
[info] primary_tag: [1584/1584/1980]: objectweb-asm4
[info] primary_tag: [1980/1980/2376]: ktp-approver
[info] primary_tag: [792/792/1188]: nginx
[info] primary_tag: [3168/3168/3564]: audex
[info] primary_tag: [2376/2376/2772]: python-sh
[info] primary_tag: [3564/3564/3960]: python-cmd2
[info] primary_tag: [2772/2772/3168]: perl-IPC-Run-SafeHandles
...
        [info] file_delete: aplus-fsf-4.22.4-21.fc16.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/aplus-fsf/4.22.4/21.fc16/src/aplus-fsf-4.22.4-21.fc16.src.rpm] -> [aplus-fsf-4.22.4-21.fc16.src.rpm]
        [info] file_delete: tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/tritonus/0.3.7/0.13.20101108cvs.fc17/src/tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm] -> [tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm]
        [info] file_delete: jffi-1.0.10-1.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/jffi/1.0.10/1.fc17/src/jffi-1.0.10-1.fc17.src.rpm] -> [jffi-1.0.10-1.fc17.src.rpm]
        [info] file_delete: nautilus-sendto-trac-0.4-2.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/nautilus-sendto-trac/0.4/2.fc17/src/nautilus-sendto-trac-0.4-2.fc17.src.rpm] -> [nautilus-sendto-trac-0.4-2.fc17.src.rpm]
        [info] file_delete: sweep-0.9.3-9.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/sweep/0.9.3/9.fc17/src/sweep-0.9.3-9.fc17.src.rpm] -> [sweep-0.9.3-9.fc17.src.rpm]
...
Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=Koji_Follow&oldid=94370"