Revision as of 13:58, 6 March 2013

Koji Follow Que Script

Notes

This script does not automatically solve circular dependency issues (which there are a lot of, especially packages like sub perl modules)
This script does not automatically solve bootstrapping or library.so name bump requires issues (which happens often during a completely new tag rebuild)
This script does not attempt to solve any package build failure issues
This script can be run in 2 modes: "mass-rebuild" and "latest-follow" and ques builds in order based on their "creation_ts" value
- mass-rebuild mode will attempt to compile a complete list of packages that were built for a given tag and then que them in the given order
- latest-follow mode will perform a "list-tagged --latest" and que each unbuilt package in the given order

Common Repository Breakage Scenarios due to basic circular dependencies

All of the following examples take place after this series of events

Start of initial-build

libA-1.0-1.fc17 is built (Provides: libA.so.1)
packageB-1.0-1.fc17 (BuildRequires: libA,packageC) is temporarily built with its packageC requirement removed (Requires: libA.so.1)
packageC-1.0-1.fc17 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-1.0-1.fc17 and is built (Requires: libA.so.1)
packageB-1.0-2.fc17 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)

Building a new library first and the packages that depend on it later

Start of mass-rebuild

libA-2.0-1.fc18 is built (Provides: libA.so.2)
- Attempting to build packageB or packageC will BuildRequire each other and which both will then Require libA.so.1 which will then only find libA.so.2 in the current repodata

Building new packages first and then the following libraries they depend on after

Start of mass-rebuild

packageB-2.0-2.fc18 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)
packageC-2.0-1.fc18 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-2.0-2.fc18 and is built (Requires: libA.so.1)
libA-2.0-1.fc18 is built (Provides: libA.so.2)
- All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageB or packageC will require libA.so.1 which is not being provided for at all since the updated libA only provides libA.so.2

Downloads

Source

Example Config and Execution Scenarios

mass-rebuild mode

~/koji-follow.conf

primary_url = "http://koji.fedoraproject.org"
primary_arch = "x86_64"
#secondary_url = "http://arm.koji.fedoraproject.org"
#secondary_arch = "armhfp"

#primary_url = "http://arm.koji.fedoraproject.org"
#primary_arch = "armhfp"
secondary_url = "http://japan.proximity.on.ca"
secondary_arch = "armv6hl"

client_cert = os.path.expanduser("~/.fedora.cert")
server_cert = os.path.expanduser("~/.fedora-server-ca.cert")

target_arch = "arm"
tag_name = "f18"
que_limit = 80

excl_list = ["glibc","kernel","koji","redhat-rpm-config","rpm","yum"]
check_tag = "all"
retry_build = "all"

python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log

latest-follow mode

~/koji-follow.conf

primary_url = "http://koji.fedoraproject.org"
primary_arch = "x86_64"
#secondary_url = "http://arm.koji.fedoraproject.org"
#secondary_arch = "armhfp"

#primary_url = "http://arm.koji.fedoraproject.org"
#primary_arch = "armhfp"
secondary_url = "http://japan.proximity.on.ca"
secondary_arch = "armv6hl"

client_cert = os.path.expanduser("~/.fedora.cert")
server_cert = os.path.expanduser("~/.fedora-server-ca.cert")

target_arch = "arm"
tag_name = "f18"
que_limit = 80

excl_list = ["glibc","kernel","koji","redhat-rpm-config","rpm","yum"]
check_tag = "latest"
retry_build = "failed"

python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log

Example Output

STDOUT Messages

[info] pkg_process: [1/6604]: {'owner_name': 'jkeating', 'package_name': 'email2trac', 'task_id': 1503441, 'creation_event_id': 1793404, 'creation_time': '2009-07-25 14:59:59.646649', 'epoch': None, 'nvr': 'email2trac-0.13-6.fc12', 'name': 'email2trac', 'completion_time': '2009-07-25 15:35:37.427707', 'state': 1, 'version': '0.13', 'package_id': 6215, 'volume_name': 'DEFAULT', 'release': '6.fc12', 'creation_ts': 1248533999.64665, 'completion_ts': 1248536137.42771, 'id': 117394, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [email2trac-0.13-6.fc12.src.rpm] -> [cli-build/1362515245.29142.RuqZKocF]
[info] pkg_process: [2/6604]: {'owner_name': 'jkeating', 'package_name': 'powerpc-utils-papr', 'task_id': 1525540, 'creation_event_id': 1839140, 'creation_time': '2009-07-28 04:43:55.459368', 'epoch': None, 'nvr': 'powerpc-utils-papr-1.1.6-2.fc12', 'name': 'powerpc-utils-papr', 'completion_time': '2009-07-28 05:47:09.877729', 'state': 1, 'version': '1.1.6', 'package_id': 5369, 'volume_name': 'DEFAULT', 'release': '2.fc12', 'creation_ts': 1248756235.45937, 'completion_ts': 1248760029.87773, 'id': 122145, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [powerpc-utils-papr-1.1.6-2.fc12.src.rpm] -> [cli-build/1362515248.044621.imzInFfA]
[info] pkg_process: [3/6604]: {'owner_name': 'jkeating', 'package_name': 'ppc64-utils', 'task_id': 1525552, 'creation_event_id': 1839195, 'creation_time': '2009-07-28 04:46:54.025335', 'epoch': None, 'nvr': 'ppc64-utils-0.14-5.fc12', 'name': 'ppc64-utils', 'completion_time': '2009-07-28 05:51:18.860184', 'state': 1, 'version': '0.14', 'package_id': 131, 'volume_name': 'DEFAULT', 'release': '5.fc12', 'creation_ts': 1248756414.02534, 'completion_ts': 1248760278.86018, 'id': 122152, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [ppc64-utils-0.14-5.fc12.src.rpm] -> [cli-build/1362515249.622192.vcbTPtpj]
[info] pkg_process: [4/6604]: {'owner_name': 'rrakus', 'package_name': 'iprutils', 'task_id': 1787705, 'creation_event_id': 2291747, 'creation_time': '2009-11-04 05:55:36.885871', 'epoch': None, 'nvr': 'iprutils-2.2.18-1.fc12', 'name': 'iprutils', 'completion_time': '2009-11-04 05:58:25.827132', 'state': 1, 'version': '2.2.18', 'package_id': 722, 'volume_name': 'DEFAULT', 'release': '1.fc12', 'creation_ts': 1257314136.88587, 'completion_ts': 1257314305.82713, 'id': 139717, 'volume_id': 0, 'owner_id': 613}
        [info] que_build: [iprutils-2.2.18-1.fc12.src.rpm] -> [cli-build/1362515252.116506.kQNgDlWL]
[info] pkg_process: [5/6604]: {'owner_name': 'sharkcz', 'package_name': 'gtksourceview2-sharp', 'task_id': 3035160, 'creation_event_id': 3682022, 'creation_time': '2011-04-29 07:07:04.359159', 'epoch': None, 'nvr': 'gtksourceview2-sharp-1.0-10.svn89788.fc16', 'name': 'gtksourceview2-sharp', 'completion_time': '2011-04-29 07:09:55.860531', 'state': 1, 'version': '1.0', 'package_id': 6142, 'volume_name': 'DEFAULT', 'release': '10.svn89788.fc16', 'creation_ts': 1304060824.35916, 'completion_ts': 1304060995.86053, 'id': 241357, 'volume_id': 0, 'owner_id': 244}
        [info] que_build: [gtksourceview2-sharp-1.0-10.svn89788.fc16.src.rpm] -> [cli-build/1362515254.528944.WTNbsaqJ]
...

STDERR Messages

[info] primary_tag: [1188/1188/1584]: rwho
[info] primary_tag: [396/396/792]: jlatexmath
[info] primary_tag: [0/0/396]: fprintd
[info] primary_tag: [1584/1584/1980]: objectweb-asm4
[info] primary_tag: [1980/1980/2376]: ktp-approver
[info] primary_tag: [792/792/1188]: nginx
[info] primary_tag: [3168/3168/3564]: audex
[info] primary_tag: [2376/2376/2772]: python-sh
[info] primary_tag: [3564/3564/3960]: python-cmd2
[info] primary_tag: [2772/2772/3168]: perl-IPC-Run-SafeHandles
...
        [info] file_delete: aplus-fsf-4.22.4-21.fc16.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/aplus-fsf/4.22.4/21.fc16/src/aplus-fsf-4.22.4-21.fc16.src.rpm] -> [aplus-fsf-4.22.4-21.fc16.src.rpm]
        [info] file_delete: tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/tritonus/0.3.7/0.13.20101108cvs.fc17/src/tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm] -> [tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm]
        [info] file_delete: jffi-1.0.10-1.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/jffi/1.0.10/1.fc17/src/jffi-1.0.10-1.fc17.src.rpm] -> [jffi-1.0.10-1.fc17.src.rpm]
        [info] file_delete: nautilus-sendto-trac-0.4-2.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/nautilus-sendto-trac/0.4/2.fc17/src/nautilus-sendto-trac-0.4-2.fc17.src.rpm] -> [nautilus-sendto-trac-0.4-2.fc17.src.rpm]
        [info] file_delete: sweep-0.9.3-9.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/sweep/0.9.3/9.fc17/src/sweep-0.9.3-9.fc17.src.rpm] -> [sweep-0.9.3-9.fc17.src.rpm]
...

@@ Line 16: / Line 16: @@
 * Start of initial-build
 # libA-1.0-1.fc17 is built (Provides: libA.so.1)
-# packageB-1.0-1.fc17 has a requires on packageC but is bootstrapped and built without it
+# packageB-1.0-1.fc17 (BuildRequires: libA,packageC) is temporarily built with its packageC requirement removed (Requires: libA.so.1)
-# packageC-1.0-1.fc17 has a requires on libA,packageB and is built with libA-1.0-1.fc17,packageB-1.0-1.fc17 (Requires: libA.so.1,packageB)
+# packageC-1.0-1.fc17 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-1.0-1.fc17 and is built (Requires: libA.so.1)
-# packageB-1.0-2.fc17 has a requires on packageC and is built with packageC-1.0-1.fc17,libA-1.0-1.fc17 (Requires: packageC)
+# packageB-1.0-2.fc17 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)
 === Building a new library first and the packages that depend on it later ===
@@ Line 24: / Line 24: @@
 * Start of mass-rebuild
 # libA-2.0-1.fc18 is built (Provides: libA.so.2)
-#* Attempting to build packageB-2.0-2.fc18 will require packageC-1.0-1.fc17 which requires libA.so.1 which only finds libA.so.2 given the current repodata (this package needs to be bootstrapped again)
+#* Attempting to build packageB or packageC will BuildRequire each other and which both will then Require libA.so.1 which will then only find libA.so.2 in the current repodata
-#* Attempting to build packageC-2.0-1.fc18 will require packageB-1.0-2.fc17 which requires packageC-1.0-1.fc17 which brings us to the same breakage error as shown above
 === Building new packages first and then the following libraries they depend on after ===
 * Start of mass-rebuild
-# packageB-2.0-2.fc18 has a requires on packageC and is built with packageC-1.0-1.fc17 which has a requires on libA,packageB which installs libA-1.0-1.fc17 and packageB-1.0-2.fc17 (Requires: packageC)
+# packageB-2.0-2.fc18 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)
-# packageC-2.0-1.fc18 has a requires on libA,packageB and is built with libA-1.0-1.fc17,packageB-2.0-1.fc18 which has a requires on packageC which installs packageC-1.0-1.fc17 (Requires: libA.so.1,packageB)
+# packageC-2.0-1.fc18 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-2.0-2.fc18 and is built (Requires: libA.so.1)
 # libA-2.0-1.fc18 is built (Provides: libA.so.2)
-#* All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageC will require libA.so.1 which is not being provided for at all since the updated libA-2.0-1.fc18 provides libA.so.2
+#* All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageB or packageC will require libA.so.1 which is not being provided for at all since the updated libA only provides libA.so.2
 = Downloads =

Difference between revisions of "Koji Follow"

Revision as of 13:58, 6 March 2013

Contents

Koji Follow Que Script

Notes

Common Repository Breakage Scenarios due to basic circular dependencies

All of the following examples take place after this series of events

Building a new library first and the packages that depend on it later

Building new packages first and then the following libraries they depend on after

Downloads

Source

Example Config and Execution Scenarios

mass-rebuild mode

latest-follow mode

Example Output

STDOUT Messages

STDERR Messages

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

get involved with CDOT

courses

course projects

links

Tools