Difference between revisions of "Koji Follow"

From CDOT Wiki
Jump to: navigation, search
(→‎Example Output)
Line 33: Line 33:
 
# libA-2.0-1.fc18 is built (Provides: libA.so.2)
 
# libA-2.0-1.fc18 is built (Provides: libA.so.2)
 
#* All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageB or packageC will require libA.so.1 which is not being provided for at all since the updated libA only provides libA.so.2
 
#* All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageB or packageC will require libA.so.1 which is not being provided for at all since the updated libA only provides libA.so.2
 +
 +
== Run Notes ==
 +
 +
* Packages are not processed in alphabetical order
 +
* You need to add any new package names to a tag before they are built
 +
* You need a lot of free space in the current directory that the script is running in
  
 
= Downloads =
 
= Downloads =
Line 70: Line 76:
 
</pre>
 
</pre>
  
−
<b>python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log</b>
+
<b>python ~/kf/koji-follow.py ~/kf/koji-follow.conf > ~/kf/kfo.log 2> ~/kf/kfe.log</b>
  
 
== latest-follow mode ==
 
== latest-follow mode ==
Line 101: Line 107:
 
== Execution Example ==
 
== Execution Example ==
  
−
<b>python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log</b>
+
<b>python ~/kf/koji-follow.py ~/kf/koji-follow.conf > ~/kf/kfo.log 2> ~/kf/kfe.log</b>
  
 
= Example Log Output =
 
= Example Log Output =

Revision as of 09:08, 8 July 2013

Koji Follow Que Script

Notes

  • This script does not automatically solve circular dependency issues (which there are a lot of, especially packages like sub perl modules)
  • This script does not automatically solve bootstrapping or library.so name bump requires issues (which happens often during a completely new tag rebuild)
  • This script does not attempt to solve any package build failure issues
  • This script can be run in 2 modes: "mass-rebuild" and "latest-follow" and ques builds in order based on their "creation_ts" value
    • mass-rebuild mode will attempt to compile a complete list of packages that were built for a given tag and then que them in the given order
    • latest-follow mode will perform a "list-tagged --latest" and que each unbuilt package in the given order

Common Repository Breakage Scenarios due to basic circular dependencies

All of the following examples take place after this series of events

  • Start of initial-build
  1. libA-1.0-1.fc17 is built (Provides: libA.so.1)
  2. packageB-1.0-1.fc17 (BuildRequires: libA,packageC) is temporarily built with its packageC requirement removed (Requires: libA.so.1)
  3. packageC-1.0-1.fc17 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-1.0-1.fc17 and is built (Requires: libA.so.1)
  4. packageB-1.0-2.fc17 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)

Building a new library first and the packages that depend on it later

  • Start of mass-rebuild
  1. libA-2.0-1.fc18 is built (Provides: libA.so.2)
    • Attempting to build packageB or packageC will BuildRequire each other and which both will then Require libA.so.1 which will then only find libA.so.2 in the current repodata

Building new packages first and then the following libraries they depend on after

  • Start of mass-rebuild
  1. packageB-2.0-2.fc18 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)
  2. packageC-2.0-1.fc18 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-2.0-2.fc18 and is built (Requires: libA.so.1)
  3. libA-2.0-1.fc18 is built (Provides: libA.so.2)
    • All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageB or packageC will require libA.so.1 which is not being provided for at all since the updated libA only provides libA.so.2

Run Notes

  • Packages are not processed in alphabetical order
  • You need to add any new package names to a tag before they are built
  • You need a lot of free space in the current directory that the script is running in

Downloads

Source

Example Config and Execution Scenarios

mass-rebuild mode

~/koji-follow.conf 

primary_url = "http://koji.fedoraproject.org"
primary_arch = "x86_64"
#secondary_url = "http://arm.koji.fedoraproject.org"
#secondary_arch = "armhfp"

#primary_url = "http://arm.koji.fedoraproject.org"
#primary_arch = "armhfp"
secondary_url = "http://japan.proximity.on.ca"
secondary_arch = "armv6hl"

client_cert = os.path.expanduser("~/.fedora.cert")
server_cert = os.path.expanduser("~/.fedora-server-ca.cert")

target_arch = "arm"
tag_name = "f18"
que_limit = 80

excl_list = ["glibc","kernel","koji","redhat-rpm-config","rpm","yum"]
check_tag = "all"
retry_build = "all"

python ~/kf/koji-follow.py ~/kf/koji-follow.conf > ~/kf/kfo.log 2> ~/kf/kfe.log

latest-follow mode

~/koji-follow.conf 

primary_url = "http://koji.fedoraproject.org"
primary_arch = "x86_64"
#secondary_url = "http://arm.koji.fedoraproject.org"
#secondary_arch = "armhfp"

#primary_url = "http://arm.koji.fedoraproject.org"
#primary_arch = "armhfp"
secondary_url = "http://japan.proximity.on.ca"
secondary_arch = "armv6hl"

client_cert = os.path.expanduser("~/.fedora.cert")
server_cert = os.path.expanduser("~/.fedora-server-ca.cert")

target_arch = "arm"
tag_name = "f18"
que_limit = 80

excl_list = ["glibc","kernel","koji","redhat-rpm-config","rpm","yum"]
check_tag = "latest"
retry_build = "failed"

Execution Example

python ~/kf/koji-follow.py ~/kf/koji-follow.conf > ~/kf/kfo.log 2> ~/kf/kfe.log

Example Log Output

STDOUT Messages

[info] pkg_process: [1/6604]: {'owner_name': 'jkeating', 'package_name': 'email2trac', 'task_id': 1503441, 'creation_event_id': 1793404, 'creation_time': '2009-07-25 14:59:59.646649', 'epoch': None, 'nvr': 'email2trac-0.13-6.fc12', 'name': 'email2trac', 'completion_time': '2009-07-25 15:35:37.427707', 'state': 1, 'version': '0.13', 'package_id': 6215, 'volume_name': 'DEFAULT', 'release': '6.fc12', 'creation_ts': 1248533999.64665, 'completion_ts': 1248536137.42771, 'id': 117394, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [email2trac-0.13-6.fc12.src.rpm] -> [cli-build/1362515245.29142.RuqZKocF]
[info] pkg_process: [2/6604]: {'owner_name': 'jkeating', 'package_name': 'powerpc-utils-papr', 'task_id': 1525540, 'creation_event_id': 1839140, 'creation_time': '2009-07-28 04:43:55.459368', 'epoch': None, 'nvr': 'powerpc-utils-papr-1.1.6-2.fc12', 'name': 'powerpc-utils-papr', 'completion_time': '2009-07-28 05:47:09.877729', 'state': 1, 'version': '1.1.6', 'package_id': 5369, 'volume_name': 'DEFAULT', 'release': '2.fc12', 'creation_ts': 1248756235.45937, 'completion_ts': 1248760029.87773, 'id': 122145, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [powerpc-utils-papr-1.1.6-2.fc12.src.rpm] -> [cli-build/1362515248.044621.imzInFfA]
[info] pkg_process: [3/6604]: {'owner_name': 'jkeating', 'package_name': 'ppc64-utils', 'task_id': 1525552, 'creation_event_id': 1839195, 'creation_time': '2009-07-28 04:46:54.025335', 'epoch': None, 'nvr': 'ppc64-utils-0.14-5.fc12', 'name': 'ppc64-utils', 'completion_time': '2009-07-28 05:51:18.860184', 'state': 1, 'version': '0.14', 'package_id': 131, 'volume_name': 'DEFAULT', 'release': '5.fc12', 'creation_ts': 1248756414.02534, 'completion_ts': 1248760278.86018, 'id': 122152, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [ppc64-utils-0.14-5.fc12.src.rpm] -> [cli-build/1362515249.622192.vcbTPtpj]
[info] pkg_process: [4/6604]: {'owner_name': 'rrakus', 'package_name': 'iprutils', 'task_id': 1787705, 'creation_event_id': 2291747, 'creation_time': '2009-11-04 05:55:36.885871', 'epoch': None, 'nvr': 'iprutils-2.2.18-1.fc12', 'name': 'iprutils', 'completion_time': '2009-11-04 05:58:25.827132', 'state': 1, 'version': '2.2.18', 'package_id': 722, 'volume_name': 'DEFAULT', 'release': '1.fc12', 'creation_ts': 1257314136.88587, 'completion_ts': 1257314305.82713, 'id': 139717, 'volume_id': 0, 'owner_id': 613}
        [info] que_build: [iprutils-2.2.18-1.fc12.src.rpm] -> [cli-build/1362515252.116506.kQNgDlWL]
[info] pkg_process: [5/6604]: {'owner_name': 'sharkcz', 'package_name': 'gtksourceview2-sharp', 'task_id': 3035160, 'creation_event_id': 3682022, 'creation_time': '2011-04-29 07:07:04.359159', 'epoch': None, 'nvr': 'gtksourceview2-sharp-1.0-10.svn89788.fc16', 'name': 'gtksourceview2-sharp', 'completion_time': '2011-04-29 07:09:55.860531', 'state': 1, 'version': '1.0', 'package_id': 6142, 'volume_name': 'DEFAULT', 'release': '10.svn89788.fc16', 'creation_ts': 1304060824.35916, 'completion_ts': 1304060995.86053, 'id': 241357, 'volume_id': 0, 'owner_id': 244}
        [info] que_build: [gtksourceview2-sharp-1.0-10.svn89788.fc16.src.rpm] -> [cli-build/1362515254.528944.WTNbsaqJ]
...

STDERR Messages

[info] primary_tag: [1188/1188/1584]: rwho
[info] primary_tag: [396/396/792]: jlatexmath
[info] primary_tag: [0/0/396]: fprintd
[info] primary_tag: [1584/1584/1980]: objectweb-asm4
[info] primary_tag: [1980/1980/2376]: ktp-approver
[info] primary_tag: [792/792/1188]: nginx
[info] primary_tag: [3168/3168/3564]: audex
[info] primary_tag: [2376/2376/2772]: python-sh
[info] primary_tag: [3564/3564/3960]: python-cmd2
[info] primary_tag: [2772/2772/3168]: perl-IPC-Run-SafeHandles
...
        [info] file_delete: aplus-fsf-4.22.4-21.fc16.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/aplus-fsf/4.22.4/21.fc16/src/aplus-fsf-4.22.4-21.fc16.src.rpm] -> [aplus-fsf-4.22.4-21.fc16.src.rpm]
        [info] file_delete: tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/tritonus/0.3.7/0.13.20101108cvs.fc17/src/tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm] -> [tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm]
        [info] file_delete: jffi-1.0.10-1.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/jffi/1.0.10/1.fc17/src/jffi-1.0.10-1.fc17.src.rpm] -> [jffi-1.0.10-1.fc17.src.rpm]
        [info] file_delete: nautilus-sendto-trac-0.4-2.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/nautilus-sendto-trac/0.4/2.fc17/src/nautilus-sendto-trac-0.4-2.fc17.src.rpm] -> [nautilus-sendto-trac-0.4-2.fc17.src.rpm]
        [info] file_delete: sweep-0.9.3-9.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/sweep/0.9.3/9.fc17/src/sweep-0.9.3-9.fc17.src.rpm] -> [sweep-0.9.3-9.fc17.src.rpm]
...
Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=Koji_Follow&oldid=97158"