Difference between revisions of "Koji Follow"

From CDOT Wiki
Jump to: navigation, search
(→‎Run Notes)
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
= Koji Follow Que Script =
 
= Koji Follow Que Script =
  
−
== Notes ==
+
== Build Notes ==
  
 
* This script does not automatically solve circular dependency issues (which there are a lot of, especially packages like sub perl modules)
 
* This script does not automatically solve circular dependency issues (which there are a lot of, especially packages like sub perl modules)
Line 16: Line 16:
 
* Start of initial-build
 
* Start of initial-build
 
# libA-1.0-1.fc17 is built (Provides: libA.so.1)
 
# libA-1.0-1.fc17 is built (Provides: libA.so.1)
−
# packageB-1.0-1.fc17 has a requires on packageC but is bootstrapped and built without it
+
# packageB-1.0-1.fc17 (BuildRequires: libA,packageC) is temporarily built with its packageC requirement removed (Requires: libA.so.1)
−
# packageC-1.0-1.fc17 has a requires on libA,packageB and is built with libA-1.0-1.fc17,packageB-1.0-1.fc17 (Requires: libA.so.1,packageB)
+
# packageC-1.0-1.fc17 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-1.0-1.fc17 and is built (Requires: libA.so.1)
−
# packageB-1.0-2.fc17 has a requires on packageC and is built with packageC-1.0-1.fc17,libA-1.0-1.fc17 (Requires: packageC)
+
# packageB-1.0-2.fc17 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)
  
 
=== Building a new library first and the packages that depend on it later ===
 
=== Building a new library first and the packages that depend on it later ===
Line 24: Line 24:
 
* Start of mass-rebuild
 
* Start of mass-rebuild
 
# libA-2.0-1.fc18 is built (Provides: libA.so.2)
 
# libA-2.0-1.fc18 is built (Provides: libA.so.2)
−
#* Attempting to build packageB-2.0-2.fc18 will require packageC-1.0-1.fc17 which requires libA.so.1 which only finds libA.so.2 given the current repodata (this package needs to be bootstrapped again)
+
#* Attempting to build packageB or packageC will BuildRequire each other and which both will then Require libA.so.1 which will then only find libA.so.2 in the current repodata
−
#* Attempting to build packageC-2.0-1.fc18 will require packageB-1.0-2.fc17 which requires packageC-1.0-1.fc17 which brings us to the same breakage error as shown above
 
  
−
=== Building a new package first and then the following libraries they depend on ===
+
=== Building new packages first and then the following libraries they depend on after ===
  
 
* Start of mass-rebuild
 
* Start of mass-rebuild
−
# packageB-2.0-2.fc18 has a requires on packageC and is built with packageC-1.0-1.fc17 which has a requires on libA,packageB which installs libA-1.0-1.fc17 and packageB-1.0-2.fc17 (Requires: packageC)
+
# packageB-2.0-2.fc18 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)
−
# packageC-2.0-1.fc18 has a requires on libA,packageB and is built with libA-1.0-1.fc17,packageB-2.0-1.fc18 which has a requires on packageC which installs packageC-1.0-1.fc17 (Requires: libA.so.1,packageB)
+
# packageC-2.0-1.fc18 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-2.0-2.fc18 and is built (Requires: libA.so.1)
 
# libA-2.0-1.fc18 is built (Provides: libA.so.2)
 
# libA-2.0-1.fc18 is built (Provides: libA.so.2)
−
#* All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageC will require libA.so.1 which is not being provided for at all since the updated libA-2.0-1.fc18 provides libA.so.2
+
#* All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageB or packageC will require libA.so.1 which is not being provided for at all since the updated libA only provides libA.so.2
 +
 
 +
== Run Notes ==
 +
 
 +
* Packages are not processed in alphabetical order and the output is not displayed in numerical order
 +
* The koji error log is for processing messages and the koji stdout log is for queing messages
 +
* You need to add any new package names to a tag before they are built
 +
* You need a lot of free space in the current directory that the script is running in
 +
* Before running the script, attempt a scratch build of a sample package with the given build tag
 +
* You can redirect stderr to stdout and write stdout to a file so that the messages are in order with each other
  
 
= Downloads =
 
= Downloads =
Line 71: Line 79:
 
</pre>
 
</pre>
  
−
<b>python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log</b>
+
<b>python ~/kf/koji-follow.py ~/kf/koji-follow.conf > ~/kf/kfo.log 2> ~/kf/kfe.log</b>
  
 
== latest-follow mode ==
 
== latest-follow mode ==
Line 100: Line 108:
 
</pre>
 
</pre>
  
−
<b>python ~/koji-follow.py ~/koji-follow.conf > ~/logs/kfo.log 2> ~/logs/kfe.log</b>
+
== Execution Example ==
 +
 
 +
<b>python ~/kf/koji-follow.py ~/kf/koji-follow.conf > ~/kf/kfo.log 2> ~/kf/kfe.log</b>
  
−
= Example Output =
+
= Example Log Output =
  
 
== STDOUT Messages ==
 
== STDOUT Messages ==

Latest revision as of 09:27, 8 July 2013

Koji Follow Que Script

Build Notes

  • This script does not automatically solve circular dependency issues (which there are a lot of, especially packages like sub perl modules)
  • This script does not automatically solve bootstrapping or library.so name bump requires issues (which happens often during a completely new tag rebuild)
  • This script does not attempt to solve any package build failure issues
  • This script can be run in 2 modes: "mass-rebuild" and "latest-follow" and ques builds in order based on their "creation_ts" value
    • mass-rebuild mode will attempt to compile a complete list of packages that were built for a given tag and then que them in the given order
    • latest-follow mode will perform a "list-tagged --latest" and que each unbuilt package in the given order

Common Repository Breakage Scenarios due to basic circular dependencies

All of the following examples take place after this series of events

  • Start of initial-build
  1. libA-1.0-1.fc17 is built (Provides: libA.so.1)
  2. packageB-1.0-1.fc17 (BuildRequires: libA,packageC) is temporarily built with its packageC requirement removed (Requires: libA.so.1)
  3. packageC-1.0-1.fc17 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-1.0-1.fc17 and is built (Requires: libA.so.1)
  4. packageB-1.0-2.fc17 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)

Building a new library first and the packages that depend on it later

  • Start of mass-rebuild
  1. libA-2.0-1.fc18 is built (Provides: libA.so.2)
    • Attempting to build packageB or packageC will BuildRequire each other and which both will then Require libA.so.1 which will then only find libA.so.2 in the current repodata

Building new packages first and then the following libraries they depend on after

  • Start of mass-rebuild
  1. packageB-2.0-2.fc18 (BuildRequires: libA,packageC) installs libA-1.0-1.fc17,packageC-1.0-1.fc17 and is built (Requires: libA.so.1)
  2. packageC-2.0-1.fc18 (BuildRequires: libA,packageB) installs libA-1.0-1.fc17,packageB-2.0-2.fc18 and is built (Requires: libA.so.1)
  3. libA-2.0-1.fc18 is built (Provides: libA.so.2)
    • All packages will now appear to be up to date and current except that the repo is now broken as trying to install packageB or packageC will require libA.so.1 which is not being provided for at all since the updated libA only provides libA.so.2

Run Notes

  • Packages are not processed in alphabetical order and the output is not displayed in numerical order
  • The koji error log is for processing messages and the koji stdout log is for queing messages
  • You need to add any new package names to a tag before they are built
  • You need a lot of free space in the current directory that the script is running in
  • Before running the script, attempt a scratch build of a sample package with the given build tag
  • You can redirect stderr to stdout and write stdout to a file so that the messages are in order with each other

Downloads

Source

Example Config and Execution Scenarios

mass-rebuild mode

~/koji-follow.conf 

primary_url = "http://koji.fedoraproject.org"
primary_arch = "x86_64"
#secondary_url = "http://arm.koji.fedoraproject.org"
#secondary_arch = "armhfp"

#primary_url = "http://arm.koji.fedoraproject.org"
#primary_arch = "armhfp"
secondary_url = "http://japan.proximity.on.ca"
secondary_arch = "armv6hl"

client_cert = os.path.expanduser("~/.fedora.cert")
server_cert = os.path.expanduser("~/.fedora-server-ca.cert")

target_arch = "arm"
tag_name = "f18"
que_limit = 80

excl_list = ["glibc","kernel","koji","redhat-rpm-config","rpm","yum"]
check_tag = "all"
retry_build = "all"

python ~/kf/koji-follow.py ~/kf/koji-follow.conf > ~/kf/kfo.log 2> ~/kf/kfe.log

latest-follow mode

~/koji-follow.conf 

primary_url = "http://koji.fedoraproject.org"
primary_arch = "x86_64"
#secondary_url = "http://arm.koji.fedoraproject.org"
#secondary_arch = "armhfp"

#primary_url = "http://arm.koji.fedoraproject.org"
#primary_arch = "armhfp"
secondary_url = "http://japan.proximity.on.ca"
secondary_arch = "armv6hl"

client_cert = os.path.expanduser("~/.fedora.cert")
server_cert = os.path.expanduser("~/.fedora-server-ca.cert")

target_arch = "arm"
tag_name = "f18"
que_limit = 80

excl_list = ["glibc","kernel","koji","redhat-rpm-config","rpm","yum"]
check_tag = "latest"
retry_build = "failed"

Execution Example

python ~/kf/koji-follow.py ~/kf/koji-follow.conf > ~/kf/kfo.log 2> ~/kf/kfe.log

Example Log Output

STDOUT Messages

[info] pkg_process: [1/6604]: {'owner_name': 'jkeating', 'package_name': 'email2trac', 'task_id': 1503441, 'creation_event_id': 1793404, 'creation_time': '2009-07-25 14:59:59.646649', 'epoch': None, 'nvr': 'email2trac-0.13-6.fc12', 'name': 'email2trac', 'completion_time': '2009-07-25 15:35:37.427707', 'state': 1, 'version': '0.13', 'package_id': 6215, 'volume_name': 'DEFAULT', 'release': '6.fc12', 'creation_ts': 1248533999.64665, 'completion_ts': 1248536137.42771, 'id': 117394, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [email2trac-0.13-6.fc12.src.rpm] -> [cli-build/1362515245.29142.RuqZKocF]
[info] pkg_process: [2/6604]: {'owner_name': 'jkeating', 'package_name': 'powerpc-utils-papr', 'task_id': 1525540, 'creation_event_id': 1839140, 'creation_time': '2009-07-28 04:43:55.459368', 'epoch': None, 'nvr': 'powerpc-utils-papr-1.1.6-2.fc12', 'name': 'powerpc-utils-papr', 'completion_time': '2009-07-28 05:47:09.877729', 'state': 1, 'version': '1.1.6', 'package_id': 5369, 'volume_name': 'DEFAULT', 'release': '2.fc12', 'creation_ts': 1248756235.45937, 'completion_ts': 1248760029.87773, 'id': 122145, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [powerpc-utils-papr-1.1.6-2.fc12.src.rpm] -> [cli-build/1362515248.044621.imzInFfA]
[info] pkg_process: [3/6604]: {'owner_name': 'jkeating', 'package_name': 'ppc64-utils', 'task_id': 1525552, 'creation_event_id': 1839195, 'creation_time': '2009-07-28 04:46:54.025335', 'epoch': None, 'nvr': 'ppc64-utils-0.14-5.fc12', 'name': 'ppc64-utils', 'completion_time': '2009-07-28 05:51:18.860184', 'state': 1, 'version': '0.14', 'package_id': 131, 'volume_name': 'DEFAULT', 'release': '5.fc12', 'creation_ts': 1248756414.02534, 'completion_ts': 1248760278.86018, 'id': 122152, 'volume_id': 0, 'owner_id': 2}
        [info] que_build: [ppc64-utils-0.14-5.fc12.src.rpm] -> [cli-build/1362515249.622192.vcbTPtpj]
[info] pkg_process: [4/6604]: {'owner_name': 'rrakus', 'package_name': 'iprutils', 'task_id': 1787705, 'creation_event_id': 2291747, 'creation_time': '2009-11-04 05:55:36.885871', 'epoch': None, 'nvr': 'iprutils-2.2.18-1.fc12', 'name': 'iprutils', 'completion_time': '2009-11-04 05:58:25.827132', 'state': 1, 'version': '2.2.18', 'package_id': 722, 'volume_name': 'DEFAULT', 'release': '1.fc12', 'creation_ts': 1257314136.88587, 'completion_ts': 1257314305.82713, 'id': 139717, 'volume_id': 0, 'owner_id': 613}
        [info] que_build: [iprutils-2.2.18-1.fc12.src.rpm] -> [cli-build/1362515252.116506.kQNgDlWL]
[info] pkg_process: [5/6604]: {'owner_name': 'sharkcz', 'package_name': 'gtksourceview2-sharp', 'task_id': 3035160, 'creation_event_id': 3682022, 'creation_time': '2011-04-29 07:07:04.359159', 'epoch': None, 'nvr': 'gtksourceview2-sharp-1.0-10.svn89788.fc16', 'name': 'gtksourceview2-sharp', 'completion_time': '2011-04-29 07:09:55.860531', 'state': 1, 'version': '1.0', 'package_id': 6142, 'volume_name': 'DEFAULT', 'release': '10.svn89788.fc16', 'creation_ts': 1304060824.35916, 'completion_ts': 1304060995.86053, 'id': 241357, 'volume_id': 0, 'owner_id': 244}
        [info] que_build: [gtksourceview2-sharp-1.0-10.svn89788.fc16.src.rpm] -> [cli-build/1362515254.528944.WTNbsaqJ]
...

STDERR Messages

[info] primary_tag: [1188/1188/1584]: rwho
[info] primary_tag: [396/396/792]: jlatexmath
[info] primary_tag: [0/0/396]: fprintd
[info] primary_tag: [1584/1584/1980]: objectweb-asm4
[info] primary_tag: [1980/1980/2376]: ktp-approver
[info] primary_tag: [792/792/1188]: nginx
[info] primary_tag: [3168/3168/3564]: audex
[info] primary_tag: [2376/2376/2772]: python-sh
[info] primary_tag: [3564/3564/3960]: python-cmd2
[info] primary_tag: [2772/2772/3168]: perl-IPC-Run-SafeHandles
...
        [info] file_delete: aplus-fsf-4.22.4-21.fc16.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/aplus-fsf/4.22.4/21.fc16/src/aplus-fsf-4.22.4-21.fc16.src.rpm] -> [aplus-fsf-4.22.4-21.fc16.src.rpm]
        [info] file_delete: tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/tritonus/0.3.7/0.13.20101108cvs.fc17/src/tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm] -> [tritonus-0.3.7-0.13.20101108cvs.fc17.src.rpm]
        [info] file_delete: jffi-1.0.10-1.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/jffi/1.0.10/1.fc17/src/jffi-1.0.10-1.fc17.src.rpm] -> [jffi-1.0.10-1.fc17.src.rpm]
        [info] file_delete: nautilus-sendto-trac-0.4-2.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/nautilus-sendto-trac/0.4/2.fc17/src/nautilus-sendto-trac-0.4-2.fc17.src.rpm] -> [nautilus-sendto-trac-0.4-2.fc17.src.rpm]
        [info] file_delete: sweep-0.9.3-9.fc17.src.rpm
        [info] file_download: [http://koji.fedoraproject.org/packages/sweep/0.9.3/9.fc17/src/sweep-0.9.3-9.fc17.src.rpm] -> [sweep-0.9.3-9.fc17.src.rpm]
...
Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=Koji_Follow&oldid=97162"