Difference between revisions of "LEAP SOP"

From CDOT Wiki
Jump to: navigation, search
 
(8 intermediate revisions by the same user not shown)
Line 5: Line 5:
  
 
== Fetching source ==
 
== Fetching source ==
Source packages are located under 7.x folder on http://vault.centos.org/. There are several types of repository:
+
Source packages are located under 7.x directory on http://vault.centos.org/. There are several types of repository:
  
 
; os: Base packages
 
; os: Base packages
Line 11: Line 11:
 
; extra: Extra packages such as docker, cockpit, golang, etc.
 
; extra: Extra packages such as docker, cockpit, golang, etc.
  
Source rpms can only be downloaded through yum or curl/wget, since there is no Rsync repository for Centos source rpms.
+
While there is no Rsync repository for CentOS source rpms, they can only be downloaded manually through curl/wget or yum.  Therefore, creating a bash script to automate download processes is highly recommended.
  
 
== Determine problem ==
 
== Determine problem ==
Line 25: Line 25:
  
 
=== Common errors and solutions ===
 
=== Common errors and solutions ===
; build.log - Code compilation error
+
; Code compilation error
: <u>Reason</u>: Certain depended library is too old
+
: <u>Reason</u>: Certain pulled library is too old
 
: <u>Solution</u>:  Try to build all other packages and then rebuild this one
 
: <u>Solution</u>:  Try to build all other packages and then rebuild this one
  
; build.log - Cannot detect system type   
+
; Cannot detect system type   
 
: <u>Reason</u>: The software uses outdated <tt>config.guess</tt> and <tt>config.sub</tt> to autodetect system type
 
: <u>Reason</u>: The software uses outdated <tt>config.guess</tt> and <tt>config.sub</tt> to autodetect system type
 
: <u>Solution</u>:  Replace both files in the source with the latest online version. See https://www.gnu.org/software/gettext/manual/html_node/config_002eguess.html  
 
: <u>Solution</u>:  Replace both files in the source with the latest online version. See https://www.gnu.org/software/gettext/manual/html_node/config_002eguess.html  
  
; build.log - Unknown system type
+
; Unknown system type
 
: <u>Reason</u>: The software only allows it to be built with kernel 2.6/3.x  
 
: <u>Reason</u>: The software only allows it to be built with kernel 2.6/3.x  
: <u>Solution</u>: Add 4.x support to the system checking script
+
: <u>Solution</u>: Add 4.x support to the system checking script (just a IF statement of checking version number)
  
; BuildError No matching arches were found
+
; Build test/check failed
: <u>Reason</u>: Package only supports the architectures on <tt>ExclusiveArch</tt> list in the spec file
+
: <u>Reason</u>: Can be random reasons or the software is just unable to run perfectly on AArch64
: <u>Solution</u>: Checkout the package on Fedora ARM Koji and CentOS AArch64 repo and determine whether the package is possible to support AArch64 or not
+
: <u>Solution</u>: Try to look for solutions, otherwise comment out the <tt>%check</tt> section in spec file
 +
 
 +
; BuildError<span>:</span> No matching arches were found
 +
: <u>Reason</u>: Package only supports specific architectures on the <tt>ExclusiveArch</tt> list in spec file
 +
: <u>Solution</u>: Checkout similar version of the package on [http://arm.koji.fedoraproject.org Fedora ARM Koji] and [http://buildlogs.centos.org/centos/7 CentOS AArch64 repo] and determine whether the package is possible to support AArch64 or not
 +
 
 +
; Fail to patch
 +
: <u>Reason</u>: Wrong patches or sources were packed into the source rpm
 +
: <u>Solution</u>: Retrieve the original CentOS src.rpm and repack it again, see also: [[#Duplicated source name]]
  
 
=== Resources ===
 
=== Resources ===
# Previous build of the package
+
* Previous build of the package
# CentOS AArch64 repo: http://buildlogs.centos.org/centos/7
+
* CentOS AArch64 repo: http://buildlogs.centos.org/centos/7
# Fedora ARM Koji: http://arm.koji.fedoraproject.org
+
* Fedora ARM Koji: http://arm.koji.fedoraproject.org
# CentOS git repo (for reverting CentOS changes): http://git.centos.org
+
* CentOS git repo (for reverting CentOS changes): http://git.centos.org
# GOOGLE!
+
* '''<span style="color:#4885ed">G</span><span style="color:#db3236">O</span><span style="color:#f4c20d">O</span><span style="color:#4885ed">G</span><span style="color:#3cba54">L</span><span style="color:#db3236">E</span>'''
  
 
== Source modification ==
 
== Source modification ==
There a few standards to follow during modification:
+
There a few standards to follow during modification to make life easier:
 
* Add comments about any line changes
 
* Add comments about any line changes
 
* Add the word ''LEAP'' at the beginning of any comments, for example:
 
* Add the word ''LEAP'' at the beginning of any comments, for example:
 
<source lang=bash>
 
<source lang=bash>
# LEAP Add AArch64 support
+
# LEAP Add AArch64 support
ExclusiveArch: x86_64 aarch64
+
ExclusiveArch: x86_64 aarch64
 +
</source>
 +
* When commenting out a existing line of code, add one more <tt>%</tt> before any rpm macro otherwise rpmbuild will treat it as a error/warning. For instance:
 +
<source lang=bash>
 +
# Before commenting out
 +
cp %{SOURCE1} $RPM_BUILD_ROOT%{_datadir}/virt-tools
 +
 
 +
# After
 +
# cp %%{SOURCE1} $RPM_BUILD_ROOT%%{_datadir}/virt-tools
 +
 
 
</source>
 
</source>
* Add ''.leap'' after the release version, add ''.leap.1'', ''.leap.2'' ... and so on for further modification
+
* Add <tt>.leap</tt> after the release version, add <tt>.leap.1</tt>, <tt>.leap.2</tt> ... and so on for further modifications
  
 
* Make sure to specify all the changes under the <tt>%changelog</tt> section:
 
* Make sure to specify all the changes under the <tt>%changelog</tt> section:
Line 63: Line 80:
 
- Add AArch64 support
 
- Add AArch64 support
 
- Roll in LEAP Branding
 
- Roll in LEAP Branding
 
 
</source>
 
</source>
  
=== Rebranding ===
+
=== Re-branding ===
The packages that contains the word ''.centos'' in their release version are branded packages. In order to properly rebranding those packages for LEAP, here is the checklist:
+
The packages that contain the extension <tt>.centos</tt> in their release version are branded packages. In order to properly re-branding those packages for LEAP, here is the checklist:
* Replace the word ''CentOS'' or ''CentOS Linux'' with ''LEAP'' in spec package descriptions and patches (make sure they are branding-related)
+
* Replace the word "CentOS" or "CentOS Linux" with "LEAP", and "centos" with "leap" in spec package descriptions and patches (make sure they are branding-related)
* Replace CentOS branding patches with LEAP branding patches
+
* Replace CentOS branding patches with LEAP one
** If certain patch file contain author names, regenerate the patch with your own name instead of modifying the patch directly
+
** If certain patch file contains author name, regenerate the patch with your own name instead of modifying the patch directly
* Replace CentOS links with LEAP links accordingly
+
* Replace CentOS links with links for LEAP accordingly
 
* Sometimes Red Hat trademarks and links have to be replaced as well
 
* Sometimes Red Hat trademarks and links have to be replaced as well
* Remove CentOS/RHEL certificate files and binaries
+
* Remove CentOS/RHEL certificates, binaries, or any enterprise support files
 +
 
 +
==== Particular cases ====
 +
; libreport / abrt
 +
: Libreport and abrt are the library and application for automatic bug detection and reporting to RHEL bug tracker servers. Since CentOS added several significant patches to both packages for supporting their own bug tracker, removing those patches can be very complicated process. The best way to solve this problem is retrieving the original RHEL versions from http://git.centos.org, add our own de-branding patches, and bump the version-release number + ''.leap'' to match the requirements of dependency.
 +
 
 +
; ntp / chrony / PackageKit / system-config-date
 +
: These packages contain network time protocol configurations pointing to the NTP pool address: ''pool.VENDERZONE.ntp.org'' . By default, they are pointing to ''pool.centos.ntp.org''. Since ntp pool for LEAP hasn't been set up, it is necessary to remove any ''.VENDERZONE'' (system-config-date one has a different name) in the spec files instead of just replacing the brand.
 +
 
 +
; grub2 / shim / shim-signed / system-config-kdump
 +
: There is a important variable <tt>efidir</tt> in the patches and specs in these packages. It represents the name of the EFI directory containing all the EFI boot files. Make sure the value of <tt>efidir</tt> changed to '''leap'''.
 +
 
 +
; oscap-anaconda-addon / oscap-security-guide
 +
: TODO
 +
 
 +
== Build order ==
 +
It is safer and more time-efficient to build certain essential packages before any other packages are built, because other packages are very likely to depend on them:
 +
* '''leap-release''' - the base of entire distribution
 +
* '''gcc''', '''glibc''', '''make''', '''binutils''' - basic toolchain
 +
* '''coreutils''' - basic system tools
 +
* '''xorg-x11-server''' - xorg-x11-drv* always require the specific version of xorg-x11-server they are built with
  
 +
== Duplicated source name ==
 +
A few packages in the package set contain patches or sources having identical file names. When installing source rpms to rpmbuild tree (<tt>$HOME/rpmbuild/*</tt>), try to avoid installing too many of them at the same time (50+).  Wipe the rpmbuild tree after finishing modification and build of those packages on Koji.  There are a few problematic packages have been discovered:
 +
* '''java-1.7.0-openjdk''' and '''java-1.8.0-openjdk'''
 +
* '''xorg-x11-drv-dummy''' and '''xorg-x11-drv-fbdev'''
  
==== Particular branded packages ====
+
== Other special cases ==
 +
* '''libreoffice''' - Require at least 50GB of space to build
 +
* '''binutils''' - was intentionally upgraded to 2.24, be careful not to downgrade it.
 +
* '''httpd''' - <tt>centos-noindex.tar.gz</tt> contains branded default webpages, replace it with <tt>leap-noindex.tar.gz</tt> from previous version of httpd
  
; libreport/abrt
+
= Fixing broken dependencies =
: '''Libreport''' and '''abrt''' are the library and application for automatic bug detection and reporting to RHEL bug tracker servers. Since CentOS added several significant patches to both packages for supporting their own bug tracker, removing those patches can be very complicated process. The best way to solve this problem is retrieving the original RHEL versions from http://git.centos.org, add our old debranding patches, and bump the version-release number + ''.leap'' to match the requirements of dependency.
+
After all of the packages have been done, it's time to fix minor broken dependencies. <tt>repoclosure</tt> and <tt>leap-closure-report.sh</tt> (by Chris Markieta) are the tools to check dependency stats of a repository. You should always check the build-tag repository since new packages may require some of the old packages.
  
 +
When a package requires a missing shared library file like <tt>xxxxx-3.so</tt>, it only need to be rebuilt again (if the library exists) in order to pick up the latest <tt>.so</tt> file from the latest version of required library.
  
 +
= Package signing =
 +
= Synchronizing Repositories =
 +
= Release =
  
 
More coming ...
 
More coming ...

Latest revision as of 21:40, 27 March 2016

Introduction

This wiki page describes the SOP (Standard Operating Procedure) for releasing LEAP like packaging-related processes. It also includes packaging standards, common fixes, patch resources, and debugging methods.

Package building

Fetching source

Source packages are located under 7.x directory on http://vault.centos.org/. There are several types of repository:

os
Base packages
updates
Package updates
extra
Extra packages such as docker, cockpit, golang, etc.

While there is no Rsync repository for CentOS source rpms, they can only be downloaded manually through curl/wget or yum. Therefore, creating a bash script to automate download processes is highly recommended.

Determine problem

If a package fails to be built on Koji, there are several possibilities causing it:

  • Package name is not added to the pkg list of the tag
  • Package only supports the architectures on ExclusiveArch list in the spec file
  • Same version of the package already existed on Koji
  • Repository of the build-tag is being regenerated or broken
  • Package spec contains RHEL or CentOS specific rpm macros
  • Missing Dependency
  • Package has to be patched in order to be built on AArch64 platform

Common errors and solutions

Code compilation error
Reason: Certain pulled library is too old
Solution: Try to build all other packages and then rebuild this one
Cannot detect system type
Reason: The software uses outdated config.guess and config.sub to autodetect system type
Solution: Replace both files in the source with the latest online version. See https://www.gnu.org/software/gettext/manual/html_node/config_002eguess.html
Unknown system type
Reason: The software only allows it to be built with kernel 2.6/3.x
Solution: Add 4.x support to the system checking script (just a IF statement of checking version number)
Build test/check failed
Reason: Can be random reasons or the software is just unable to run perfectly on AArch64
Solution: Try to look for solutions, otherwise comment out the %check section in spec file
BuildError: No matching arches were found
Reason: Package only supports specific architectures on the ExclusiveArch list in spec file
Solution: Checkout similar version of the package on Fedora ARM Koji and CentOS AArch64 repo and determine whether the package is possible to support AArch64 or not
Fail to patch
Reason: Wrong patches or sources were packed into the source rpm
Solution: Retrieve the original CentOS src.rpm and repack it again, see also: #Duplicated source name

Resources

Source modification

There a few standards to follow during modification to make life easier:

  • Add comments about any line changes
  • Add the word LEAP at the beginning of any comments, for example:
# LEAP Add AArch64 support
ExclusiveArch: x86_64 aarch64
  • When commenting out a existing line of code, add one more % before any rpm macro otherwise rpmbuild will treat it as a error/warning. For instance:
# Before commenting out
cp %{SOURCE1} $RPM_BUILD_ROOT%{_datadir}/virt-tools

# After
# cp %%{SOURCE1} $RPM_BUILD_ROOT%%{_datadir}/virt-tools
  • Add .leap after the release version, add .leap.1, .leap.2 ... and so on for further modifications
  • Make sure to specify all the changes under the %changelog section:
*  Fri Mar 25 2016 Foo Bar <foo.bar@example.com> - 1:1.28.1-1.55.leap
- Add AArch64 support
- Roll in LEAP Branding

Re-branding

The packages that contain the extension .centos in their release version are branded packages. In order to properly re-branding those packages for LEAP, here is the checklist:

  • Replace the word "CentOS" or "CentOS Linux" with "LEAP", and "centos" with "leap" in spec package descriptions and patches (make sure they are branding-related)
  • Replace CentOS branding patches with LEAP one
    • If certain patch file contains author name, regenerate the patch with your own name instead of modifying the patch directly
  • Replace CentOS links with links for LEAP accordingly
  • Sometimes Red Hat trademarks and links have to be replaced as well
  • Remove CentOS/RHEL certificates, binaries, or any enterprise support files

Particular cases

libreport / abrt
Libreport and abrt are the library and application for automatic bug detection and reporting to RHEL bug tracker servers. Since CentOS added several significant patches to both packages for supporting their own bug tracker, removing those patches can be very complicated process. The best way to solve this problem is retrieving the original RHEL versions from http://git.centos.org, add our own de-branding patches, and bump the version-release number + .leap to match the requirements of dependency.
ntp / chrony / PackageKit / system-config-date
These packages contain network time protocol configurations pointing to the NTP pool address: pool.VENDERZONE.ntp.org . By default, they are pointing to pool.centos.ntp.org. Since ntp pool for LEAP hasn't been set up, it is necessary to remove any .VENDERZONE (system-config-date one has a different name) in the spec files instead of just replacing the brand.
grub2 / shim / shim-signed / system-config-kdump
There is a important variable efidir in the patches and specs in these packages. It represents the name of the EFI directory containing all the EFI boot files. Make sure the value of efidir changed to leap.
oscap-anaconda-addon / oscap-security-guide
TODO

Build order

It is safer and more time-efficient to build certain essential packages before any other packages are built, because other packages are very likely to depend on them:

  • leap-release - the base of entire distribution
  • gcc, glibc, make, binutils - basic toolchain
  • coreutils - basic system tools
  • xorg-x11-server - xorg-x11-drv* always require the specific version of xorg-x11-server they are built with

Duplicated source name

A few packages in the package set contain patches or sources having identical file names. When installing source rpms to rpmbuild tree ($HOME/rpmbuild/*), try to avoid installing too many of them at the same time (50+). Wipe the rpmbuild tree after finishing modification and build of those packages on Koji. There are a few problematic packages have been discovered:

  • java-1.7.0-openjdk and java-1.8.0-openjdk
  • xorg-x11-drv-dummy and xorg-x11-drv-fbdev

Other special cases

  • libreoffice - Require at least 50GB of space to build
  • binutils - was intentionally upgraded to 2.24, be careful not to downgrade it.
  • httpd - centos-noindex.tar.gz contains branded default webpages, replace it with leap-noindex.tar.gz from previous version of httpd

Fixing broken dependencies

After all of the packages have been done, it's time to fix minor broken dependencies. repoclosure and leap-closure-report.sh (by Chris Markieta) are the tools to check dependency stats of a repository. You should always check the build-tag repository since new packages may require some of the old packages.

When a package requires a missing shared library file like xxxxx-3.so, it only need to be rebuilt again (if the library exists) in order to pick up the latest .so file from the latest version of required library.

Package signing

Synchronizing Repositories

Release

More coming ...