Difference between revisions of "CreateRepo Performance Improvements"

From CDOT Wiki
Jump to: navigation, search
(CreateRepo Performance Improvements)
(Project Plan)
 
(4 intermediate revisions by the same user not shown)
Line 34: Line 34:
 
Key contacts:
 
Key contacts:
 
<!-- Add links to any mentors or key participants in the community. -->
 
<!-- Add links to any mentors or key participants in the community. -->
*Paul Whalen
+
*Paul Whalen - http://zenit.senecac.on.ca/wiki/index.php/User:Paul.W
 
*Dgilmore
 
*Dgilmore
  
Line 40: Line 40:
 
<!-- Note: each contributor is expected to have unique goals. These goals may be ''related'' to other students' work, but must be ''distinct'' and ''attainable'' regardless of the state of the other students' work. For example, under the umbrella of one project title, one student may work on packaging a piece of software and another may work on documentation, or one may work on solving one bug and another on solving another bug, but two students must not work on the same bug or depend on the other students' work in order to be able to complete their own project. -->
 
<!-- Note: each contributor is expected to have unique goals. These goals may be ''related'' to other students' work, but must be ''distinct'' and ''attainable'' regardless of the state of the other students' work. For example, under the umbrella of one project title, one student may work on packaging a piece of software and another may work on documentation, or one may work on solving one bug and another on solving another bug, but two students must not work on the same bug or depend on the other students' work in order to be able to complete their own project. -->
 
* 0.1 - Set up a 1 Gbps LAN.
 
* 0.1 - Set up a 1 Gbps LAN.
* 0.2 - Change the store set up. This could be change the main hard drive on HongKong to SSD.
+
* 0.2 - Change the storages. This could be change the main hard drive on HongKong to SSD.
 
* 0.3 - Optimizing the CreateRepo tasks on HongKong for best speed.
 
* 0.3 - Optimizing the CreateRepo tasks on HongKong for best speed.
  
Line 53: Line 53:
 
=== Mailing Lists ===
 
=== Mailing Lists ===
 
<!-- Add any appropriate mailing lists to which you are subscribed (e.g., see http://lists.fedoraproject.org -->
 
<!-- Add any appropriate mailing lists to which you are subscribed (e.g., see http://lists.fedoraproject.org -->
buildsys@lists.fedoraproject.org
+
*buildsys@lists.fedoraproject.org
  
arm@lists.fedoraproject.org
+
*arm@lists.fedoraproject.org
  
 +
*arm-builds@lists.fedoraproject.org
  
 
=== Upsteam Wiki and Web ===
 
=== Upsteam Wiki and Web ===
Line 77: Line 78:
 
==== Planets ====
 
==== Planets ====
 
<!-- Links to any planets related to this project -->
 
<!-- Links to any planets related to this project -->
 +
*http://zenit.senecac.on.ca/~chris.tyler/planet/
 +
*http://planet.fedoraproject.org/
  
 
== Project News ==
 
== Project News ==
  
 
<!-- This is where a permanent record of your releases and updates will go.  In these you should discuss the status or your work, your interactions with other members of the community (e.g., Seneca and Mozilla), problems you have encountered, etc. -->
 
<!-- This is where a permanent record of your releases and updates will go.  In these you should discuss the status or your work, your interactions with other members of the community (e.g., Seneca and Mozilla), problems you have encountered, etc. -->

Latest revision as of 18:52, 7 March 2011

CreateRepo Performance Improvements

Project Description

The Fedora-ARM build farm consists of the koji-hub/koji-web system (on HongKong, an x86_64 machine) and a group of ARM builders. HongKong also handles the CreateRepo tasks, which heavily load that machine. It might make sense to move those tasks to another machine, but doing so makes the CreateRepo jobs take a lot longer (15 minutes on HongKong vs. 55 minutes on another x86_64 server). This is presumably due to the overhead of sharing files between HongKong and the other server over NFS on the 100 Mbps Seneca network.

Project Leader(s)

ctyler, PaulW, Dgilmore

Project Contributor(s)

Project Details

This project involves figuring out how to run the CreateRepo jobs more quickly. Possible solutions include a 1 Gbps LAN, a redistribution of the file storage or a change to a different file storage technology, or optimizing the CreateRepo tasks on HongKong for best speed.

  • Skills required: system administration, benchmarking
  • Resources: CDOT server systems (HongKong, Ireland, Scotland)
  • Expected result: significant reduction in CreateRepo times, especially when multiple CreateRepo tasks are running

Project Plan

Tracking mechanism (bugzilla, trac, github, ...):

Key contacts:

Goals for each release and plans for reaching those goals:

  • 0.1 - Set up a 1 Gbps LAN.
  • 0.2 - Change the storages. This could be change the main hard drive on HongKong to SSD.
  • 0.3 - Optimizing the CreateRepo tasks on HongKong for best speed.

Communication

https://dboddie46.wordpress.com/

IRC Channel Listing

  • fedora-devel
  • fedora-Arm
  • koji
  • arm

Mailing Lists

  • buildsys@lists.fedoraproject.org
  • arm@lists.fedoraproject.org
  • arm-builds@lists.fedoraproject.org

Upsteam Wiki and Web

Links/Bugs/Tracking

Source Code Control

Blogs

Seneca Particpants

Non-Seneca Participants

Planets

Project News