Fedora ARM Secondary Architecture/ARM Fedora Backup Management
It is still under construction and content may change. Do not rely on this information.
Contents
Seneca CDOT ARM Project: Management Server Backup and Recovery
- Phone: X33463
- Email: fedora-arm at senecacollege dot ca
Scope
This document is intended for those who are familiar with administering the ARM Project's management servers. Having knowledge of the CDOT ARM standard operation procedures may lead to a better understanding of this document.
Purpose
The purpose of this document is to describe the process for the backup and synchronization of critical data and provide example of some possible disaster recovery scenarios.
Introduction
The Fedora ARM Project is a highly active R & D project. Because of the rapid change, the project required some specific backup strategies to be implemented on top of regular backup strategies. In the event of an emergency, this document should provide enough information to re-create a similar management platform that will keep the project running.
Background
Like all Fedora projects, the ARM Project has a small yet active global community. The build firm is located in Ontario, Canada. Infrastructure support for the build firm is provided by the Seneca Center for Development and Technology at Seneca College in Toronto. The number of available ARM builders in the firm can vary. The builders are not backed up as those are easily recreated from available images.
As for the management servers, there are five major ones that are critical to this operation. For a complete list of servers and the services those offer please consult the necessary documents. These four servers are Australia (Koji builders temporary work storage), Chile (The complete ARM repository), Hongkong (Koji hub and web interface), Iraq (Central backup management and storage) and Ireland (Koji database).
Description
The backup system has been broken into two custom scripts. First one is to create a local copy of the configurations and to back up the repository to removable drives, it is called blaze. The second script is called wildfire and it is to synchronize those local copies to different locations over the network. The servers get the copies of those scripts from the central backup management and storage server (iraq) via scp and runs blaze first and then wildfire.
Backup Script Source
The scripts are located at (any server with backup enabled):
/usr/local/bin/
Backup Directory Structure
The backups (both local copies and networked synced copies) are stored inside /archive/blaze-backup (/archive/repo-backup for the repository sync) in each servers respective directory. Example:
[root@australia ~]# tree /archive /archive ├── blaze-backup │ ├── australia │ │ ├── australia-cron-Feb-27-2012.tar.bz2 │ │ ├── australia-etc-Feb-27-2012.tar.bz2 │ │ └── australia-userlocalbin-Feb-27-2012.tar.bz2 │ ├── chile │ │ ├── chile-cron-Feb-27-2012.tar.bz2 │ │ ├── chile-etc-Feb-27-2012.tar.bz2 │ │ └── chile-userlocalbin-Feb-27-2012.tar.bz2 │ └── ireland │ ├── ireland-cron-Feb-27-2012.tar.bz2 │ ├── ireland-etc-Feb-27-2012.tar.bz2 │ ├── ireland-postgresql-Mar-27-2012.sql.bz2 │ └── ireland-userlocalbin-Feb-27-2012.tar.bz2 └── lost+found
Space Consideration and Allocation
For backing up the configurations and keeping copies of those for the previous 30 days and also allocating room for possible synced copies from other servers, minimum space requirements is 10GB. To accommodate for the synced copy of the repository that could go up to 2TB
Data Retention
- Removable drives will keep data up to 3 days old
- Network synced drives/servers will keep data up to 30 days old
Initial Setup
This applies to any new server that will be backed up (this process could be scriptable eventually)
- Create a user named "backup"
- Give that user certificate access through ssh (to and from) the other servers (in the backup group). Tutorials available here
- Add a new lvm called archive that will be mounted as /archive on startup (add an entry to /etc/fstab). LVM tutorial available here
- Create the following directories and give "backup" write permission to those directories
/archive/blaze-backup /archive/repo-backup /var/log/blaze-backup
- Schedule backup as root (crontab entry). It will look something similar to this:
20 2 * * * ADMHOST=$(cat /etc/sysconfig/blaze); BKU=backup; export ADMHOST BKU; scp $BKU@$ADMHOST:/usr/local/bin/blaze \ /usr/local/bin/; chown -R $BKU:$BKU /usr/local/bin/blaze; chmod 755 /usr/local/bin/blaze; /usr/local/bin/blaze
Scheduling
- In Chile it runs at 01:35 AM UTC
- In Hongkong it runs at 02:05 AM UTC
- In Australia it runs at 02:20 AM UTC
- In Ireland it runs at 03:35 AM UTC
- In Iraq it runs at 01:30 AM EDT
Repo Sync
- Add Content (to do for max)
Backing up in Removable Drives
- Add Content (to do for max)
Restore from Backups
This process is not automated yet. If an older copy is required for the postgresql dump or any scripts or configuration files, please copy the necessary zip file from the /archive/blaze-backup/[hostname]/ and unzip it to a new location and copy the desired file.
Conclusion
ARM is still a developing technology and expectations are that The Fedora ARM Project at Seneca CDOT will be highly active for the next few years. Depending on the level of activity the backup strategy in place may become out of date in a matter of months or it may need to be supplemented with other solutions.
Appendix
Backup Layout
| ||||
| Hongkong | /etc/ /usr/local/bin/ /var/spool/cron/ | Configurations, Cronjobs and Scripts | Iraq and Ireland | /archive/blaze-backup/hongkong/ |
| Ireland | /etc/ /usr/local/bin/ /var/spool/cron/ pg_dump | Configurations, Cronjobs, PostgreSQL and Scripts | Iraq and Chile | /archive/blaze-backup/ireland/ |
| Australia | /etc/ /usr/local/bin/ /var/spool/cron/ | Configurations, Cronjobs and Scripts | Iraq and Hongkong | /archive/blaze-backup/australia/ |
| Chile | /etc/ /usr/local/bin/ /var/spool/cron/ /mnt/koji/ | Configurations, Cronjobs, Repo and Scripts | Iraq and Australia | /archive/blaze-backup/chile/ /archive/repo-backup/ |
| Iraq | /etc/ /usr/local/bin/ /var/spool/cron/ | Configurations, Cronjobs and Scripts | Chile | /archive/blaze-backup/iraq/ |
.
