Changes

← Older edit

OSTEP Infrastructure

13,402 bytes added, 11:18, 7 January 2015

→‎Introduction

[[Category:OSTEP SOP]][[Category:Pidora]]

= OSTEP Infrastructure SOP =

This page should be used for all OSTEP Infrastructure. Explaining the standard operating procedure of all tools, configuration, and programs.

= Pidora Repos =

* The Pidora Repositories are hosted at: http://pidora.ca/pidora/

== Symbolic Link Setup ==

=== Mash Repos ===

<pre>

~/pidora-rsync/mash/pidora-18-latest/

├── mash.log

├── pidora-18

│ ├── armhfp

│ └── source

├── pidora-18-rpfr-updates

│ ├── armhfp

│ └── SRPMS

├── pidora-18-rpfr-updates-testing

│ ├── armhfp

│ └── SRPMS

├── pidora-18-updates

│ ├── armhfp

│ └── SRPMS

└── pidora-18-updates-testing

├── armhfp

└── SRPMS

</pre>

=== Pidora main repo sym links ===

<pre>

~/public_html/pidora/releases/18/packages/

├── armhfp

│ ├── debug -> ~/pidora-rsync/mash/pidora-18-latest/pidora-18/armhfp/debug

│ └── os -> ~/pidora-rsync/mash/pidora-18-latest/pidora-18/armhfp/os

└── source

└── SRPMS -> ~/pidora-rsync/mash/pidora-18-latest/pidora-18/source/SRPMS

</pre>

=== Pidora updates repo sym links ===

<pre>

~/public_html/pidora/

├── rpfr-updates

│ ├── 18

│ │ ├── armhfp ->~/pidora-rsync/mash/pidora-18-latest/pidora-18-rpfr-updates/armhfp/

│ │ └── SRPMS ->~/pidora-rsync/mash/pidora-18-latest/pidora-18-rpfr-updates/SRPMS/

│ └── testing

│ └── 18

│ ├── armhfp ->~/pidora-rsync/mash/pidora-18-latest/pidora-18-rpfr-updates-testing/armhfp/

│ └── SRPMS ->~/pidora-rsync/mash/pidora-18-latest/pidora-18-rpfr-updates-testing/SRPMS/

└── updates

├── 18

│ ├── armhfp ->~/pidora-rsync/mash/pidora-18-latest/pidora-18-updates/armhfp/

│ └── SRPMS -> ~/pidora-rsync/mash/pidora-18-latest/pidora-18-updates/SRPMS/

└── testing

└── 18

├── armhfp ->~/pidora-rsync/mash/pidora-18-latest/pidora-18-updates-testing/armhfp/

└── SRPMS ->~/pidora-rsync/mash/pidora-18-latest/pidora-18-updates-testing/SRPMS/

</pre>

= Ansible Builder Configuration Management =

This group should be used for all builders that require static ip addresses. It will setup the custom ip address based on the resolved hostname inside the /etc/ansible/ansible_hosts file.

= Backup System =

== Introduction ==

Here is a copy of the script: [[Smart-Backup-Source]]

Here at CDOT, our current backup solution was a little archaic, and hard to expand on. I decided to make a new method of backup that can be run from a single computer and backup our entire infrastructure. This script is currently, as I'm writing this not in a finished state, however it is in a state where it works and is usable as a replacement to our previous system. I would like to pose a warning that this method of backup across systems is not a very secure method, and it does pose security threats. Since it does require you to give some users nopasswd sudo access to some or all programs. I am looking for a way around this, and would appreciate any input on this matter.

The script is to be run on the computer: bahamas

The script is to be run with the user: backup

=== Dependencies ===

The following software should be installed before using:

<pre>

yum install gcc

yum install autoconf

yum install python-pip

pip install pycrypto

pip install pysftp

pip install ecdsa

</pre>

== Goals ==

There were a few goals that were kept in mind with this script as it was developed:

<pre>

- Script resides on a single computer (complete)

- Do not run multiple backups using the same hard drive (complete)

- Check space requirements before performing a backup on source and destination (cancelled)

- Emails out daily reports on success or fail (work around with mail command)

- Logs all information /var/log/smart-bk/ (complete)

- Easy(ish) to add a new backup schedule (complete)

- Can view all backups that are currently running (complete)

- Can view all the backups in the queue to run (complete)

- Can view all the schedules that are added (complete)

- Records a record of all previously run backups (complete)

- Website to view status of currently running backups (complete)

</pre>

At this time a website can now view the information from the database, and the database can now keep a record of all previously run backups. Email was a issue, so, the output of a report is piped to a mail command running in the crontab. Checking space requirements was a silly goal for now, since trying to do this on the source and destination would increase the backup time by way too much(recursively looking through directories, which contain 100G+ data).

<s>At this time, not all of these goals have been completed, but I would like them to be sooner or later. Right now I'm setting up a little documentation on how it currently works, what it's missing, and what my next steps will be.</s>

== Scheduler System ==

The scheduler system is the main part of the script and uses a list of fields. Using this list of fields, the script can determine exactly when the backup should take place, if it has already run for the day, if it is still running, and all the details surrounding the specified backup. A person or bash script will add backups they would like to be performed to a schedule using specific parameters. A schedule looks like this:

<pre>

----------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------------

</pre>

Field details and explanations:

<pre>

id - This is just a unique field identifier.

day - This is the day the backup last was last run. This is used to check if the schedule is expired(in the past) or has already completed.

time - This is the time at which the backup will start. This allows you to order different schedules to happen earlier or later in the day.

type - This is the type of backup. Currently there are 3.

- archive backup wraps the directory specified in a tar archive and compresses it with bzip. Uses options: tar -cpjvf

- rsync is a very simple rsync that preserves most things. Uses options: rsync -aHAXEvz

- dbdump backup, this is specifically a koji db backup currently. Uses options: pg_dump koji

source_host - This host is the target for backup. You want the files backup up from here.

dest_host - This host is your backup storage location. All files backed up will go here.

source_dir - This directory correlates to source_host. This is the directory that is backed up.

dest_dir - This directory correlates to dest_host. This is where the backup is stored.

source_user - User to use on the source host.

dest_user - User to use on the dest host.

</pre>

== Database Structure ==

All data for this script is stored inside a sqlite3 db. The database file used should be called "schedules.db".

Create a new sqlite3 database:

<pre>

sqlite3 schedules.db

</pre>

Run the following sql statements to create the proper tables within the database

<pre>

sqlite> .schema

CREATE TABLE Logs(scheduleid INTEGER, status TEXT, errors TEXT, start_date TEXT, start_time TEXT, end_date TEXT, end_time TEXT);

CREATE TABLE Queue(scheduleid INTEGER, queuetime TEXT, FOREIGN KEY(scheduleid) REFERENCES Schedule(id));

CREATE TABLE Running(scheduleid INTEGER, starttime TEXT, FOREIGN KEY(scheduleid) REFERENCES Schedule(id));

CREATE TABLE Schedule(id INTEGER PRIMARY KEY, day TEXT, time TEXT, type TEXT, source_host TEXT, dest_host TEXT, source_dir TEXT, dest_dir TEXT, source_user TEXT, dest_user TEXT, desc TEXT);

</pre>

== How To Use sbk ==

sbk is a python script that allows you to specify options which change what functions the script will perform.

=== Help ===

Use the help option for help. This will show a description of the program, all available options, and the options descriptions.

<pre>

sbk -h

or

sbk --help

</pre>

=== View Schedule Information ===

sbk allows users to view all the schedules that have been created, all the current schedules in the queue, and all scheduled backups currently in progress.

<pre>

sbk -s

or

sbk --show

</pre>

=== Adding New Schedules/Backups ===

The backup user is able to add new backup schedules. These schedules will be run at their designated times. Unfortunately I couldn't find a way to add schedules without specifying a lot of options. However this is all done through the command line, so you can easily bash script a way of automating this.

Add a new schedule:

<pre>

[backup@bahamas ~]$ sbk --add --time "11:00" --backup-type archive --source-host japan --dest-host bahamas --source-dir /etc/ --dest-dir /data/backup/japan/etc/ --source-user backup --dest-user backup --desc "archive of japan /etc-> bahamas"

</pre>

=== Removing Schedules/Backups ===

In order to remove a schedule, a "sid" must be specified. This is simply the "id" of the schedule, which is unique to schedules. You can get this "id" by running "sbk -s".

Remove a schedule:

<pre>

[backup@bahamas ~]$ sbk --remove --sid=1

</pre>

=== Start the Backups ===

Start intelligently queuing schedules and initiate the backups. This next part is good to run in a crontab or in some automated way(see multiple instances).

<pre>

sbk -q

or

sbk --queue

</pre>

==== Multiple Instances ====

The way the script currently runs, it only runs one backup at a time, however you can run multiple instances of the "sbk --queue". If you run multiple instances, they will work together and increase the number of backups at the same time(as long as there are multiple backups!), this in turn should increase the speed to complete all backups. Running multiple instances should be safe and will not harm backups that are already running. It also will not run backups that are already running, or interfere with them in any way. My plan for the future of this script is to allow it to spawn it's own child processes which will perform multiple backups as well, HOWEVER, it still needs to be run multiple times during the day because it will only start backups whose time has expired(ex. if you run it at 11:00 and a backup has a time of 13:00, the schedule at 13:00 will not start until it's day field is expired).

So to get the most speed out of your backups, and finish all schedules as fast as possible, run this script in the crontab at a high frequency. What a high frequency is, is really up to you, there is hardly any harm in running it too often, as it will just end itself if there is nothing to do. Whether you run the script once a day or once every 10 minutes, the script will get the job done.

=== Advanced sbk Options ===

Most of these options are just for me to debug while making the script, however some of them may be useful for managing backups. Note: when using the "--sid=<id>" please replace <id> with a number.

Remove a single schedule from the queue.

<pre>

sbk --remove-queue --sid=<id>

</pre>

Remove a single schedule from running(this will NOT currently end the backup that is running).

<pre>

sbk --remove-run --sid=<id>

</pre>

Expire a schedule so that it appears as though it has not run today, useful if you want to force a backup to run a second time.

<pre>

sbk --expire --sid=<id>

</pre>

Add a schedule to the queue. This is similar to expire in function, except you don't have to wait for the time field to expire. Next time "sbk --queue" is run, it will run this backup.

<pre>

sbk --add-queue --sid=<id>

</pre>

=== Logging ===

[UPDATE]

A new logging section has been added to the schedule.db. This allows for specific logging events to be saved with dates and success or failure, for more accurate logging.

I could not figure out the format for the logging. Too many options. I went with a procedure where it makes a new log file each time the program is run. This could be a problem if you run the script too frequently, since it will make so many log files. I think the best idea would be to log to a single file, or to log into the sqlite3 database. I have not had time to change this yet.

Logging directory:

<pre>

/var/log/smart-bk/

</pre>

Logging format:

<pre>

smart-bk-yyyy-MM-dd-hh-mm.log

</pre>

== Backup Host Configuration ==

One of the main drawbacks to using this script is, it requires a bit of configuration on all computers using the backup system.

# backup user created on all computers

# backup user must be able to ssh without a password from any computer to any other as backup user

# backup user must have sudo access with the nopasswd option on the rsync program and tar program(Security risk! Giving rsync sudo access allows backup user to modify any file.)visudo: backup ALL=(ALL) NOPASSWD: /usr/bin/rsync, /bin/tar

# root user must be able to ssh to all backup users from any computer(This is annoying, trying to find a way around this.)

# add custom users such as koji to work with ssh no password to all backup users, give root access to koji user in the same way

# WARNING, make sure you disable the passwords on all these backup accounts, that way they can't log in and get access to root without a private key

# IMPORTANT - visudo on each machine you connect would like to connect to: Defaults:backup !requiretty

otherwise sudo will complain about not having a tty

This list of configurations, that need to be done to each computer, is annoying and could be done better. Currently looking for ways to change it. After these configurations are made, you can use this host in any backup schedule.

Oatley

198

edits

CDOT Wiki β

Changes

OSTEP Infrastructure

CDOT Wiki ^β