OPS435 Python3 Lab 8

This lab is currently being reviewed. The final version will be ready by November 11, 2019

LAB OBJECTIVES

Completing this course will give you the prerequisites for getting into the DevOps field. A DevOps professional is a system/network administrator with programming skills. As an introduction to that field, we will look at Fabric in this lab. Using Fabric you can automate deploying software, monitoring, and updating many systems at the same time.

REFERENCE

These links are helpful for learning more about Fabric's features:

Category	Resource Link
Official Fabric website	[1]
Official Fabric tutorial	[2]
Better Fabric tutorial	[3]

You should have learned the following topics in OPS235 and or OPS335. Please review them to prepare for some of the tasks in this lab:

• How to allow a regular user to run the sudo command.
• man page on sudo
• How to use the sudoers configuration file: /etc/sudoers.

INVESTIGATION 1: Extra VM Setup

In order to experience Fabric's features in a realistic way, we're going to set up several virtual machines. To begin with they are all going to have the same configuration. Please make sure that each VM has direct network connect with other VMs you wish to control and configure.

PART 1 - Set up your controller

In this lab you will use your existing vm centos7 as a workstation to control other VMs which we'll call workers.

Install fabric using yum. Once it's installed you should have a fab command available.

PART 2 - Create master Worker image

Create a new virtual machine, and allocate for it 1GB of RAM and 8GB of disk space. Install a Basic Web Server configuration of CentOS in that VM using the same CentOS .iso file you used for your first machine in this course.

Make sure that:

• The hostname of the system is worker1.
• It has a static IP address appropriate for your virtual network.
• Create a regular user using your Seneca email name as the user name: [seneca_id].
• Add this new regular user to the wheel group using the following command:

    usermod -a -G wheel [seneca_id]

  This will allow the user to run the sudo command.
• After installation ensure that you can access worker1 from your main vm using the static IP address you've assigned to it.

Set up SSH key login

In order for an automated system to be able to connect to your VM and administer it - you will need to be able to connect to it using SSH keys. You've done this in both OPS235 and OPS335.

Create a new SSH key on your main VM with your regular user. Please do not use root. Then set things up so that your regular user on your controller VM can SSH to the worker VM as (the same regular user) without putting in a password.

PART 3 - Clone the Workers

We're only simulating the real world where you'd have hundreds of VMs in one or more clouds, but you can just imagine that the VMs you're creating on your computer are actually being created on an Amazon or Microsoft server.

** Optional ** Make four clones of the master worker image you've just created. Then make sure that each of them has a unique IP address. That's all you're required to change manually. All the other configuration on the workers (inlcuding the hostnames) will be set by Fabric. Normally you would have some kind of automation doing all this cloning and IP address assignment as well, but we don't have time for that this semester.

Make snapshots of all your workers so that you can easily restore them to the original state after you modify them.

INVESTIGATION 2: Fabric practice

We will start with some basics. Fabric runs python programs on the controller and the workers. You create an "instruction" file on your controller, and execute it on the controller using the fab program. When you do that - you specify which workers you want your instructions to be executed on.

The instructions are stored in a python file. Let's start with a simple one named fabfile.py (the default filename for fab):

PART 1: Simplest example

from fabric.api import *

# Will get the hostname of this worker:
def getHostname():
    name = run("hostname")
    print(name)

All this will do is get the hostname of the worker and print it (on the controller). We run it on the controller like this:

fab --fabfile=fabfile.py -H 192.168.56.11 getHostname

In the command above we're using the fab program to read the file fabfile.py and execute the getHostname function on the worker 192.168.56.11. Note that the IP address of your first worker will likely be different.

If you did all the setup right and you try to execute the command above - you will get a password prompt. That shouldn't happen, the point is for everything to be automated, and that's why you've set up SSH keys. To fix it, add the following to your fab file:

env.user = '[seneca_id]'

That should have worked, and you'd get output like this:

$ fab --fabfile=fabfile.py -H 192.168.56.11 getHostname
[192.168.56.11] Executing task 'getHostname'
[192.168.56.11] run: hostname
[192.168.56.11] out: www
[192.168.56.11] out: 

worker1

Done.
Disconnecting from 192.168.56.11... done.

In the above you have:

• Lines with an IP address telling you which worker the output is for/from.
• Messages from the controller (e.g. "Executing task...", and "run: ...").
• Output from the worker ("out: ...")
• Output on the controller from your fab file ("worker1" which came from the "print()" call)

You should get used to the above. It's a lot of output but it's important to understand where every part is coming from, so you are able to debug problems when they happen.

Part 2: Set up web server

Let's pretend that we needed to deploy a web server on several machines. We'll set up a simple example of such a deployment here.

Install Apache

Add a setupWebServer() function to your python file:

# Will set up a working web server with a pre-built website
def setupWebServer():
    run("hostnamectl set-hostname www")
    run("yum install httpd")
    run("systemctl enable httpd")
    run("systemctl start httpd")

Note that each call to "run()" will run a command on the worker. In this function we set the hostname of the machine to "www", install Apache, enable the Apache service, and start that service now. Pretty simple commands.

If you try to run it the same way as before:

$ fab --fabfile=fabfile.py -H 192.168.56.11 setupWebServer

You'll find that yum prompts you to answer questions, which you don't want to do in an automated environment. And also yum prints too much output, which also isn't helpful in an automated environment. We'll fix it by adding two switches to yum: "-y" and "-d1":

Notice also that all of the four commands can be run as many times as you want, the result will be the same. This is not always so easy.

At this point if you log in to worker1 - you should see a new hostname, and httpd installed and running (try with "systemctl status").

Deploy a website

Now that we have a web server running, we also want to put a website on it. The website can be of any complexity, but to keep this demonstration simple we'll have a single HTML file. You can pretend that it's as complex as you like. Create an index.html file like this:

<h1>My fancy web server</h1>

And since we're pretending that it's a large website with many files and directories, we'll compress it into an archive named webcontents.tar.bz2 using a tar command. You've done this since OPS235.

Once you have your archive, make sure it's in the same directory as your fab file. Then add the following to your setupWebServer() function:

    with cd("/var/www/html/"):
        put("webcontents.tar.bz2", ".")
        run("tar xvf webcontents.tar.bz2")
        run("rm webcontents.tar.bz2")

There is something weird in the code above that you haven't seen before but it's required for some uses of Fabric: the with statement.

The problem is that separate run commands each execute in a brand new session, each with its own shell. They are not like separate lines in a single shell script even though they look like they should be.

That means if you run a cd command and then a tar command separately - the tar command will not run in the directory where you think it will. In order to fix this you have to nest commands inside a with - it's like a run but with persistant results.

The code we added to the function will cd to the default web site directory on the worker, upload your web contents tarball from your controller to that directory on the worker, extract it, and delete the tarball.

After it's done - you should have a working web server and simple website on your worker1.

Except you won't be able to access it because of the firewall. We'll deal with that in the next section.

Part 2: Set up the firewall

Recall that in our OPS courses we've been using iptables instead of firewalld, which is installed by default in CentOS. Let's make sure that our workers have that set up as well. In the same fabfile.py you've been using all along, add a new function like this:

# Will uninstall firewalld and replace it with iptables
def setupFirewall():
    run("yum -y -d1 remove firewalld")
    run("yum -y -d1 install iptables-services")
    run("systemctl enable iptables")
    run("systemctl start iptables")

That should by now look prett obvious. On the worker you're going to uninstall firewalld, install iptables, and make sure that the iptables service is running.

Execute the function for worker1 and double-check that it worked.

Allow access to Apache through the firewall

The default setup of iptables also doesn't allow access to our web server. We'll need to add some more to our function to allow it. This would probably make more sense in setupWebServer() but for now let's put it into setupFirewall():

    run("iptables -I INPUT -p tcp --dport 80 -j ACCEPT")
    run("iptables-save > /etc/sysconfig/iptables")

Easy enough, but there's on problem - if we run this more than once, we're going to end up with duplicate iptables rules for port 80 (check with iptables -L).

In order to avoid that - we have to first check whether the rule exists before we add it. We can do that like this:

iptables -C INPUT -p tcp --dport 80 -j ACCEPT"

Unfortunately that command answers "yes" or "no" by succeeding or failing depending on whether that rule exists. In Fabric when a command fails - the entire fab file execution stops, assuming that it's an unrecoverable error. We need to prevent that with another with statement:

    with settings(warn_only=True):
        firewallAlreadySetUp = run("iptables -C INPUT -p tcp --dport 80 -j ACCEPT")
        if firewallAlreadySetUp.return_code == 1:
            ... move your iptables rules setup here ...

Test your new setupFirewall function on worker1, and make sure it opens access to Apache but does not create duplicate rules every time it's run.

INVESTIGATION 3: Multiplying your work

After completing all the previous parts of the lab - you should have a working fabfile.py with two working functions: setupFirewall() and setupWebServer().

** Optional **You were asked to test them on worker1. Now let's run these two functions on all your workers at the same time. The command is almost the same, except for the list of IP addresses:

fab --fabfile=fabfile.py -H 192.168.56.11,192.168.56.12,192.168.56.13,192.168.56.14,192.168.56.15 setupWebServer

Again - your IP addresses will be different but the command will be the same.

You can also reconfigure the firewall on all the workers at the same time, using a command like this on your controller:

fab --fabfile=fabfile.py -H 192.168.56.11,192.168.56.12,192.168.56.13,192.168.56.14,192.168.56.15 setupFirewall

And imagine that you might have 10, 50, 100 servers to do this on - could you do it without the automation?

Final Task - Apply fabfile.py to your VM on myvmlab

Since your account on your vm on myvmlab is a regular user with sudo privilege. You need to make the following changes to your fabfile.py before applying it to your vm on myvmlab:

• Change env.user from 'root' to your account on your vm in myvmlab.
• Change all the commands that need super user privilege from calling the run() function to instead calling the sudo() function.

Test your updated fabfile.py until you get the same result as when you apply it to your own worker VM.

LAB 8 SIGN-OFF (SHOW INSTRUCTOR)

Have Ready to Show Your Instructor:

• Complete all the parts of the lab and show the version of your fabfile.py which works on your vm on myvmlab.