Difference between revisions of "OPS435 Python3 Lab 8"

From CDOT Wiki
Jump to: navigation, search
(→‎INVESTIGATION 1: Extra VM Setup)
 
(50 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
= LAB OBJECTIVES =
 
= LAB OBJECTIVES =
−
 
+
:0. Review SSH setup and remote shell execution
−
:1. Explore the Fabric package (written in Python) and study its command line tool "fab".
+
:1. Explore the Fabric Python library and its command line tool "fab".
−
:2. Create remote tasks using Python module that interface with the Fabric's API.
+
:2. Create Fabric scripts utilizing Fabric's API to define tasks for the '''fab''' command.
−
:3. Use the '''fab CLI''' (command line interface) to execute regular/administrative tasks on remote host with the '''Fabric''' framework.
+
:3. Use the '''fab''' command to execute fabric script to perform pre-defined tasks on remote Linux machines.
  
 
== Overview ==
 
== Overview ==
−
: Fabric is a Python library and command-line tool for streamling the use of SSH for application deployment or system administration tasks. It has two major components:
+
: Fabric is a Python library and command-line tool for streamlining the use of SSH for application deployment or system administration tasks. It has two major components:
 
:# a command-line interface program called "fab" that lets you execute arbitrary Python functions  
 
:# a command-line interface program called "fab" that lets you execute arbitrary Python functions  
 
:# a set of Python APIs that you can use and call in your Python functions to make executing shell commands over SSH much easier.  
 
:# a set of Python APIs that you can use and call in your Python functions to make executing shell commands over SSH much easier.  
−
: We are going use the "fab" and API of Fabric to write and execute Python functions (or tasks), to automate interactions with remote Linux machines in this lab.  
+
: We are going use the Fabric API and its '''fab''' command to define and execute Python functions (or tasks), to automate interactions with remote Linux machines in this lab.  
  
 
== REFERENCE ==
 
== REFERENCE ==
Line 37: Line 37:
 
:[http://www.fabfile.org/]
 
:[http://www.fabfile.org/]
 
|}
 
|}
−
: <font color='blue'>Please note that the version of Fabric we are going to use on matrix.senecacollege.ca for this lab is 1.14 and it supports only Python version 2.</font>
+
: <font color='blue'>Please note that the version of Fabric installed on matrix.senecacollege.ca for this lab is 1.14 and it supports only Python version 2.</font> The Fabric script files we are going to create in this lab have to meet Python version 2.x requirements. (e.g. print is a keyword, not a built-in function in Python 2.x)
 
:2. You should have some experience on the following topics in OPS235 and or OPS335. Please review them to prepare for the activities in this lab:
 
:2. You should have some experience on the following topics in OPS235 and or OPS335. Please review them to prepare for the activities in this lab:
 
:* create and configure a regular user on a Linux system.  
 
:* create and configure a regular user on a Linux system.  
Line 45: Line 45:
 
:* Retrieve current firewall setting using the iptables -L -n -v command
 
:* Retrieve current firewall setting using the iptables -L -n -v command
  
−
= INVESTIGATION 1: The Fabric Framework =
+
= INVESTIGATION 1: The Fabric Environment =
  
−
: The Fabric framework consists of the following components:
+
: The Fabric environment consists of the following components:
−
:# the Fabric Python Library - the fabric package
+
 
−
:# the Fabric API - fabric.ap
+
:# Controller workstation - the machine that has the Fabric package installed and runs the "fab" command
−
:# the Fabric CLI - fab - run Fabric script, default to fabfile.py in the current working directory
+
:## the Fabric Python Library - the fabric package (already installed on matrix)
−
:# Fabric script: contains Python functions (or tasks) to be executed by the "fab" CLI.
+
:## the Fabric API - fabric.api
−
:# Controller workstation - the machine that has the Fabric package installed and will run the "fab" CLI
+
:## the Fabric command - '''fab''': run Fabric script, name of the script is default to fabfile.py in the current working directory unless specified otherwise with the '-f' option.
−
:# Remote machine: the target machine on which a Fabric task will be executed.
+
:## Fabric script: contains fabric environment object value and Python functions (or tasks) to be executed by the '''fab''' command.
 +
:# Remote machine: the target machine on which one or more Fabric tasks will be executed.
 +
:## running the ssh server daemon
 +
:## use public key (or password based) authentication for ssh connection
  
−
== PART 1 - Configure your controller workstation ==
+
== PART 1 - Configure and test your controller workstation ==
−
: In this lab you will use your user account on matrix.senecacollege.ca as your Fabric controller workstation.  
+
: In this lab you will use your login account on matrix.senecacollege.ca as your Fabric controller workstation.  
  
−
: The Fabric package version 1.14.0 has already been installed on matrix.senecacollege.ca. You should have access to the '''fab''' command on matrix. Login to matrix.senecacollege.ca and run the following command to confirm the version of the fabric package:
+
: The Fabric package version 1.14.0 has already been installed on matrix.senecacollege.ca. You should have access to the '''fab''' command on matrix. Login to matrix.senecacollege.ca and run the following command to confirm the version of the fabric package:<source lang='bash'>
−
<pre>
+
fab --version
−
    $ fab --version
+
</source>
−
</pre>
+
: Type the following command to get the command line options of the fab command:<source lang='bash'>
−
: Type the following command to get the command line option of fab:<source lang='bash'>
 
 
fab --help
 
fab --help
 
</source>
 
</source>
−
You should get something similar to the following:<source lang="bash">
+
:You should get something similar to the following:<source lang="bash">
 
Usage: fab [options] <command>[:arg1,arg2=val2,host=foo,hosts='h1;h2',...] ...
 
Usage: fab [options] <command>[:arg1,arg2=val2,host=foo,hosts='h1;h2',...] ...
  
Line 139: Line 141:
 
                         number of concurrent processes to use in parallel mode
 
                         number of concurrent processes to use in parallel mode
 
</source>
 
</source>
−
<font color='green'><b>Please note and study the following command-line options:
+
<font color='green'><b>Please note and study the following command-line options as they will be used in some of the activities in this lab:
 
:# -H,  
 
:# -H,  
 
:# -f,
 
:# -f,
Line 149: Line 151:
 
</b></font>
 
</b></font>
  
−
=== Set up SSH key login ===
+
== PART 2: Connect to VM in myvmlab.senecacollege.ca ==
−
: In order for an automated system to be able to connect to your VM and administer it - you will need to be able to connect to it using SSH keys. You've done this in both OPS235 and OPS335.
+
: You should have received an email from ITS containing the following information:
 +
:* account name: (usually 'student')
 +
:* password: (let's assume it is 'P@ssw0rd' for the following instruction in this lab)
 +
:* port number for SSH access via myvmlab.senecacollege.ca (e.g. 7200)
 +
: This VM will be used as the remote Linux machine in our Fabric environment. Login to matrix and try the following SSH command to test the connectivity between matrix and your assignment VM:
 +
<pre>
 +
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca
 +
student@myvmlab.senecacollege.ca's password:
 +
Last login: Fri Jul  3 11:06:24 2020 from mtrx-node05pd.dcm.senecacollege.ca
 +
</pre>
 +
: Once you are on your VM, try the following commands: hostname, id, and df, and record the results for later comparison with the results of other commands:<source lang='bash'>
 +
[student@centos7 ~]$ hostname
 +
centos7
 +
[student@centos7 ~]$ id
 +
uid=1002(student) gid=1002(student) groups=1002(student),10(wheel)
 +
[student@centos7 ~]$ df
 +
Filesystem              1K-blocks    Used Available Use% Mounted on
 +
devtmpfs                  878260      0    878260  0% /dev
 +
tmpfs                      889792      0    889792  0% /dev/shm
 +
tmpfs                      889792    9492    880300  2% /run
 +
tmpfs                      889792      0    889792  0% /sys/fs/cgroup
 +
/dev/mapper/centos-root  38680112 1745524  36934588  5% /
 +
/dev/sda2                1038336  331228    707108  32% /boot
 +
/dev/sda1                  204580  11296    193284  6% /boot/efi
 +
/dev/mapper/centos-home  18880512  33160  18847352  1% /home
 +
tmpfs                      177960      0    177960  0% /run/user/1002
 +
</source>
 +
:Logout from your VM and get back to matrix.
 +
:The previous SSH command when executed successfully, created a login shell on the remote machine. If the previous SSH command is followed by a specific bash command, it will be executed on the remote host instead of creating a login shell. Consider the following:<source lang='bash'>
 +
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca 'hostname;id;df'
 +
student@myvmlab.senecacollege.ca's password:
 +
centos7
 +
uid=1002(student) gid=1002(student) groups=1002(student),10(wheel)
 +
Filesystem              1K-blocks    Used Available Use% Mounted on
 +
devtmpfs                  878260      0    878260  0% /dev
 +
tmpfs                      889792      0    889792  0% /dev/shm
 +
tmpfs                      889792    9492    880300  2% /run
 +
tmpfs                      889792      0    889792  0% /sys/fs/cgroup
 +
/dev/mapper/centos-root  38680112 1745608  36934504  5% /
 +
/dev/sda2                1038336  331228    707108  32% /boot
 +
/dev/sda1                  204580  11296    193284  6% /boot/efi
 +
/dev/mapper/centos-home  18880512  33160  18847352  1% /home
 +
tmpfs                      177960      0    177960  0% /run/user/1002
 +
</source>
 +
:The three shell commands: hostname, id, and df were executed sequentially. Compare the outputs above with the previous results when executing the corresponding commands in the login shell.
 +
:Please note that you were asked to provide the user's password for every SSH connection.
  
−
: Create a new SSH key pair (one private, and one public) on your main VM with your '''regular user''' (don't do it under root). Once you have both keys, set things up so that
+
==PART 3: Set up SSH login with public key authentication ==
−
:* your regular user on your '''controller VM''' can SSH to the worker VM as the same regular user without prompting for a password. (ie. add the contents of your pub key to ~/.ssh/authorized_keys)
+
: In order for your controller workstation to automate tasks execution on your VM, you need to configure your VM to SSH public key authentication instead of password authentication. You've done this in both OPS235 and OPS335, and here is a summary on how to do it between your account on matrix and your VM:
−
:* your regular user on your '''controller VM''' can SSH to the worker VM as root without propmting for a password. (ie. add the contents of your pub key to /root/.ssh/authorized_keys)
 
  
−
== PART 3 - Clone the Workers ==
+
: Create a new SSH key pair (one private, and one public) under your account on matrix.senecacollege.ca.
−
: We're only simulating the real world where you'd have hundreds of VMs in one or more clouds, but you can just imagine that the VMs you're creating on your computer are actually being created on an Amazon or Microsoft Cloud.
+
: Once you have both keys, you can use the '''ssh-copy-id''' command to copy your public key to the student account on your VM, replace the port number with the correct value for your VM:<source lang='bash'>
 +
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 7200 student@myvmlab.senecacollege.ca
 +
</source>
 +
: The above command should add the contents of your pub key to ~/.ssh/authorized_keys under your student account on your VM.
 +
: Verify and confirm that your account on matrix can SSH to your VM as 'student' without prompting for a password:<source lang='bash'>
 +
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca
 +
Last login: Fri Jul  3 12:46:19 2020 from mtrx-node05pd.dcm.senecacollege.ca
 +
[student@centos7 ~]$ exit
 +
logout
 +
Connection to myvmlab.senecacollege.ca closed.
  
−
: '''** Optional **''' Make four clones of the master worker image you've just created. Then make sure that each of them has a unique IP address. That's all you're required to change manually. All the other configuration on the workers (inlcuding the hostnames) will be set by Fabric. Normally you would have some kind of automation doing all this cloning and IP address assignment as well, but we don't have time for that this semester.
+
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca 'date;hostname;id'
 +
Fri Jul  3 12:55:22 EDT 2020
 +
centos7
 +
uid=1002(student) gid=1002(student) groups=1002(student),10(wheel)
 +
[raymond.chan@mtrx-node05pd lab8]$
 +
</source>
 +
: If you got similar result as above, you have successfully configure your controller workstation and your VM to use public key authentication.
  
−
: Make snapshots of all your workers so that you can easily restore them to the original state after you modify them.
+
=INVESTIGATION 2 - Running the fab command in ad-hoc mode =
 +
: The fab command relies on SSH to make the connection to the remote machine before executing the intended commands. The fab command can run in ad-hoc mode:<source lang='bash'>
 +
fab [options] -- [shell commands]
 +
</source>
 +
: When running the fab command in ad-hoc mode, it is very similar to running the SSH with commands attached at the end.
 +
== PART 1: running non-privileged shell commands on remote machines ==
 +
: In the following example, we use the '''fab''' to execute the "date", "hostname", and "id" command remotely on our VM. Try the following ad-hoc fab commands and record their results for later use, replace the port number with the correct value for your VM:<source lang='bash'>
 +
[raymond.chan@mtrx-node05pd lab8]$ fab --host=myvmlab.senecacollege.ca --port=7200 --user=student -- 'date;hostname;id'
 +
[myvmlab.senecacollege.ca] Executing task '<remainder>'
 +
[myvmlab.senecacollege.ca] run: date;hostname;id
 +
[myvmlab.senecacollege.ca] out: Fri Jul  3 13:05:39 EDT 2020
 +
[myvmlab.senecacollege.ca] out: centos7
 +
[myvmlab.senecacollege.ca] out: uid=1002(student) gid=1002(student) groups=1002(student),10(wheel)
 +
[myvmlab.senecacollege.ca] out:
  
−
= INVESTIGATION 2: Fabric practice =
 
−
: We will start with some basics. Fabric runs python programs on the controller and the workers. You create an "instruction" file on your controller, and execute it on the controller using the '''fab''' program. When you do that - you specify which workers you want your instructions to be executed on.
 
  
−
: The instructions are stored in a python file. Let's start with a simple one named '''fabfile.py''' (the default filename used by fab without the '-f' optino):
+
Done.
 +
Disconnecting from myvmlab.senecacollege.ca:7200... done.
 +
[raymond.chan@mtrx-node05pd lab8]$
 +
</source>
 +
: Note that there is no password prompting if you complete part 3 successfully, otherwise, the SSH server daemon on your VM will prompt you for a password. The output from the fab's ad-hoc mode is not much different from the SSH command with shell command attached at the end, however, please note that the additional information on the output from the fab command can be very useful for record keeping purpose - what has been done and whether the commands had been carried out successfully or not.
  
−
== PART 1: Simplest example ==
+
== PART 2: running privileged commands on remote machines ==
−
===Getting the hostname on the remote worker===
+
: We say that running an ad-hoc fab command is very similar to the SSH command with shell commands attached at the end. Let's try both with privileged commands, like the "yum" command.
−
: <source lang="python">from fabric.api import *
 
  
−
# set the name of the user on the remote host
+
=== Run the "yum" command on remote machine with SSH ===
−
env.user = '[seneca_id]'
+
: By default, your VM doesn't have the "tree" rpm package installed. You can verify this with the following SSH command (remember to replace the port number with the correct value for your VM):<source lang='bash'>
−
 
+
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca "yum list tree"
−
# Will get the hostname of this worker:
+
Loaded plugins: fastestmirror
−
 
+
Loading mirror speeds from cached hostfile
−
def getHostname():
+
* base: centos.mirror.colo-serv.net
−
    name = run("hostname")
+
* extras: centos.mirror.colo-serv.net
−
    print(name)
+
* updates: centos.mirror.ca.planethoster.net
 +
Available Packages
 +
tree.x86_64                          1.6.0-10.el7                          base
 +
[raymond.chan@mtrx-node05pd lab8]$
 
</source>
 
</source>
−
: To check for syntax error, run the following command in the same directory as your fabfile.py:<source lang="bash">
+
: Please note that the tree package is "Available", but not yet installed.
−
fab -l
+
: Let't try to install the "tree" package with the shell command "yum install tree -y":<source lang='bash'>
 +
[raymond.chan@mtrx-node05pd lab8]$ ssh -p student@myvmlab.senecacollege.ca "yum install tree -y"
 +
Loaded plugins: fastestmirror
 +
You need to be root to perform this command.
 
</source>
 
</source>
−
: you should get a list of tasks stored in your fabfile.py:<source lang="bash">
+
: Using the "yum" command to query rpm package doesn't need special privilege, however, it does when you try to install or remove rpm packages.
−
[rchan@centos7 lab8]$ fab -f fabfile.py -l
+
: Your "student" account on your VM was configured to allow you to run the "sudo" command to perform software management using the "yum" command. Let's login to your VM and try the following "sudo" command to install and then remove the "tree" rpm package:<source lang='bash'>
−
Available commands:
+
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca
 +
Last login: Fri Jul  3 16:51:07 2020 from mtrx-node05pd.dcm.senecacollege.ca
 +
[student@centos7 ~]$ sudo yum install tree -y
 +
[sudo] password for student:
 +
Loaded plugins: fastestmirror
 +
Loading mirror speeds from cached hostfile
 +
* base: less.cogeco.net
 +
* extras: centos.mirror.colo-serv.net
 +
* updates: mirror.calgah.com
 +
Resolving Dependencies
 +
--> Running transaction check
 +
---> Package tree.x86_64 0:1.6.0-10.el7 will be installed
 +
--> Finished Dependency Resolution
  
−
    getHostname
+
Dependencies Resolved
−
</source>
 
−
: To perform the task of getHostname on the worker machine 192.168.122.169 (replace with the actual IP of your worker VM), we run it on the controller machine like this:<source lang="bash">
 
−
[rchan@centos7 lab8]$ fab -f fabfile.py -H 192.168.122.169 getHostname
 
−
[192.168.122.169] Executing task 'getHostname'
 
−
[192.168.122.169] run: hostname
 
−
[192.168.122.169] out: c7-rchan
 
−
[192.168.122.169] out:
 
  
−
c7-rchan
+
========================================================================================================================
 +
Package                  Arch                      Version                            Repository                Size
 +
========================================================================================================================
 +
Installing:
 +
tree                    x86_64                    1.6.0-10.el7                        base                      46 k
  
−
Done.
+
Transaction Summary
−
Disconnecting from 192.168.122.169... done.
+
========================================================================================================================
−
</source>
+
Install  1 Package
−
:All this has done is get the hostname of the worker and print it (on the controller).
 
−
:In the command above we're using the fab program to import the file fabfile.py and execute the getHostname function on the worker 192.168.122.169. Note that the IP address of your first worker will likely be different.
 
  
−
: If you did all the setup right and you get a password prompt when execute the above command, read the prompt carefully and see who's password it prompted you for. If it is not the same as your [seneca_id], verify that you have the following line in your fabfile and you can ssh to your worker vm without password:
+
Total download size: 46 k
 +
Installed size: 87 k
 +
Downloading packages:
 +
tree-1.6.0-10.el7.x86_64.rpm                                                                    |  46 kB  00:00:00
 +
Running transaction check
 +
Running transaction test
 +
Transaction test succeeded
 +
Running transaction
 +
  Installing : tree-1.6.0-10.el7.x86_64                                                                            1/1
 +
  Verifying  : tree-1.6.0-10.el7.x86_64                                                                            1/1
  
−
:<source lang="python">env.user = '[seneca_id]'</source>
+
Installed:
 +
  tree.x86_64 0:1.6.0-10.el7
  
−
: In the above you have:
+
Complete!
−
:* Lines with an IP address telling you which worker the output is for/from.
+
[student@centos7 ~]$
−
:* Messages from the controller (e.g. "Executing task...", and "run: ...").
+
</source>
−
:* Output from the worker ("out: ...")
+
: Please note that when you run the "sudo" command the first time, it asks you for the user's password (i.e. user student's password). Let's now remote the "tree" package:<source lang='bash'>
−
:* Output on the controller from your fab file ("worker1" which came from the "print()" call)
+
[student@centos7 ~]$ yum remove tree -y
 +
Loaded plugins: fastestmirror
 +
You need to be root to perform this command.
 +
[student@centos7 ~]$ sudo yum remove tree -y
 +
Loaded plugins: fastestmirror
 +
Resolving Dependencies
 +
--> Running transaction check
 +
---> Package tree.x86_64 0:1.6.0-10.el7 will be erased
 +
--> Finished Dependency Resolution
  
−
:You should get used to the above. It's a lot of output but it's important to understand where every part is coming from, so you are able to debug problems when they happen.
+
Dependencies Resolved
  
−
== Part 2: Set up more administrative tasks==
+
========================================================================================================================
 +
Package                  Arch                      Version                            Repository                Size
 +
========================================================================================================================
 +
Removing:
 +
tree                    x86_64                    1.6.0-10.el7                      @base                      87 k
  
−
:Let's pretend that we need collect the disk usage on several machines so that we can plan for storage maintenance. We'll set up a simple example of such a deployment here.
+
Transaction Summary
 +
========================================================================================================================
 +
Remove  1 Package
  
−
=== Getting the disk usage on remote worker ===
+
Installed size: 87 k
 +
Downloading packages:
 +
Running transaction check
 +
Running transaction test
 +
Transaction test succeeded
 +
Running transaction
 +
  Erasing    : tree-1.6.0-10.el7.x86_64                                                                            1/1
 +
  Verifying  : tree-1.6.0-10.el7.x86_64                                                                            1/1
  
−
:Add a getDiskUsage() function to your fabfile.py file:
+
Removed:
 +
  tree.x86_64 0:1.6.0-10.el7
  
−
:<source lang="python"># to get the disk usage on remote worker
+
Complete!
−
def getDiskUsage():
+
[student@centos7 ~]$
−
    current_time = run('date')
+
</source>
−
    diskusage = run('df -H')
+
: The above tests confirm that the student user is allowed to run the yum command to install and remove rpm package. Now let's logout from the VM and go back to matrix. On matrix, try to run the sudo command using SSH:<source lang='bash'>
−
    header = 'Current Disk Usage at '+current_time
+
[student@centos7 ~]$ exit
−
    print(header)
+
logout
−
    print(diskusage)
+
Connection to myvmlab.senecacollege.ca closed.
 +
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7211 student@myvmlab.senecacollege.ca "sudo yum install tree -y"
 +
sudo: no tty present and no askpass program specified
 +
[raymond.chan@mtrx-node05pd lab8]$
 
</source>
 
</source>
 +
: The above error indicated that you need a tty for the SSH session to prompt you for the sudo password. Please look up the ssh man page to find out the option which turn on a tty for the SSH session.
  
−
:Note that each call to "run()" will run a command on the worker. In this function we get the date/time of the remote work, and then get the disk usage. The print() function print out both the values returned.
+
=== Run the privileged yum command on remote machine using ad-hoc fab command ===
 +
: Let's try the corresponding ad-hoc fab command on your VM:<source lang='bash'>
 +
fab --host=myvmlab.senecacollege.ca --port=7200 --user=student -- 'sudo yum install tree -y'
 +
</source>
 +
: Type in your user student's password when prompted for "sudo password", the yum install command to install the tree rpm package should be executed successfully. If the tree rpm package is already installed, you can remove it with the following ad-hoc fab command: <source lang='bash'>
 +
fab --host=myvmlab.senecacollege.ca --port=7200 --user=student -- 'sudo yum remove tree -y'
 +
</source>
 +
: Try remove the "tree" rpm package with the appropriate ad-hoc fab command.
  
−
:If you try to run it the same way as before:
+
= INVESTIGATION 3: Running the fab command in script mode =
 +
: From investigation 2, we can see that running '''fab''' in ad-hoc mode is quick, straight forward, and easy. However, the rich output generated can not be easily captured and processed. If you have a need to capture and process the output generated by the commands executed on the remote machines, the solution is to run the '''fab''' command in script mode.
 +
: The first step in running the '''fab''' command in script mode is to create a fabric script file.
 +
: Let's start with a simple fabric script file to demonstrate some basic concepts that use the API from the Fabric python library.
 +
: On matrix, cd to your lab8 directory and create a simple fabric script file named '''fabfile.py''' (this is the default filename used by the fab command when you invoke it without the '-f' optino):
  
−
<pre>$ fab --fabfile=fabfile.py -H 192.168.122.169 getDiskUsage</pre>
+
== PART 1: Non-privileged task example ==
 +
===Create non-privileged tasks: Getting the hostname of remote machines===
 +
: Add the following contents to the default fabric script called "fabfile.py" in your lab8 directory:<source lang="python">
 +
from fabric.api import *
  
−
:You should get the following output:<source lang="bash">
+
# set the name of the user login to the remote host
−
[rchan@centos7 lab8]$ fab --fabfile=fabfile.py -H 192.168.122.169 getDiskUsage
+
env.user = 'student'
−
[192.168.122.169] Executing task 'getDiskUsage'
 
−
[192.168.122.169] run: date
 
−
[192.168.122.169] out: Sun Nov 10 13:17:16 EST 2019
 
−
[192.168.122.169] out:
 
  
−
[192.168.122.169] run: df -H
+
# Define the task to get the hostname of remote machines:
−
[192.168.122.169] out: Filesystem              Size  Used Avail Use% Mounted on
+
def getHostname():
−
[192.168.122.169] out: devtmpfs                947M    0  947M  0% /dev
+
     name = run("hostname")
−
[192.168.122.169] out: tmpfs                    964M     0  964M  0% /dev/shm
+
     print("The host name is:",name)
−
[192.168.122.169] out: tmpfs                    964M  9.7M  954M  2% /run
+
</source>
−
[192.168.122.169] out: tmpfs                    964M     0  964M  0% /sys/fs/cgroup
 
−
[192.168.122.169] out: /dev/mapper/centos-root  7.7G  5.6G  2.1G  73% /
 
−
[192.168.122.169] out: /dev/vda1                1.1G  298M  766M  29% /boot
 
−
[192.168.122.169] out: tmpfs                    193M  17k  193M  1% /run/user/42
 
−
[192.168.122.169] out: tmpfs                    193M    0  193M  0% /run/user/1000
 
−
[192.168.122.169] out:
 
  
−
Current Disk Usage at Sun Nov 10 13:17:16 EST 2019
+
: To check for syntax error in the fabric script, run the following command in the lab8 directory where it contains the fabric script named "fabfile.py":<source lang="bash">
−
Filesystem              Size  Used Avail Use% Mounted on
+
fab -l
−
devtmpfs                947M    0  947M  0% /dev
 
−
tmpfs                    964M    0  964M  0% /dev/shm
 
−
tmpfs                    964M  9.7M  954M  2% /run
 
−
tmpfs                    964M    0  964M  0% /sys/fs/cgroup
 
−
/dev/mapper/centos-root  7.7G  5.6G  2.1G  73% /
 
−
/dev/vda1                1.1G  298M  766M  29% /boot
 
−
tmpfs                    193M  17k  193M  1% /run/user/42
 
−
tmpfs                    193M    0  193M  0% /run/user/1000
 
−
 
 
−
Done.
 
−
Disconnecting from 192.168.122.169... done.
 
 
</source>
 
</source>
 +
: you should get a list of tasks defined in your fabfile.py:<source lang="bash">
 +
[rchan@centos7 lab8]$ fab -l
 +
Available commands:
  
−
=== Update all the rpm packages on remote worker ===
+
     getHostname
−
:Let's pretend that we need to update software packages installed on several machines due to security patches. Let's name the task as 'performSoftwareUpdate()':<source lang="python">
 
−
# to perform software update on remote worker
 
−
def performSoftwareUpdate():
 
−
     status = run('yum update -y')
 
−
    print(status)
 
 
</source>
 
</source>
−
: Do a syntax check with the "fab -l" command.
+
: To perform the task of getHostname on your VM (replace with the actual port # for connecting to your VM), run the fab command on matrix:<source lang="bash">
−
: When you try to run it the same way as before, you encounter some issue as shown below:<source lang="bash">
+
[raymond.chan@mtrx-node05pd lab8]$ fab --hosts=myvmlab.senecacollege.ca --port=7200 getHostname
−
[rchan@centos7 lab8]$ fab --fabfile=fabfile.py -H 192.168.122.169 performSoftwareUpdate
+
[myvmlab.senecacollege.ca] Executing task 'getHostname'
−
[192.168.122.169] Executing task 'performSoftwareUpdate'
+
[myvmlab.senecacollege.ca] run: hostname
−
[192.168.122.169] run: yum update -y
+
[myvmlab.senecacollege.ca] out: centos7
−
[192.168.122.169] out: Loaded plugins: fastestmirror, langpacks
+
[myvmlab.senecacollege.ca] out:
−
[192.168.122.169] out: You need to be root to perform this command.
 
−
[192.168.122.169] out:  
 
  
 +
The host name is: centos7
  
−
Fatal error: run() received nonzero return code 1 while executing!
+
Done.
−
 
+
Disconnecting from myvmlab.senecacollege.ca:7200... done.
−
Requested: yum update -y
+
[raymond.chan@mtrx-node05pd lab8]$
−
Executed: /bin/bash -l -c "yum update -y"
 
−
 
 
−
Aborting.
 
−
Disconnecting from 192.168.122.169... done.
 
 
</source>
 
</source>
−
: As you already know, you need superuser privilege in order to perform software update on a Linux system. There are two ways to do it on Fabric. The first one is simple. Edit you fabfile.py and change the env.user line as shown below:<source lang="python">
+
: Notice that there is no need to specify the user name at the '''fab''' command line since we defined it in the fabric script file (env.user = 'student'). Also notice that we can capture the host name returned from the "hostname" command and print it out together with an descriptive text in a line.
  
−
env.user = 'root'
+
:In the above executed '''fab''' command, the fab program imports the fabric script named "fabfile.py" and execute the getHostname function on the VM connect at port 7200 on myvmlab.senecacollege.ca. Note that the port number for your first will likely be of a different value.
  
−
</source>
+
: If you did all the setup right and you got a password prompt when execute the above command, read the prompt carefully and see who's password it was prompting you for. If it is not for the user student, verify that you have the following line in your fabfile.py and you can ssh to your VM as the user student without password:
−
: Save the fabfile.py with the change and run it again.
 
−
: If you see the password prompt again, make sure that you can ssh from your controller as a regular user to your worker vm as root without password.
 
−
: The other way is to replace all the run() function calls for commands that need superuser privilege by the sudo() function calls in your fabfile.py. You are asked to investigate this in the final investigation of this lab.
 
  
−
== Part 3: Setting and Checking Security Configuration ==
+
:<source lang="python">env.user = 'student'</source>
  
−
: Recall that in our OPS courses we've been using iptables instead of firewalld, which is installed by default in CentOS. Let's make sure that our workers have that set up as well. In the same '''fabfile.py''' you've been using all along, add a new function like this:
+
: In the above output from the '''fab''' command, you have:
 +
:* Lines with the FQDN of the remote machine you are working on.
 +
:* Messages from the controller workstation (e.g. "Executing task...", and "run: ...").
 +
:* Output from the remote machine ("out: ...")
 +
:* Output generated on the controller workstation from your fab file (the print statement)
  
−
: <source lang="python">
+
:You should get used to the above messages from the '''fab''' command. It's a lot of output but it's important to understand where every part is coming from, so you are able to debug problems when they happen.
−
# Will uninstall firewalld and replace it with iptables
 
−
def setupFirewall():
 
−
    run("yum -y -d1 remove firewalld")
 
−
    run("yum -y -d1 install iptables-services")
 
−
    run("systemctl enable iptables")
 
−
    run("systemctl start iptables")
 
−
</source>
 
  
−
: That should by now look pretty obvious. On the worker you're going to uninstall firewalld, install iptables, and make sure that the iptables service is running.
+
== PART 2: Privileged Tasks Examples ==
 +
===Creat privileged tasks: install and remove rpm package on remote machines===
 +
: Add the following two new functions to the end of the fabric script "fabfile.py" in your lab8 directory:<source lang='bash'>
  
−
: Execute the function for worker1 and double-check that it worked.
+
def installPackage(pkg='dummy'):
−
: <font color='red'>'''**Warning**''' </font>Do not do this on your vm on myvmlab. If you do, you may lock yourself out for good.
+
    cmd = 'yum install ' + pkg + ' -y'
 +
    status = sudo(cmd)
 +
    print(status)
  
−
=== Check firewall configuration ===
+
def removePackage(pkg):
 +
    if pkg == '':
 +
      cmd = 'yum remove dummy -y'
 +
    else:
 +
      cmd = 'yum remove ' + pkg + ' -y'
 +
    status = sudo(cmd)
 +
    print(status)
 +
</source>
 +
: Note that both functions take one function argument in different ways. However, if no function argument is passed when calling the function, both will default to a string value of "dummy". Both functions call the sudo() from the fabric.api to execute the command contained in the "cmd" object on the remote machine via sudo.
 +
: To check for any syntax error in your updated fabric script, run the following command in the same directory as the fabfile.py:<source lang='bash'>
 +
fab -l
 +
</source>
 +
: You should get a list of tasks defined similar to the following:<source lang='bash'>
 +
[raymond.chan@mtrx-node05pd lab8]$ fab -l
 +
Available commands:
  
−
: To check your firewall configuration your remote worker, you can retrieve its current configuration by creating another Fabric task called "getFirewallConfigure(). Let's put the following code to your fabfile.py:<source lang="python">
+
    getHostname
−
def getFirewallConfig():
+
    installPackage
−
     fw_config = run("iptables -L -n -v")
+
     removePackage
−
    print(fw_config)
+
[raymond.chan@mtrx-node05pd lab8]$
 
</source>
 
</source>
 +
: If you only need to connect to the same remote machine, you can specify the host and port number in the fabfile.py to save some typing when executing the fab command. Add the following two lines after the env.user line in your fabfile.py:<source lang='bash'>
 +
env.port = '7200' # <-- please replace with the actual value of your VM's port number
 +
env.hosts =['myvmlab.senecacollege.ca']
 +
</source>
 +
: You can also store the user's password in this file so that it will respond to the "sudo password" prompt for sudo() call. It is not safe to do so as you can configure the sudo module on the remote machine not to ask for sudo password.
 +
: Now you can run the fab command without the "--host" and "--port" option.
 +
: Run the following two fab commands, note the results and compare their difference:<source lang='bash'>
 +
fab installPackage
  
−
: Try to run the getFirewallConfig() task the same way as before.
+
fab installPackage:tree
−
: Troubleshoot if you encounter any issue.
+
</source>
 +
: Run the following two fab commands, note the results and compare their difference:<source lang='bash'>
 +
fab removePackage
  
−
= INVESTIGATION 3: Multiplying your work =
+
fab removePackage:tree
 +
</source>
  
−
: After completing all the previous parts of the lab - you should have a working fabfile.py with three working functions: getDiskUsage(), performSoftwareUpdate() and getFirewallConfig().
+
== Part 2: Create remote task for updating rpm packages ==
 +
: Add a new function called "updatePackage" to your fabfile.py according to the following requirements:
 +
:* Accept optional function argument as the rpm package name
 +
:* If no function argument was given when called, default to all the packages installed
 +
: The output of the updatePackage when executed, should produce similar output as shown below:
 +
:1. Update a single package:<source lang='bash'>
 +
fab updatePackage:tree
 +
</source>
 +
: Sample output:<source lang='bash'>
 +
[raymond.chan@mtrx-node05pd lab8]$ fab updatePackage:tree
 +
[myvmlab.senecacollege.ca] Executing task 'updatePackage'
 +
[myvmlab.senecacollege.ca] sudo: yum update tree -y
 +
[myvmlab.senecacollege.ca] out: sudo password:
 +
[myvmlab.senecacollege.ca] out: Loaded plugins: fastestmirror
 +
[myvmlab.senecacollege.ca] out: Loading mirror speeds from cached hostfile
 +
[myvmlab.senecacollege.ca] out:  * base: less.cogeco.net
 +
[myvmlab.senecacollege.ca] out:  * extras: centos.mirror.ca.planethoster.net
 +
[myvmlab.senecacollege.ca] out:  * updates: less.cogeco.net
 +
[myvmlab.senecacollege.ca] out: No packages marked for update
 +
[myvmlab.senecacollege.ca] out:
  
−
'''** Optional **'''You were asked to test them on worker1. Now let's run these three functions on all your workers at the same time. The command is almost the same, except for the list of IP addresses:
+
Loaded plugins: fastestmirror
 +
Loading mirror speeds from cached hostfile
 +
* base: less.cogeco.net
 +
* extras: centos.mirror.ca.planethoster.net
 +
* updates: less.cogeco.net
 +
No packages marked for update
  
−
<source lang="bash">fab --fabfile=fabfile.py -H 192.168.122.169,192.168.122.170,192.168.122.171,192.168.122.172 getDiskUsage</source>
+
Done.
 +
Disconnecting from myvmlab.senecacollege.ca:7200... done.
 +
[raymond.chan@mtrx-node05pd lab8]$
 +
</source>
 +
:2. Update all installed package:<source lang='bash'>
 +
fab updatePackage:
 +
</source>
 +
: The following output had been trimmed, only showing the first few lines:<source lang='bash'>
 +
[myvmlab.senecacollege.ca] Executing task 'updatePackage'
 +
[myvmlab.senecacollege.ca] sudo: yum update -y
 +
[myvmlab.senecacollege.ca] out: sudo password:
 +
[myvmlab.senecacollege.ca] out: Loaded plugins: fastestmirror
 +
[myvmlab.senecacollege.ca] out: Loading mirror speeds from cached hostfile
 +
[myvmlab.senecacollege.ca] out:  * base: less.cogeco.net
 +
[myvmlab.senecacollege.ca] out:  * extras: centos.mirror.ca.planethoster.net
 +
[myvmlab.senecacollege.ca] out:  * updates: less.cogeco.net
 +
...
  
−
: Again - your IP addresses will be different but the command will be the same.
+
  Verifying  : systemd-219-73.el7_8.5.x86_64                                                                      53/54
 +
  Verifying  : systemd-libs-219-73.el7_8.5.x86_64                                                                54/54
  
−
: You can also run all three tasks on all the workers at the same time, by adding any task to your fabfile.py:<source lang="python">
+
Removed:
−
def doAllThree():
+
  kernel.x86_64 0:3.10.0-862.el7
−
    getDiskUsage()
 
−
    getFirewallConfig()
 
−
    performSoftwareUpdate()
 
−
</source>
 
−
: And run the following command on your controller:
 
  
−
<source lang="bash">fab --fabfile=fabfile.py -H 192.168.122.169,192.168.122.170,192.168.122.171,192.168.122.172 doAllThree</source>
+
Installed:
 +
  kernel.x86_64 0:3.10.0-1127.13.1.el7
  
−
And imagine that you might have 10 tasks to be done on 10, 50, 100 servers - could you do it without the automation?
+
Updated:
 +
  bind-export-libs.x86_64 32:9.11.4-16.P2.el7_8.6          binutils.x86_64 0:2.27-43.base.el7_8.1
 +
  ca-certificates.noarch 0:2020.2.41-70.0.el7_8            device-mapper.x86_64 7:1.02.164-7.el7_8.2
 +
  device-mapper-event.x86_64 7:1.02.164-7.el7_8.2          device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2
 +
  device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2          kernel-tools.x86_64 0:3.10.0-1127.13.1.el7
 +
  kernel-tools-libs.x86_64 0:3.10.0-1127.13.1.el7          lvm2.x86_64 7:2.02.186-7.el7_8.2
 +
  lvm2-libs.x86_64 7:2.02.186-7.el7_8.2                    microcode_ctl.x86_64 2:2.1-61.10.el7_8
 +
  net-snmp.x86_64 1:5.7.2-48.el7_8.1                      net-snmp-agent-libs.x86_64 1:5.7.2-48.el7_8.1
 +
  net-snmp-libs.x86_64 1:5.7.2-48.el7_8.1                  net-snmp-utils.x86_64 1:5.7.2-48.el7_8.1
 +
  ntp.x86_64 0:4.2.6p5-29.el7.centos.2                    ntpdate.x86_64 0:4.2.6p5-29.el7.centos.2
 +
  python-perf.x86_64 0:3.10.0-1127.13.1.el7                rsyslog.x86_64 0:8.24.0-52.el7_8.2
 +
  selinux-policy.noarch 0:3.13.1-266.el7_8.1              selinux-policy-targeted.noarch 0:3.13.1-266.el7_8.1
 +
  systemd.x86_64 0:219-73.el7_8.8                          systemd-libs.x86_64 0:219-73.el7_8.8
 +
  systemd-sysv.x86_64 0:219-73.el7_8.8                    yum-plugin-fastestmirror.noarch 0:1.1.31-54.el7_8
  
−
= INVESTIGATION 4 - Apply fabfile.py to your VM on myvmlab =
+
Complete!
−
== Replace run() function calls with sudo() ==
 
−
: Since your account on your vm on myvmlab is a regular user with sudo privilege. You need to make the following changes to your fabfile.py before applying it to your vm on myvmlab:
 
−
:* Change env.user from 'root' to your account on your vm in myvmlab.
 
−
:* Change all the commands that need super user privilege from calling the run() function to instead calling the sudo() function. Here is an example on replacing run() with sudo():<source lang="python">
 
  
−
def getFirewallConfig():
+
Done.
−
    fw_config = sudo("iptables -L -n -v")
+
Disconnecting from myvmlab.senecacollege.ca:7200... done.
−
    print(fw_config)
+
[raymond.chan@mtrx-node05pd lab8]$
 
</source>
 
</source>
  
−
: Test your updated fabfile.py until you get the same result as when you apply it to your own worker VM.
+
= Lab Exercise: Create a Fabric task called makeUser() =
−
 
+
: Study the Fabric API run(), sudo(), and put() and utilize them to create a new task called makeUser()
−
== Create a Fabric task called makeUser() ==
 
 
: The makeUser() function should perform the following:
 
: The makeUser() function should perform the following:
−
::* create a new user called "ops435p" with home directory "/home/ops435p".
+
:* create a new user called "ops435p" with home directory "/home/ops435p".
−
::* add it to the sudo group called "wheel".  
+
:* add it to the sudo group called "wheel".  
−
::* add your professor's ssh public key to the file named "authorized_keys" in ~ops435p/.ssh directory. Make sure that you set the proper permissions on both the directory ~ops435p/.ssh and the file "~ops435p/.ssh/authorized_keys.
+
:* ask your professor for a ssh public key and add it to the file named "authorized_keys" in the ~ops435p/.ssh directory. Make sure that you set the proper permissions on both the directory ~ops435p/.ssh and the file "~ops435p/.ssh/authorized_keys.
 
:Add the makeUser() to your final version of fabfile.py.  
 
:Add the makeUser() to your final version of fabfile.py.  
−
:Test the new task makeUser() on your local VM first, and deploy to your vm on myvmlab.
+
:Run the new task makeUser() on your VM.
−
:After the successful deployment of the makeUser() task on your vm on myvmlab, ask your professor to verify and confirm that the new user account "ops435p"  on myvmlab has been created correctly.
+
:Verify and confirm that your new makeUser() task is working correctly.
  
 
= LAB 8 SIGN-OFF (SHOW INSTRUCTOR) =
 
= LAB 8 SIGN-OFF (SHOW INSTRUCTOR) =
−
:'''Have Ready to Show Your Instructor:'''
+
: Complete all the parts of the lab and upload the version of your fabfile.py to Blackboard by the due date.
−
* Complete all the parts of the lab and upload the version of your fabfile.py which works on your vm on myvmlab to Blackboard.
 
  
−
[[Category:OPS435-Python]]
+
[[Category:OPS435-Python]][[Category:rchan]]

Latest revision as of 18:00, 4 July 2020

LAB OBJECTIVES

0. Review SSH setup and remote shell execution
1. Explore the Fabric Python library and its command line tool "fab".
2. Create Fabric scripts utilizing Fabric's API to define tasks for the fab command.
3. Use the fab command to execute fabric script to perform pre-defined tasks on remote Linux machines.

Overview

Fabric is a Python library and command-line tool for streamlining the use of SSH for application deployment or system administration tasks. It has two major components:
  1. a command-line interface program called "fab" that lets you execute arbitrary Python functions
  2. a set of Python APIs that you can use and call in your Python functions to make executing shell commands over SSH much easier.
We are going use the Fabric API and its fab command to define and execute Python functions (or tasks), to automate interactions with remote Linux machines in this lab.

REFERENCE

1. These links are helpful for learning more about Fabric's features:
Category Resource Link
Official Fabric tutorial
[1]
Better Fabric tutorial
[2]
Official Fabric website
[3]
Please note that the version of Fabric installed on matrix.senecacollege.ca for this lab is 1.14 and it supports only Python version 2. The Fabric script files we are going to create in this lab have to meet Python version 2.x requirements. (e.g. print is a keyword, not a built-in function in Python 2.x)
2. You should have some experience on the following topics in OPS235 and or OPS335. Please review them to prepare for the activities in this lab:
  • create and configure a regular user on a Linux system.
  • configure and manage sudo privilege for a regular user
  • Configure sudoers using the visudo command
  • using the yum command to install, remove, and update rpm packages
  • Retrieve current firewall setting using the iptables -L -n -v command

INVESTIGATION 1: The Fabric Environment

The Fabric environment consists of the following components:
  1. Controller workstation - the machine that has the Fabric package installed and runs the "fab" command
    1. the Fabric Python Library - the fabric package (already installed on matrix)
    2. the Fabric API - fabric.api
    3. the Fabric command - fab: run Fabric script, name of the script is default to fabfile.py in the current working directory unless specified otherwise with the '-f' option.
    4. Fabric script: contains fabric environment object value and Python functions (or tasks) to be executed by the fab command.
  2. Remote machine: the target machine on which one or more Fabric tasks will be executed.
    1. running the ssh server daemon
    2. use public key (or password based) authentication for ssh connection

PART 1 - Configure and test your controller workstation

In this lab you will use your login account on matrix.senecacollege.ca as your Fabric controller workstation.
The Fabric package version 1.14.0 has already been installed on matrix.senecacollege.ca. You should have access to the fab command on matrix. Login to matrix.senecacollege.ca and run the following command to confirm the version of the fabric package:
fab --version
Type the following command to get the command line options of the fab command:
fab --help
You should get something similar to the following:
Usage: fab [options] <command>[:arg1,arg2=val2,host=foo,hosts='h1;h2',...] ...

Options:
  -h, --help            show this help message and exit
  -d NAME, --display=NAME
                        print detailed info about command NAME
  -F FORMAT, --list-format=FORMAT
                        formats --list, choices: short, normal, nested
  -I, --initial-password-prompt
                        Force password prompt up-front
  --initial-sudo-password-prompt
                        Force sudo password prompt up-front
  -l, --list            print list of possible commands and exit
  --set=KEY=VALUE,...   comma separated KEY=VALUE pairs to set Fab env vars
  --shortlist           alias for -F short --list
  -V, --version         show program's version number and exit
  -a, --no_agent        don't use the running SSH agent
  -A, --forward-agent   forward local agent to remote end
  --abort-on-prompts    abort instead of prompting (for password, host, etc)
  -c PATH, --config=PATH
                        specify location of config file to use
  --colorize-errors     Color error output
  -D, --disable-known-hosts
                        do not load user known_hosts file
  -e, --eagerly-disconnect
                        disconnect from hosts as soon as possible
  -f PATH, --fabfile=PATH
                        python module file to import, e.g. '../other.py'
  -g HOST, --gateway=HOST
                        gateway host to connect through
  --gss-auth            Use GSS-API authentication
  --gss-deleg           Delegate GSS-API client credentials or not
  --gss-kex             Perform GSS-API Key Exchange and user authentication
  --hide=LEVELS         comma-separated list of output levels to hide
  -H HOSTS, --hosts=HOSTS
                        comma-separated list of hosts to operate on
  -i PATH               path to SSH private key file. May be repeated.
  -k, --no-keys         don't load private key files from ~/.ssh/
  --keepalive=N         enables a keepalive every N seconds
  --linewise            print line-by-line instead of byte-by-byte
  -n M, --connection-attempts=M
                        make M attempts to connect before giving up
  --no-pty              do not use pseudo-terminal in run/sudo
  -p PASSWORD, --password=PASSWORD
                        password for use with authentication and/or sudo
  -P, --parallel        default to parallel execution method
  --port=PORT           SSH connection port
  -r, --reject-unknown-hosts
                        reject unknown hosts
  --sudo-password=SUDO_PASSWORD
                        password for use with sudo only
  --system-known-hosts=SYSTEM_KNOWN_HOSTS
                        load system known_hosts file before reading user
                        known_hosts
  -R ROLES, --roles=ROLES
                        comma-separated list of roles to operate on
  -s SHELL, --shell=SHELL
                        specify a new shell, defaults to '/bin/bash -l -c'
  --show=LEVELS         comma-separated list of output levels to show
  --skip-bad-hosts      skip over hosts that can't be reached
  --skip-unknown-tasks  skip over unknown tasks
  --ssh-config-path=PATH
                        Path to SSH config file
  -t N, --timeout=N     set connection timeout to N seconds
  -T N, --command-timeout=N
                        set remote command timeout to N seconds
  -u USER, --user=USER  username to use when connecting to remote hosts
  -w, --warn-only       warn, instead of abort, when commands fail
  -x HOSTS, --exclude-hosts=HOSTS
                        comma-separated list of hosts to exclude
  -z INT, --pool-size=INT
                        number of concurrent processes to use in parallel mode

Please note and study the following command-line options as they will be used in some of the activities in this lab:

  1. -H,
  2. -f,
  3. -i,
  4. -l,
  5. --port
  6. --user
  7. --initial-sudo-password-prompt

PART 2: Connect to VM in myvmlab.senecacollege.ca

You should have received an email from ITS containing the following information:
  • account name: (usually 'student')
  • password: (let's assume it is 'P@ssw0rd' for the following instruction in this lab)
  • port number for SSH access via myvmlab.senecacollege.ca (e.g. 7200)
This VM will be used as the remote Linux machine in our Fabric environment. Login to matrix and try the following SSH command to test the connectivity between matrix and your assignment VM:
 [raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca
 student@myvmlab.senecacollege.ca's password:
 Last login: Fri Jul  3 11:06:24 2020 from mtrx-node05pd.dcm.senecacollege.ca
Once you are on your VM, try the following commands: hostname, id, and df, and record the results for later comparison with the results of other commands:
[student@centos7 ~]$ hostname
centos7
[student@centos7 ~]$ id
uid=1002(student) gid=1002(student) groups=1002(student),10(wheel)
[student@centos7 ~]$ df
Filesystem              1K-blocks    Used Available Use% Mounted on
devtmpfs                   878260       0    878260   0% /dev
tmpfs                      889792       0    889792   0% /dev/shm
tmpfs                      889792    9492    880300   2% /run
tmpfs                      889792       0    889792   0% /sys/fs/cgroup
/dev/mapper/centos-root  38680112 1745524  36934588   5% /
/dev/sda2                 1038336  331228    707108  32% /boot
/dev/sda1                  204580   11296    193284   6% /boot/efi
/dev/mapper/centos-home  18880512   33160  18847352   1% /home
tmpfs                      177960       0    177960   0% /run/user/1002
Logout from your VM and get back to matrix.
The previous SSH command when executed successfully, created a login shell on the remote machine. If the previous SSH command is followed by a specific bash command, it will be executed on the remote host instead of creating a login shell. Consider the following:
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca 'hostname;id;df'
student@myvmlab.senecacollege.ca's password:
centos7
uid=1002(student) gid=1002(student) groups=1002(student),10(wheel)
Filesystem              1K-blocks    Used Available Use% Mounted on
devtmpfs                   878260       0    878260   0% /dev
tmpfs                      889792       0    889792   0% /dev/shm
tmpfs                      889792    9492    880300   2% /run
tmpfs                      889792       0    889792   0% /sys/fs/cgroup
/dev/mapper/centos-root  38680112 1745608  36934504   5% /
/dev/sda2                 1038336  331228    707108  32% /boot
/dev/sda1                  204580   11296    193284   6% /boot/efi
/dev/mapper/centos-home  18880512   33160  18847352   1% /home
tmpfs                      177960       0    177960   0% /run/user/1002
The three shell commands: hostname, id, and df were executed sequentially. Compare the outputs above with the previous results when executing the corresponding commands in the login shell.
Please note that you were asked to provide the user's password for every SSH connection.

PART 3: Set up SSH login with public key authentication

In order for your controller workstation to automate tasks execution on your VM, you need to configure your VM to SSH public key authentication instead of password authentication. You've done this in both OPS235 and OPS335, and here is a summary on how to do it between your account on matrix and your VM:
Create a new SSH key pair (one private, and one public) under your account on matrix.senecacollege.ca.
Once you have both keys, you can use the ssh-copy-id command to copy your public key to the student account on your VM, replace the port number with the correct value for your VM:
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 7200 student@myvmlab.senecacollege.ca
The above command should add the contents of your pub key to ~/.ssh/authorized_keys under your student account on your VM.
Verify and confirm that your account on matrix can SSH to your VM as 'student' without prompting for a password:
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca
Last login: Fri Jul  3 12:46:19 2020 from mtrx-node05pd.dcm.senecacollege.ca
[student@centos7 ~]$ exit
logout
Connection to myvmlab.senecacollege.ca closed.

[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca 'date;hostname;id'
Fri Jul  3 12:55:22 EDT 2020
centos7
uid=1002(student) gid=1002(student) groups=1002(student),10(wheel)
[raymond.chan@mtrx-node05pd lab8]$
If you got similar result as above, you have successfully configure your controller workstation and your VM to use public key authentication.

INVESTIGATION 2 - Running the fab command in ad-hoc mode

The fab command relies on SSH to make the connection to the remote machine before executing the intended commands. The fab command can run in ad-hoc mode:
fab [options] -- [shell commands]
When running the fab command in ad-hoc mode, it is very similar to running the SSH with commands attached at the end.

PART 1: running non-privileged shell commands on remote machines

In the following example, we use the fab to execute the "date", "hostname", and "id" command remotely on our VM. Try the following ad-hoc fab commands and record their results for later use, replace the port number with the correct value for your VM:
[raymond.chan@mtrx-node05pd lab8]$ fab --host=myvmlab.senecacollege.ca --port=7200 --user=student -- 'date;hostname;id'
[myvmlab.senecacollege.ca] Executing task '<remainder>'
[myvmlab.senecacollege.ca] run: date;hostname;id
[myvmlab.senecacollege.ca] out: Fri Jul  3 13:05:39 EDT 2020
[myvmlab.senecacollege.ca] out: centos7
[myvmlab.senecacollege.ca] out: uid=1002(student) gid=1002(student) groups=1002(student),10(wheel)
[myvmlab.senecacollege.ca] out:


Done.
Disconnecting from myvmlab.senecacollege.ca:7200... done.
[raymond.chan@mtrx-node05pd lab8]$
Note that there is no password prompting if you complete part 3 successfully, otherwise, the SSH server daemon on your VM will prompt you for a password. The output from the fab's ad-hoc mode is not much different from the SSH command with shell command attached at the end, however, please note that the additional information on the output from the fab command can be very useful for record keeping purpose - what has been done and whether the commands had been carried out successfully or not.

PART 2: running privileged commands on remote machines

We say that running an ad-hoc fab command is very similar to the SSH command with shell commands attached at the end. Let's try both with privileged commands, like the "yum" command.

Run the "yum" command on remote machine with SSH

By default, your VM doesn't have the "tree" rpm package installed. You can verify this with the following SSH command (remember to replace the port number with the correct value for your VM):
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca "yum list tree"
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: centos.mirror.colo-serv.net
 * extras: centos.mirror.colo-serv.net
 * updates: centos.mirror.ca.planethoster.net
Available Packages
tree.x86_64                          1.6.0-10.el7                           base
[raymond.chan@mtrx-node05pd lab8]$
Please note that the tree package is "Available", but not yet installed.
Let't try to install the "tree" package with the shell command "yum install tree -y":
[raymond.chan@mtrx-node05pd lab8]$ ssh -p student@myvmlab.senecacollege.ca "yum install tree -y"
Loaded plugins: fastestmirror
You need to be root to perform this command.
Using the "yum" command to query rpm package doesn't need special privilege, however, it does when you try to install or remove rpm packages.
Your "student" account on your VM was configured to allow you to run the "sudo" command to perform software management using the "yum" command. Let's login to your VM and try the following "sudo" command to install and then remove the "tree" rpm package:
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca
Last login: Fri Jul  3 16:51:07 2020 from mtrx-node05pd.dcm.senecacollege.ca
[student@centos7 ~]$ sudo yum install tree -y
[sudo] password for student:
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: less.cogeco.net
 * extras: centos.mirror.colo-serv.net
 * updates: mirror.calgah.com
Resolving Dependencies
--> Running transaction check
---> Package tree.x86_64 0:1.6.0-10.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================
 Package                  Arch                       Version                             Repository                Size
========================================================================================================================
Installing:
 tree                     x86_64                     1.6.0-10.el7                        base                      46 k

Transaction Summary
========================================================================================================================
Install  1 Package

Total download size: 46 k
Installed size: 87 k
Downloading packages:
tree-1.6.0-10.el7.x86_64.rpm                                                                     |  46 kB  00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : tree-1.6.0-10.el7.x86_64                                                                             1/1
  Verifying  : tree-1.6.0-10.el7.x86_64                                                                             1/1

Installed:
  tree.x86_64 0:1.6.0-10.el7

Complete!
[student@centos7 ~]$
Please note that when you run the "sudo" command the first time, it asks you for the user's password (i.e. user student's password). Let's now remote the "tree" package:
[student@centos7 ~]$ yum remove tree -y
Loaded plugins: fastestmirror
You need to be root to perform this command.
[student@centos7 ~]$ sudo yum remove tree -y
Loaded plugins: fastestmirror
Resolving Dependencies
--> Running transaction check
---> Package tree.x86_64 0:1.6.0-10.el7 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

========================================================================================================================
 Package                  Arch                       Version                            Repository                 Size
========================================================================================================================
Removing:
 tree                     x86_64                     1.6.0-10.el7                       @base                      87 k

Transaction Summary
========================================================================================================================
Remove  1 Package

Installed size: 87 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Erasing    : tree-1.6.0-10.el7.x86_64                                                                             1/1
  Verifying  : tree-1.6.0-10.el7.x86_64                                                                             1/1

Removed:
  tree.x86_64 0:1.6.0-10.el7

Complete!
[student@centos7 ~]$
The above tests confirm that the student user is allowed to run the yum command to install and remove rpm package. Now let's logout from the VM and go back to matrix. On matrix, try to run the sudo command using SSH:
[student@centos7 ~]$ exit
logout
Connection to myvmlab.senecacollege.ca closed.
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7211 student@myvmlab.senecacollege.ca "sudo yum install tree -y"
sudo: no tty present and no askpass program specified
[raymond.chan@mtrx-node05pd lab8]$
The above error indicated that you need a tty for the SSH session to prompt you for the sudo password. Please look up the ssh man page to find out the option which turn on a tty for the SSH session.

Run the privileged yum command on remote machine using ad-hoc fab command

Let's try the corresponding ad-hoc fab command on your VM:
fab --host=myvmlab.senecacollege.ca --port=7200 --user=student -- 'sudo yum install tree -y'
Type in your user student's password when prompted for "sudo password", the yum install command to install the tree rpm package should be executed successfully. If the tree rpm package is already installed, you can remove it with the following ad-hoc fab command: 
fab --host=myvmlab.senecacollege.ca --port=7200 --user=student -- 'sudo yum remove tree -y'
Try remove the "tree" rpm package with the appropriate ad-hoc fab command.

INVESTIGATION 3: Running the fab command in script mode

From investigation 2, we can see that running fab in ad-hoc mode is quick, straight forward, and easy. However, the rich output generated can not be easily captured and processed. If you have a need to capture and process the output generated by the commands executed on the remote machines, the solution is to run the fab command in script mode.
The first step in running the fab command in script mode is to create a fabric script file.
Let's start with a simple fabric script file to demonstrate some basic concepts that use the API from the Fabric python library.
On matrix, cd to your lab8 directory and create a simple fabric script file named fabfile.py (this is the default filename used by the fab command when you invoke it without the '-f' optino):

PART 1: Non-privileged task example

Create non-privileged tasks: Getting the hostname of remote machines

Add the following contents to the default fabric script called "fabfile.py" in your lab8 directory:
from fabric.api import *

# set the name of the user login to the remote host
env.user = 'student'

# Define the task to get the hostname of remote machines:
def getHostname():
    name = run("hostname")
    print("The host name is:",name)
To check for syntax error in the fabric script, run the following command in the lab8 directory where it contains the fabric script named "fabfile.py":
fab -l
you should get a list of tasks defined in your fabfile.py:
[rchan@centos7 lab8]$ fab -l
Available commands:

    getHostname
To perform the task of getHostname on your VM (replace with the actual port # for connecting to your VM), run the fab command on matrix:
[raymond.chan@mtrx-node05pd lab8]$ fab --hosts=myvmlab.senecacollege.ca --port=7200 getHostname
[myvmlab.senecacollege.ca] Executing task 'getHostname'
[myvmlab.senecacollege.ca] run: hostname
[myvmlab.senecacollege.ca] out: centos7
[myvmlab.senecacollege.ca] out:

The host name is: centos7

Done.
Disconnecting from myvmlab.senecacollege.ca:7200... done.
[raymond.chan@mtrx-node05pd lab8]$
Notice that there is no need to specify the user name at the fab command line since we defined it in the fabric script file (env.user = 'student'). Also notice that we can capture the host name returned from the "hostname" command and print it out together with an descriptive text in a line.
In the above executed fab command, the fab program imports the fabric script named "fabfile.py" and execute the getHostname function on the VM connect at port 7200 on myvmlab.senecacollege.ca. Note that the port number for your first will likely be of a different value.
If you did all the setup right and you got a password prompt when execute the above command, read the prompt carefully and see who's password it was prompting you for. If it is not for the user student, verify that you have the following line in your fabfile.py and you can ssh to your VM as the user student without password:
env.user = 'student'
In the above output from the fab command, you have:
  • Lines with the FQDN of the remote machine you are working on.
  • Messages from the controller workstation (e.g. "Executing task...", and "run: ...").
  • Output from the remote machine ("out: ...")
  • Output generated on the controller workstation from your fab file (the print statement)
You should get used to the above messages from the fab command. It's a lot of output but it's important to understand where every part is coming from, so you are able to debug problems when they happen.

PART 2: Privileged Tasks Examples

Creat privileged tasks: install and remove rpm package on remote machines

Add the following two new functions to the end of the fabric script "fabfile.py" in your lab8 directory:
def installPackage(pkg='dummy'):
    cmd = 'yum install ' + pkg + ' -y'
    status = sudo(cmd)
    print(status)

def removePackage(pkg):
    if pkg == '':
       cmd = 'yum remove dummy -y'
    else:
       cmd = 'yum remove ' + pkg + ' -y'
    status = sudo(cmd)
    print(status)
Note that both functions take one function argument in different ways. However, if no function argument is passed when calling the function, both will default to a string value of "dummy". Both functions call the sudo() from the fabric.api to execute the command contained in the "cmd" object on the remote machine via sudo.
To check for any syntax error in your updated fabric script, run the following command in the same directory as the fabfile.py:
fab -l
You should get a list of tasks defined similar to the following:
[raymond.chan@mtrx-node05pd lab8]$ fab -l
Available commands:

    getHostname
    installPackage
    removePackage
[raymond.chan@mtrx-node05pd lab8]$
If you only need to connect to the same remote machine, you can specify the host and port number in the fabfile.py to save some typing when executing the fab command. Add the following two lines after the env.user line in your fabfile.py:
env.port = '7200' # <-- please replace with the actual value of your VM's port number
env.hosts =['myvmlab.senecacollege.ca']
You can also store the user's password in this file so that it will respond to the "sudo password" prompt for sudo() call. It is not safe to do so as you can configure the sudo module on the remote machine not to ask for sudo password.
Now you can run the fab command without the "--host" and "--port" option.
Run the following two fab commands, note the results and compare their difference:
fab installPackage

fab installPackage:tree
Run the following two fab commands, note the results and compare their difference:
fab removePackage

fab removePackage:tree

Part 2: Create remote task for updating rpm packages

Add a new function called "updatePackage" to your fabfile.py according to the following requirements:
  • Accept optional function argument as the rpm package name
  • If no function argument was given when called, default to all the packages installed
The output of the updatePackage when executed, should produce similar output as shown below:
1. Update a single package:
fab updatePackage:tree
Sample output:
[raymond.chan@mtrx-node05pd lab8]$ fab updatePackage:tree
[myvmlab.senecacollege.ca] Executing task 'updatePackage'
[myvmlab.senecacollege.ca] sudo: yum update tree -y
[myvmlab.senecacollege.ca] out: sudo password:
[myvmlab.senecacollege.ca] out: Loaded plugins: fastestmirror
[myvmlab.senecacollege.ca] out: Loading mirror speeds from cached hostfile
[myvmlab.senecacollege.ca] out:  * base: less.cogeco.net
[myvmlab.senecacollege.ca] out:  * extras: centos.mirror.ca.planethoster.net
[myvmlab.senecacollege.ca] out:  * updates: less.cogeco.net
[myvmlab.senecacollege.ca] out: No packages marked for update
[myvmlab.senecacollege.ca] out:

Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: less.cogeco.net
 * extras: centos.mirror.ca.planethoster.net
 * updates: less.cogeco.net
No packages marked for update

Done.
Disconnecting from myvmlab.senecacollege.ca:7200... done.
[raymond.chan@mtrx-node05pd lab8]$
2. Update all installed package:
fab updatePackage:
The following output had been trimmed, only showing the first few lines:
[myvmlab.senecacollege.ca] Executing task 'updatePackage'
[myvmlab.senecacollege.ca] sudo: yum update -y
[myvmlab.senecacollege.ca] out: sudo password:
[myvmlab.senecacollege.ca] out: Loaded plugins: fastestmirror
[myvmlab.senecacollege.ca] out: Loading mirror speeds from cached hostfile
[myvmlab.senecacollege.ca] out:  * base: less.cogeco.net
[myvmlab.senecacollege.ca] out:  * extras: centos.mirror.ca.planethoster.net
[myvmlab.senecacollege.ca] out:  * updates: less.cogeco.net
...

  Verifying  : systemd-219-73.el7_8.5.x86_64                                                                      53/54
  Verifying  : systemd-libs-219-73.el7_8.5.x86_64                                                                 54/54

Removed:
  kernel.x86_64 0:3.10.0-862.el7

Installed:
  kernel.x86_64 0:3.10.0-1127.13.1.el7

Updated:
  bind-export-libs.x86_64 32:9.11.4-16.P2.el7_8.6          binutils.x86_64 0:2.27-43.base.el7_8.1
  ca-certificates.noarch 0:2020.2.41-70.0.el7_8            device-mapper.x86_64 7:1.02.164-7.el7_8.2
  device-mapper-event.x86_64 7:1.02.164-7.el7_8.2          device-mapper-event-libs.x86_64 7:1.02.164-7.el7_8.2
  device-mapper-libs.x86_64 7:1.02.164-7.el7_8.2           kernel-tools.x86_64 0:3.10.0-1127.13.1.el7
  kernel-tools-libs.x86_64 0:3.10.0-1127.13.1.el7          lvm2.x86_64 7:2.02.186-7.el7_8.2
  lvm2-libs.x86_64 7:2.02.186-7.el7_8.2                    microcode_ctl.x86_64 2:2.1-61.10.el7_8
  net-snmp.x86_64 1:5.7.2-48.el7_8.1                       net-snmp-agent-libs.x86_64 1:5.7.2-48.el7_8.1
  net-snmp-libs.x86_64 1:5.7.2-48.el7_8.1                  net-snmp-utils.x86_64 1:5.7.2-48.el7_8.1
  ntp.x86_64 0:4.2.6p5-29.el7.centos.2                     ntpdate.x86_64 0:4.2.6p5-29.el7.centos.2
  python-perf.x86_64 0:3.10.0-1127.13.1.el7                rsyslog.x86_64 0:8.24.0-52.el7_8.2
  selinux-policy.noarch 0:3.13.1-266.el7_8.1               selinux-policy-targeted.noarch 0:3.13.1-266.el7_8.1
  systemd.x86_64 0:219-73.el7_8.8                          systemd-libs.x86_64 0:219-73.el7_8.8
  systemd-sysv.x86_64 0:219-73.el7_8.8                     yum-plugin-fastestmirror.noarch 0:1.1.31-54.el7_8

Complete!

Done.
Disconnecting from myvmlab.senecacollege.ca:7200... done.
[raymond.chan@mtrx-node05pd lab8]$

Lab Exercise: Create a Fabric task called makeUser()

Study the Fabric API run(), sudo(), and put() and utilize them to create a new task called makeUser()
The makeUser() function should perform the following:
  • create a new user called "ops435p" with home directory "/home/ops435p".
  • add it to the sudo group called "wheel".
  • ask your professor for a ssh public key and add it to the file named "authorized_keys" in the ~ops435p/.ssh directory. Make sure that you set the proper permissions on both the directory ~ops435p/.ssh and the file "~ops435p/.ssh/authorized_keys.
Add the makeUser() to your final version of fabfile.py.
Run the new task makeUser() on your VM.
Verify and confirm that your new makeUser() task is working correctly.

LAB 8 SIGN-OFF (SHOW INSTRUCTOR)

Complete all the parts of the lab and upload the version of your fabfile.py to Blackboard by the due date.
Retrieved from "https://wiki.cdot.senecapolytechnic.ca/w/index.php?title=OPS435_Python3_Lab_8&oldid=146604"