1,760
edits
Changes
no edit summary
= LAB OBJECTIVES =
:0. Review SSH setup and remote shell execution
:1. Explore the Fabric Python library and its command line tool "fab".
:2. Create Fabric scripts utilize utilizing Fabric's API to define tasks that can be executed by for the '''fab''' programcommand.:3. Use the '''fab''' command to execute fabric script to perform regular/administrative pre-defined tasks on remote Linux machines.
== Overview ==
:[http://www.fabfile.org/]
|}
: <font color='blue'>Please note that the version of Fabric we are going to use installed on matrix.senecacollege.ca for this lab is 1.14 and it supports only Python version 2.</font>The Fabric script files we are going to create in this lab have to meet Python version 2.x requirements. (e.g. print is a keyword, not a built-in function in Python 2.x)
:2. You should have some experience on the following topics in OPS235 and or OPS335. Please review them to prepare for the activities in this lab:
:* create and configure a regular user on a Linux system.
: Create a new SSH key pair (one private, and one public) under your account on matrix.senecacollege.ca.
: Once you have both keys, you can use the '''ssh-copy-id''' command to copy your public key to the student account on your VM, replace the port number with the correct value for your VM:<source lang='bash'>
ssh-copy-id -i ~/.ssh/id_rsa.pub -p 7200 student@myvmlab.senecacollege.ca
</source>
: If you got similar result as above, you have successfully configure your controller workstation and your VM to use public key authentication.
=Investigation INVESTIGATION 2 - Running the fab command in ad-hoc mode =
: The fab command relies on SSH to make the connection to the remote machine before executing the intended commands. The fab command can run in ad-hoc mode:<source lang='bash'>
fab [options] -- [shell commands]
: When running the fab command in ad-hoc mode, it is very similar to running the SSH with commands attached at the end.
== PART 1: running non-privileged shell commands on remote machines ==
: In the following example, we use the '''fab''' to execute the "date", "hostname", and "id" command remotely on our VM. Try the following ad-hoc fab commands and record their results for later use, replace the port number with the correct value for your VM:<source lang='bash'>
[raymond.chan@mtrx-node05pd lab8]$ fab --host=myvmlab.senecacollege.ca --port=7200 --user=student -- 'date;hostname;id'
[myvmlab.senecacollege.ca] Executing task '<remainder>'
=== Run the "yum" command on remote machine with SSH ===
: By default, your VM doesn't have the "tree" rpm package installed. You can verify this with the following SSH command(remember to replace the port number with the correct value for your VM):<source lang='bash'>
[raymond.chan@mtrx-node05pd lab8]$ ssh -p 7200 student@myvmlab.senecacollege.ca "yum list tree"
Loaded plugins: fastestmirror
</source>
: The above error indicated that you need a tty for the SSH session to prompt you for the sudo password. Please look up the ssh man page to find out the option which turn on a tty for the SSH session.
=== Run the privileged yum command on remote machine using ad-hoc fab command ===: Let's try the corresponding ad-hoc fab commandon your VM:<source lang='bash'>
fab --host=myvmlab.senecacollege.ca --port=7200 --user=student -- 'sudo yum install tree -y'
</source>
: Type in your user student's password when prompted for "sudo password", the yum install command to install the tree rpm package should be executed successfully.If the tree rpm package is already installed, you can remove it with the following ad-hoc fab command: <source lang='bash'>fab --host=myvmlab.senecacollege.ca --port=7200 --user=student -- 'sudo yum remove tree -y'</source>
: Try remove the "tree" rpm package with the appropriate ad-hoc fab command.
= INVESTIGATION 3: Running the fab command in script mode =: We will start with some basicsFrom investigation 2, we can see that running '''fab''' in ad-hoc mode is quick, straight forward, and easy. Fabric runs python programs on However, the controller rich output generated can not be easily captured and processed. If you have a need to capture and process the output generated by the workers. You create an "instruction" file commands executed on your controllerthe remote machines, and execute it on the controller using solution is to run the '''fab''' programcommand in script mode. When you do : The first step in running the '''fab''' command in script mode is to create a fabric script file.: Let's start with a simple fabric script file to demonstrate some basic concepts that - you specify which workers you want use the API from the Fabric python library.: On matrix, cd to your instructions to be executed onlab8 directory and create a simple fabric script file named '''fabfile.py''' (this is the default filename used by the fab command when you invoke it without the '-f' optino):
== PART 1: The instructions are stored in a python file. Let's start with a simple one named '''Non-privileged task example =====Create non-privileged tasks: Getting the hostname of remote machines===: Add the following contents to the default fabric script called "fabfile.py''' (the default filename used by fab without the '-f' optino)" in your lab8 directory:<source lang="python">from fabric.api import *
# Define the task to get the hostname of remote machines:
def getHostname():
name = run("hostname")
print("The host name is:",name)
</source>
: To check for syntax errorin the fabric script, run the following command in the same lab8 directory as your where it contains the fabric script named "fabfile.py":<source lang="bash">
fab -l
</source>
: you should get a list of tasks stored defined in your fabfile.py:<source lang="bash">[rchan@centos7 lab8]$ fab -f fabfile.py -l
Available commands:
getHostname
</source>
: To perform the task of getHostname on the worker machine 192.168.122.169 your VM (replace with the actual IP of port # for connecting to your worker VM), we run it the fab command on the controller machine like thismatrix:<source lang="bash">[rchanraymond.chan@centos7 mtrx-node05pd lab8]$ fab -f fabfile.py -H 192hosts=myvmlab.168senecacollege.122.169 ca --port=7200 getHostname[192.168myvmlab.122senecacollege.169ca] Executing task 'getHostname'[192myvmlab.168senecacollege.122.169ca] run: hostname[192.168myvmlab.122senecacollege.169ca] out: c7-rchancentos7[192.168myvmlab.122senecacollege.169ca] out:
Done.
Disconnecting from 192myvmlab.168senecacollege.122ca:7200.169..done. done[raymond.chan@mtrx-node05pd lab8]$
</source>
:All this has done Notice that there is get no need to specify the hostname of user name at the worker and print '''fab''' command line since we defined it in the fabric script file (on the controllerenv.user = 'student').:In the command above Also notice that we're using can capture the fab program to import host name returned from the file fabfile.py "hostname" command and execute the getHostname function on the worker 192.168.122.169. Note that the IP address of your first worker will likely be differentprint it out together with an descriptive text in a line.
: If you did all the setup right and you get a password prompt when execute In the above executed '''fab''' command, read the prompt carefully fab program imports the fabric script named "fabfile.py" and see who's password it prompted you forexecute the getHostname function on the VM connect at port 7200 on myvmlab.senecacollege.ca. If it is not the same as your [seneca_id], verify Note that you have the following line in port number for your fabfile and you can ssh to your worker vm without password:first will likely be of a different value.
:<source lang="python">envIf you did all the setup right and you got a password prompt when execute the above command, read the prompt carefully and see who's password it was prompting you for.If it is not for the user = '[seneca_id]'</source>student, verify that you have the following line in your fabfile.py and you can ssh to your VM as the user student without password:
: In the above you have::* Lines with an IP address telling you which worker the output is for/from.:* Messages from the controller (e.g. <source lang="Executing task...python", and "run: >env...").:* Output from the worker ("out: ..."):* Output on the controller from your fab file ("worker1" which came from the "print()" call)user = 'student'</source>
:You should get used to In the above. Itoutput from the '''fab''s a lot of output but it's important to understand where every part is coming fromcommand, so you have::* Lines with the FQDN of the remote machine you are able to debug problems when they happenworking on.:* Messages from the controller workstation (e.g. "Executing task...", and "run: ...").:* Output from the remote machine ("out: ..."):* Output generated on the controller workstation from your fab file (the print statement)
== PART 2:Let's pretend that we need collect the disk usage Privileged Tasks Examples =====Creat privileged tasks: install and remove rpm package on several remote machines so that we can plan for storage maintenance===: Add the following two new functions to the end of the fabric script "fabfile. Wepy" in your lab8 directory:<source lang='bash'll set up a simple example of such a deployment here.>
def installPackage(pkg='dummy'): cmd ='yum install ' + pkg + ' -y' status = Getting the disk usage on remote worker ===sudo(cmd) print(status)
def removePackage(pkg): if pkg == '':Add cmd = 'yum remove dummy -y' else: cmd = 'yum remove ' + pkg + ' -y' status = sudo(cmd) print(status)</source>: Note that both functions take one function argument in different ways. However, if no function argument is passed when calling the function, both will default to a getDiskUsagestring value of "dummy". Both functions call the sudo() function from the fabric.api to execute the command contained in the "cmd" object on the remote machine via sudo.: To check for any syntax error in your updated fabric script, run the following command in the same directory as the fabfile.py file:<source lang='bash'>fab -l</source>: You should get a list of tasks defined similar to the following:<source lang='bash'>[raymond.chan@mtrx-node05pd lab8]$ fab -lAvailable commands:
</source>
:Note that each call If you only need to connect to "run()" will run a command on the worker. In this function we get the date/time of the same remote workmachine, you can specify the host and then get port number in the disk usagefabfile. The print() function print out both py to save some typing when executing the values returnedfab command. :If you try to run it Add the following two lines after the same way as before: <pre>$ fab --fabfile=env.user line in your fabfile.py -H 192.168.122.169 getDiskUsage</pre> :You should get the following output:<source lang="'bash"'>[rchan@centos7 lab8]$ fab --fabfileenv.port =fabfile.py -H 192.168.122.169 getDiskUsage[192.168.122.169] Executing task 'getDiskUsage7200'[192.168.122.169] run: date[192.168.122.169] out: Sun Nov 10 13:17:16 EST 2019[192.168.122.169] out: [192.168.122.169] run: df # <-H[192.168.122.169] out: Filesystem Size Used Avail Use% Mounted on[192.168.122.169] out: devtmpfs 947M 0 947M 0% /dev[192.168.122.169] out: tmpfs 964M 0 964M 0% /dev/shm[192.168.122.169] out: tmpfs 964M 9.7M 954M 2% /run[192.168.122.169] out: tmpfs 964M 0 964M 0% /sys/fs/cgroup[192.168.122.169] out: /dev/mapper/centos-root 7.7G 5.6G 2.1G 73% /please replace with the actual value of your VM's port number[192env.168.122.169] out: /dev/vda1 1.1G 298M 766M 29% /boothosts =[192'myvmlab.168senecacollege.122.169] out: tmpfs 193M 17k 193M 1% /run/user/42[192.168.122.169ca'] out: tmpfs 193M 0 193M 0% /run/user/1000[192.168.122.169] out: Current Disk Usage at Sun Nov 10 13:17:16 EST 2019Filesystem Size Used Avail Use% Mounted ondevtmpfs 947M 0 947M 0% /devtmpfs 964M 0 964M 0% /dev/shmtmpfs 964M 9.7M 954M 2% /runtmpfs 964M 0 964M 0% /sys/fs/cgroup/dev/mapper/centos-root 7.7G 5.6G 2.1G 73% //dev/vda1 1.1G 298M 766M 29% /boottmpfs 193M 17k 193M 1% /run/user/42tmpfs 193M 0 193M 0% /run/user/1000 Done.Disconnecting from 192.168.122.169... done.
</source>
: You can also store the user's password in this file so that it will respond to the "sudo password" prompt for sudo() call. It is not safe to do so as you can configure the sudo module on the remote machine not to ask for sudo password.
: Now you can run the fab command without the "--host" and "--port" option.
: Run the following two fab commands, note the results and compare their difference:<source lang='bash'>
fab installPackage
</source>
: Do a syntax check with Run the "following two fab -l" command.: When you try to run it commands, note the same way as before, you encounter some issue as shown belowresults and compare their difference:<source lang="'bash"'>[rchan@centos7 lab8]$ fab --fabfile=fabfile.py -H 192.168.122.169 performSoftwareUpdate[192.168.122.169] Executing task 'performSoftwareUpdate'[192.168.122.169] run: yum update -y[192.168.122.169] out: Loaded plugins: fastestmirror, langpacks[192.168.122.169] out: You need to be root to perform this command.[192.168.122.169] out: Fatal error: run() received nonzero return code 1 while executing! Requested: yum update -yExecuted: /bin/bash -l -c "yum update -y"removePackage
</source>
== Part 2: Create remote task for updating rpm packages ==
: Add a new function called "updatePackage" to your fabfile.py according to the following requirements:
:* Accept optional function argument as the rpm package name
:* If no function argument was given when called, default to all the packages installed
: The output of the updatePackage when executed, should produce similar output as shown below:
:1. Update a single package:<source lang='bash'>
fab updatePackage:tree
</source>
: Save the fabfileSample output:<source lang='bash'>[raymond.py with the change and run it againchan@mtrx-node05pd lab8]$ fab updatePackage:tree[myvmlab.senecacollege.ca] Executing task 'updatePackage'[myvmlab.senecacollege.ca] sudo: yum update tree -y[myvmlab.senecacollege.ca] out: If you see the sudo password prompt again, make sure that you can ssh :[myvmlab.senecacollege.ca] out: Loaded plugins: fastestmirror[myvmlab.senecacollege.ca] out: Loading mirror speeds from your controller as a regular user to your worker vm as root without passwordcached hostfile[myvmlab.senecacollege.ca] out: * base: less.cogeco.net[myvmlab.senecacollege.ca] out: * extras: centos.mirror.ca.planethoster.net[myvmlab.senecacollege.ca] out: * updates: less.cogeco.net[myvmlab.senecacollege.ca] out: The other way is to replace all the run() function calls No packages marked for commands that need superuser privilege by the sudo() function calls in your fabfileupdate[myvmlab.py. You are asked to investigate this in the final investigation of this labsenecacollege.ca] out:
Done.Disconnecting from myvmlab.senecacollege.ca: Recall that in our OPS courses we've been using iptables instead of firewalld, which is installed by default in CentOS7200.. Let's make sure that our workers have that set up as well. In the same '''fabfiledone.py''' you've been using all along, add a new function like this: : <source lang="python"># Will uninstall firewalld and replace it with iptablesdef setupFirewall(): run("yum -y -d1 remove firewalld") run("yum [raymond.chan@mtrx-y -d1 install iptables-services") run("systemctl enable iptables") run("systemctl start iptables")node05pd lab8]$
</source>
: That should by now look pretty obvious. On the worker you're going to uninstall firewalld, install iptables, and make sure that the iptables service is running2. : Execute the function for worker1 and double-check that it worked.Update all installed package: <font colorsource lang='red'>'''**Warning**bash''' </font>Do not do this on your vm on myvmlab. If you do, you may lock yourself out for good. === Check firewall configuration === fab updatePackage: To check your firewall configuration your remote worker, you can retrieve its current configuration by creating another Fabric task called "getFirewallConfigure(). Let's put the following code to your fabfile.py:<source lang="python">def getFirewallConfig(): fw_config = run("iptables -L -n -v") print(fw_config)
</source>
: The following output had been trimmed, only showing the first few lines:<source lang='bash'>
[myvmlab.senecacollege.ca] Executing task 'updatePackage'
[myvmlab.senecacollege.ca] sudo: yum update -y
[myvmlab.senecacollege.ca] out: sudo password:
[myvmlab.senecacollege.ca] out: Loaded plugins: fastestmirror
[myvmlab.senecacollege.ca] out: Loading mirror speeds from cached hostfile
[myvmlab.senecacollege.ca] out: * base: less.cogeco.net
[myvmlab.senecacollege.ca] out: * extras: centos.mirror.ca.planethoster.net
[myvmlab.senecacollege.ca] out: * updates: less.cogeco.net
...
Verifying : Try to run the getFirewallConfig() task the same way as beforesystemd-219-73.el7_8.5.x86_64 53/54 Verifying : Troubleshoot if you encounter any issuesystemd-libs-219-73.el7_8.5.x86_64 54/54
Installed: After completing all the previous parts of the lab kernel.x86_64 0:3.10.0- you should have a working fabfile1127.py with three working functions: getDiskUsage(), performSoftwareUpdate() and getFirewallConfig()13.1.el7
</source>
== Lab Exercise: Create a Fabric task called makeUser() ==: Study the Fabric API run(), sudo(), and put() and utilize them to create a new task called makeUser()
: The makeUser() function should perform the following:
:Add the makeUser() to your final version of fabfile.py.
:Test Run the new task makeUser() on your local VM first, and deploy to your vm on myvmlab.:After the successful deployment of the Verify and confirm that your new makeUser() task on your vm on myvmlab, ask your professor to verify and confirm that the new user account "ops435p" on myvmlab has been created is working correctly.
= LAB 8 SIGN-OFF (SHOW INSTRUCTOR) =
:'''Have Ready to Show Your Instructor:'''* Complete all the parts of the lab and upload the version of your fabfile.py which works on your vm on myvmlab to Blackboardby the due date.
[[Category:OPS435-Python]][[Category:rchan]]