Difference between revisions of "OPS335 Lab 2b"

From CDOT Wiki
Jump to: navigation, search
(INVESTIGATION 1: IPTABLES TROUBLESHOOTING CHECKLIST)
(Online Submission (Ahad Mammadov's Classes only))
 
(65 intermediate revisions by 5 users not shown)
Line 1: Line 1:
==ADDITIONAL IPTABLES TROUBLESHOOTING==
+
[[Category:OPS335]][[Category:OPS335 Labs]]
  
===Purpose===
+
== OBJECTIVE & PREPARATION ==
 +
In Lab 2a, we set the firewall rules for your '''host''' machine. In this lab, we will '''create firewall rules for our virtual machines''' within our virtual private network. This lab will also apply '''best practices''' and '''troubleshooting techniques''' using iptables.
  
This lab will provide more hands-on experience with troubleshooting iptables issues.
+
=== Online Resources===
  
===IPTABLES and Routing Troubleshooting Resources===
+
* [http://www.microhowto.info/troubleshooting/troubleshooting_iptables.html Troubleshooting iptables]
  
 +
==INVESTIGATION 1: CUSTOM IPTABLES RULES ON A VM==
  
Some articles that you can use as a reference if you are still experiencing problems with iptables:
+
We will now '''set iptables rules for your vm1 machine'''.
* [http://zenit.senecac.on.ca/wiki/index.php/OPS335_Lab_1#Linux_Network_Connection_Configuration_Troubleshooting Test Network Connectivity] (From Lab1)
+
 
* [http://www.microhowto.info/troubleshooting/troubleshooting_iptables.html Troubleshooting iptables]
+
'''Perform the following Steps:'''
* [https://community.rackspace.com/products/f/25/t/248 Basic IPTABLES Troubleshooting]
+
 
 +
# Start your '''host''' machine, and launch your '''vm1''' machine.
 +
# Login to your '''root account''' on your '''vm1''' machine.
 +
# Issue a command (like you did in lab2a) to copy your default iptables rules to the file pathname:<br>'''/etc/sysconfig/iptables.original'''
 +
# Issue an ''iptables command'' to set the policy to disable '''all forwarding traffic''', and remove the rule that is rejecting it.
 +
# Next,  set the default policy to drop '''all inbound traffic''', and remove the rule that is rejecting traffic.
 +
# Issue an iptables command to list rules for verification.<br /><br />The remaining tasks will relate to that same '''inbound''' traffic chain:<br /><br />
 +
# Issue an ''iptables command'' to delete the default ssh rule.
 +
# Issue an ''iptables command'' to add a rule that allows ssh traffic (i.e. tcp packets with destination port 22) that originates from any machine within your virtual network.
 +
# Issue an ''iptables command'' to delete the default icmp rule.
 +
# Issue an ''iptables command'' to allow icmp traffic from addresses in your virtual network.
 +
# Test that your machines can still use ping and ssh to communicate with each other.
 +
# Save your rules in the location that iptables will automatically read from when it starts.
 +
# Reboot your machine and check that the new rules are being applied.  If they are not, resolve this issue before moving on.
 +
# Now copy the file to your other VMs and make it apply to them when they boot as well.
 +
# Reboot each machine and make sure this works before you move on.
 +
 
 +
'''Record your observations in this section on your OPS335 lab log-book'''
  
==INVESTIGATION 1: IPTABLES TROUBLESHOOTING CHECKLIST==
+
==INVESTIGATION 2: IPTABLES TROUBLESHOOTING CHECKLIST==
  
 
By now, you have probably discovered that a simple mistake in your iptables rules can have very serious and unexpected consequences for not only your services, but the network connectivity in general. There is a general process (checklist) that you can following to help troubleshoot iptables in order to fix the problem.
 
By now, you have probably discovered that a simple mistake in your iptables rules can have very serious and unexpected consequences for not only your services, but the network connectivity in general. There is a general process (checklist) that you can following to help troubleshoot iptables in order to fix the problem.
Line 22: Line 41:
 
<table border="1" cellspacing="0" cellpadding="5" >
 
<table border="1" cellspacing="0" cellpadding="5" >
 
<tr><th>Step</th><td>'''Procedure'''</td><td>'''Explanation'''</td></tr>
 
<tr><th>Step</th><td>'''Procedure'''</td><td>'''Explanation'''</td></tr>
<tr>  <th>1</th><td>'''Test Network Connectivity'''</td><td>You can use the [http://zenit.senecac.on.ca/wiki/index.php/OPS335_Lab_1#Linux_Network_Connection_Configuration_Troubleshooting steps in lab 1] as a guide, but keep in mind the firewall may be blocking pings and DNS requests.</td></tr>
+
<tr>  <th>1</th><td>'''Test Network Connectivity'''</td><td>You can use the [https://wiki.cdot.senecacollege.ca/wiki/OPS335_Lab_1#Linux_Network_Connection_Configuration_Troubleshooting steps in lab 1] as a guide, but keep in mind the firewall may be blocking pings and DNS requests.</td></tr>
<tr>  <th>2</th><td>'''Verify Service is Running &amp; listening on the correct interfaces'''</td><td>You should learn to read the output of '''netstat -atnp''' and '''netstat -aunp''' to complement the '''systemctl status''' command.</td></tr>
+
<tr>  <th>2</th><td>'''Verify Service is Running &amp; listening on the correct interfaces'''</td><td>You should learn to read the output of '''ss -atnp''' and '''ss -aunp''' to complement the '''systemctl status''' command.</td></tr>
<tr>  <th>3</th><td>'''Verify Network Connectivity by Deleting iptables Rules'''</td><td>If you have no idea what's going on and need to confirm that you're still sane - clear all the iptables rules and check your configuration then. Keep in mind that the '''iptables -F''' command will delete all your rules but will not set the deafult policies to ACCEPT. This will tell you for sure whether your problem was (or was not) caused by iptables.<br><br>If you do this - have a ready way to restore the rules you just deleted. Restarting the iptables service is usually a good start and a '''shell script''' to add your custom rules is a reasonable next step.</td></tr>
+
<tr>  <th>3</th><td>'''List your iptables Rules &amp; Perform a "Walk-Thru"'''</td><td>For many decades, when troubleshooting programs that don't run properly, programmers will resort to reading their "source-code" line-by-line and pretend they are the computer to perform the operation. The programmer "walks-through" the code to force them to think like a computer in order to spot and fix subtle problems.<br><br>Therefore, you can follow a packet's path as you understand it should follow. Keep in mind [https://wiki.cdot.senecacollege.ca/wiki/OPS335_Lab_2#How_Firewalls_.28iptables.29_Relate_to_the_Labs_in_this_Course the diagram from the lecture last week]. What chain applies first on which machine? What's the first rule that matches the packet? What happens if no rules match the packet?<br><br>Don't forget that even if you're tracing the path of outgoing traffic - the INPUT chain on your machine still applies (for the response that comes back to your request).</td></tr>
<tr>  <th>4</th><td>'''List your iptables Rules &amp; Perform a "Walk-Thru"'''</td><td>For many decades, when troubleshooting programs that don't run properly, programmers will resort to reading their "source-code" line-by-line and pretend they are the computer to perform the operation. The programmer "walks-through" the code to force them to think like a computer in order to spot and fix subtle problems.<br><br>Therefore, you can follow a packet's path as you understand it should follow. Keep in mind [http://zenit.senecac.on.ca/wiki/index.php/OPS335_Lab_2#How_Firewalls_.28iptables.29_Relate_to_the_Labs_in_this_Course the diagram from the lecture last week]. What chain applies first on which machine? What's the first rule that matches the packet? What happens if no rules match the packet?<br><br>Don't forget that even if you're tracing the path of outgoing traffic - the INPUT chain on your mahchine still applies (for the response that comes back to your request).</td></tr>
+
<tr>  <th>4</th><td>'''Use the log target to list unexpected traffic'''</td><td>Add a final rule to your input chain to log all traffic.  Any traffic you are allowing will have already been accepted and will not reach this rule, so you will start a log of all the packets you are not allowing.  Observing the logs while you attempt to use the service that is not being allowed will show you the type of traffic you need to allow.</td></tr>
</table>
+
<tr>  <th>5</th><td>'''Verify Network Connectivity by Deleting iptables Rules'''</td><td>As a last resort, if you have no idea what's going on and need to confirm that you're still sane - clear all the iptables rules and check your configuration then. Keep in mind that the '''iptables -F''' command will delete all your rules but will not set the default policies to ACCEPT. This will tell you for sure whether your problem was (or was not) caused by iptables.<br><br>
 +
Stopping the iptable service with '''systemctl stop iptables''' will also clear all iptables rules.  Additionally, it will reset all policy to ACCEPT. <br><br>
 +
If you do this - have a ready way to restore the rules you just deleted. Restarting the iptables service is usually a good start and a '''shell script''' to add your custom rules is a reasonable next step.
 +
Don't forget to restart libvirtd service as well if this is being done on a kvm host</td></tr></table>
  
  
 
At this point, you should be able to understand any iptables rules you experience in this course, including the <u>default</u> ones in CentOS. If you see a iptables rule that you don't understand, you can <u>delete</u> it and see what happens. But if you simply delete this rule, take the time to figure out what that rule did and why you needed to delete it. It was likely there for a purpose (other than to drive you crazy).
 
At this point, you should be able to understand any iptables rules you experience in this course, including the <u>default</u> ones in CentOS. If you see a iptables rule that you don't understand, you can <u>delete</u> it and see what happens. But if you simply delete this rule, take the time to figure out what that rule did and why you needed to delete it. It was likely there for a purpose (other than to drive you crazy).
  
 +
'''Record the troubleshooting checklist in your OPS335 lab log-book'''
  
'''Record troubleshooting checklist for INVESTIGATION 1 in your OPS335 lab log-book'''
+
==INVESTIGATION 3: HANDS-ON IPTABLES TROUBLESHOOTING==
 
 
==INVESTIGATION 2: HANDS-ON IPTABLES TROUBLESHOOTING==
 
  
 
You will now get additional practice on troubleshooting iptables by downloading a running a shell script that will create iptables rules that will cause problems. You will then need to use tools and procedures (IPTABLES Troubleshooting Checklist) to determine the cause of the problem and fix that problem.
 
You will now get additional practice on troubleshooting iptables by downloading a running a shell script that will create iptables rules that will cause problems. You will then need to use tools and procedures (IPTABLES Troubleshooting Checklist) to determine the cause of the problem and fix that problem.
  
'''Perform the following steps:'''
+
'''Perform the following steps on your HOST:'''
  
#Download and run the following script: http://scs.senecacollege.ca/~andrew.smith/ops335/labcheck_network_backup.sh<br><br>This will display a menu of exercises. You can choose any of the items in order, but you should attempt all of them. The script will first reset the firewall settings to CentOS defaults and then make some modifications from those defaults.<br><br>
+
#Download and run the following script: http://scs.senecacollege.ca/~andrew.smith/ops335/lab_practice_iptables.sh<br><br>This will display a menu of exercises. You can choose any of the items in order, but you should attempt all of them. The script will first reset the firewall settings to CentOS defaults and then make some modifications from those defaults.<br><br>
#It is highly recommended to attempt to troubleshoot and fix the problem instead of reading the shell script and "cheating" to solve the problem. The point of the exercises is for you to find the problem using regular troubleshooting tools, not to reverse-engineer the shell script!
+
#Troubleshoot and fix the problem as you would on a real server. The point of the exercises is for you to find the problem using regular troubleshooting tools, not to reverse-engineer the shell script.<br><br>
 
#Finish the exercises, and record any information you feel you'll need to remember to solve problems like this in the future (e.g. in an assignment and/or in a practical test).
 
#Finish the exercises, and record any information you feel you'll need to remember to solve problems like this in the future (e.g. in an assignment and/or in a practical test).
  
  
'''Record steps, commands, and your observations in INVESTIGATION 2 in your OPS335 lab log-book'''
+
{{Admon/important |Backup your VMs!|You MUST perform a '''full backup''' of ALL of your VMs whenever you complete your '''OPS335 labs''' or when working on your '''OPS335 assignments'''. You should be using the dump or rsync command, and you should use  the Bash shell script that you were adviced to create in order to backup all of your VMs.}}
  
 +
 +
'''Record your observations in this section on your OPS335 lab log-book'''
  
 
== COMPLETING THE LAB ==
 
== COMPLETING THE LAB ==
[[Image:lab1_signoff.png|thumb|right|300px|Students should be prepared with '''all required commands (system information) displayed in a terminal (or multiple terminals) prior to calling the instructor for signoff'''.]]
+
In completing this lab you have gained further practice using iptables. Each of your machines should now be protected by a custom firewall that we will continue to build on throughout the course.  You have also gained experience troubleshooting iptables and determining what rules might need to be changed to allow desired traffic (or block undesired traffic).
'''Arrange evidence (command output) for each of these items on your screen, then ask your instructor to review them and sign off on the lab's completion:'''
+
 
 +
===Online Submission ===
 +
Follow the instructions for lab 2b on blackboard.
 +
<!--
 +
===Andrew's sections===
 +
 
 +
You may choose to:
 +
* Submit screenshots of your work on Blackboard, in which case you don't need to come to the lab.
 +
* Or come to the lab, show me your work, and talk to me about it. I want to hear what you've learned and answer any questions you have.
 +
 
 +
You'll get the same grade regardless of how you choose to submit your work.
  
::<span style="color:green;font-size:1.5em;">&#x2713;</span>x.
+
Expected results of this lab are:
  
 +
::<span style="color:green;font-size:1.5em;">&#x2713;</span>List iptables rules for ALL machines.
 +
::<span style="color:green;font-size:1.5em;">&#x2713;</span>Prove that you can ping and ssh from your host machines to all of your vms.
 +
::<span style="color:green;font-size:1.5em;">&#x2713;</span>Download and run https://ict.senecacollege.ca/~andrew.smith/ops335/labcheck2b.bash
 +
::<span style="color:green;font-size:1.5em;">&#x2713;</span>Be able to explain how you debug a connectivity problem caused by iptables.
 +
-->
  
 
==EXPLORATION QUESTIONS==
 
==EXPLORATION QUESTIONS==
  
#x
+
#List 3 separate techniques that you used to help troubleshoot to detect and fix iptables from running the shell script in the previous section.
 +
#Without looking at the table above, list tips for troubleshooting iptables.
 +
#After completing this lab, how does the above-mentioned shell script work to cause problems with iptables?

Latest revision as of 11:11, 29 January 2021


OBJECTIVE & PREPARATION

In Lab 2a, we set the firewall rules for your host machine. In this lab, we will create firewall rules for our virtual machines within our virtual private network. This lab will also apply best practices and troubleshooting techniques using iptables.

Online Resources

INVESTIGATION 1: CUSTOM IPTABLES RULES ON A VM

We will now set iptables rules for your vm1 machine.

Perform the following Steps:

  1. Start your host machine, and launch your vm1 machine.
  2. Login to your root account on your vm1 machine.
  3. Issue a command (like you did in lab2a) to copy your default iptables rules to the file pathname:
    /etc/sysconfig/iptables.original
  4. Issue an iptables command to set the policy to disable all forwarding traffic, and remove the rule that is rejecting it.
  5. Next, set the default policy to drop all inbound traffic, and remove the rule that is rejecting traffic.
  6. Issue an iptables command to list rules for verification.

    The remaining tasks will relate to that same inbound traffic chain:

  7. Issue an iptables command to delete the default ssh rule.
  8. Issue an iptables command to add a rule that allows ssh traffic (i.e. tcp packets with destination port 22) that originates from any machine within your virtual network.
  9. Issue an iptables command to delete the default icmp rule.
  10. Issue an iptables command to allow icmp traffic from addresses in your virtual network.
  11. Test that your machines can still use ping and ssh to communicate with each other.
  12. Save your rules in the location that iptables will automatically read from when it starts.
  13. Reboot your machine and check that the new rules are being applied. If they are not, resolve this issue before moving on.
  14. Now copy the file to your other VMs and make it apply to them when they boot as well.
  15. Reboot each machine and make sure this works before you move on.

Record your observations in this section on your OPS335 lab log-book

INVESTIGATION 2: IPTABLES TROUBLESHOOTING CHECKLIST

By now, you have probably discovered that a simple mistake in your iptables rules can have very serious and unexpected consequences for not only your services, but the network connectivity in general. There is a general process (checklist) that you can following to help troubleshoot iptables in order to fix the problem.


Refer to the following IPTABLES Troubleshooting Checklist:

StepProcedureExplanation
1Test Network ConnectivityYou can use the steps in lab 1 as a guide, but keep in mind the firewall may be blocking pings and DNS requests.
2Verify Service is Running & listening on the correct interfacesYou should learn to read the output of ss -atnp and ss -aunp to complement the systemctl status command.
3List your iptables Rules & Perform a "Walk-Thru"For many decades, when troubleshooting programs that don't run properly, programmers will resort to reading their "source-code" line-by-line and pretend they are the computer to perform the operation. The programmer "walks-through" the code to force them to think like a computer in order to spot and fix subtle problems.

Therefore, you can follow a packet's path as you understand it should follow. Keep in mind the diagram from the lecture last week. What chain applies first on which machine? What's the first rule that matches the packet? What happens if no rules match the packet?

Don't forget that even if you're tracing the path of outgoing traffic - the INPUT chain on your machine still applies (for the response that comes back to your request).
4Use the log target to list unexpected trafficAdd a final rule to your input chain to log all traffic. Any traffic you are allowing will have already been accepted and will not reach this rule, so you will start a log of all the packets you are not allowing. Observing the logs while you attempt to use the service that is not being allowed will show you the type of traffic you need to allow.
5Verify Network Connectivity by Deleting iptables RulesAs a last resort, if you have no idea what's going on and need to confirm that you're still sane - clear all the iptables rules and check your configuration then. Keep in mind that the iptables -F command will delete all your rules but will not set the default policies to ACCEPT. This will tell you for sure whether your problem was (or was not) caused by iptables.

Stopping the iptable service with systemctl stop iptables will also clear all iptables rules. Additionally, it will reset all policy to ACCEPT.

If you do this - have a ready way to restore the rules you just deleted. Restarting the iptables service is usually a good start and a shell script to add your custom rules is a reasonable next step.

Don't forget to restart libvirtd service as well if this is being done on a kvm host


At this point, you should be able to understand any iptables rules you experience in this course, including the default ones in CentOS. If you see a iptables rule that you don't understand, you can delete it and see what happens. But if you simply delete this rule, take the time to figure out what that rule did and why you needed to delete it. It was likely there for a purpose (other than to drive you crazy).

Record the troubleshooting checklist in your OPS335 lab log-book

INVESTIGATION 3: HANDS-ON IPTABLES TROUBLESHOOTING

You will now get additional practice on troubleshooting iptables by downloading a running a shell script that will create iptables rules that will cause problems. You will then need to use tools and procedures (IPTABLES Troubleshooting Checklist) to determine the cause of the problem and fix that problem.

Perform the following steps on your HOST:

  1. Download and run the following script: http://scs.senecacollege.ca/~andrew.smith/ops335/lab_practice_iptables.sh

    This will display a menu of exercises. You can choose any of the items in order, but you should attempt all of them. The script will first reset the firewall settings to CentOS defaults and then make some modifications from those defaults.

  2. Troubleshoot and fix the problem as you would on a real server. The point of the exercises is for you to find the problem using regular troubleshooting tools, not to reverse-engineer the shell script.

  3. Finish the exercises, and record any information you feel you'll need to remember to solve problems like this in the future (e.g. in an assignment and/or in a practical test).


Important.png
Backup your VMs!
You MUST perform a full backup of ALL of your VMs whenever you complete your OPS335 labs or when working on your OPS335 assignments. You should be using the dump or rsync command, and you should use the Bash shell script that you were adviced to create in order to backup all of your VMs.


Record your observations in this section on your OPS335 lab log-book

COMPLETING THE LAB

In completing this lab you have gained further practice using iptables. Each of your machines should now be protected by a custom firewall that we will continue to build on throughout the course. You have also gained experience troubleshooting iptables and determining what rules might need to be changed to allow desired traffic (or block undesired traffic).

Online Submission

Follow the instructions for lab 2b on blackboard.

EXPLORATION QUESTIONS

  1. List 3 separate techniques that you used to help troubleshoot to detect and fix iptables from running the shell script in the previous section.
  2. Without looking at the table above, list tips for troubleshooting iptables.
  3. After completing this lab, how does the above-mentioned shell script work to cause problems with iptables?