OPS235 Lab 7

LAB PREPARATION

Purpose / Objectives of Lab 7

Protecting a computer network from unauthorized access is one of the many day-to-day operations for a Linux system administrator and/or security specialist

Setting up a computer network is very important, but there are many other operations that occur on a daily basis that can include trouble-shooting, repairing network connection issues as well as maintaining network security. System administrators need to protect or "harden" their computer networks from "penetration" from unauthorized computer users. Hardening a computer system can range from running an IDS (Intrusion Detection System) to monitoring and flagging suspicious activity to implementing screen savers on workstations.

In this lab, you will learn how to use ssh, scp, sftp commands to securely access and share data between authorized personnel. In addition, you will learn various methods of running and configuring an ssh server which include: using Public Key Authentication, setting up an SSH tunnel in order to securely run graphical applications safely among computers in the network, disabling root login, and changing the default ssh communication port to mislead potential penetration testers (also known as "pen-testers" or "hackers").

Main Objectives

• To set up, configure Secure Shell Services (ssh/sshd)
• To use the ssh, scp, and sftp clients to access, copy, or transfer data among Linux servers in a secure manner
• Use ssh to tunnel X applications
• To tunnel network traffic via other communication ports
• To customize sshd to create a more private, secure system

Minimum Required Materials			Linux Command Reference
Solid State Drive	USB key (for backups)	Lab7 Log Book	Networking Utilities ssh ssh-keygen ssh-copy-id scp sftp netstat ifconfig ping arp iptables	Additional Utilities hostname restorecon chkconfig Configuration Files ssh_config sshd_config	SSH Reference A good ssh tutorial A good HOW-TO to make ssh more secure

INVESTIGATION 1: CONFIGURING AND ESTABLISHING AN SSH CONNECTION

So far, you have learned to use the ssh utility to establish a secure connection to a remote server to perform tasks, administer the server, etc. For these common operations, you have issued the ssh command, which is the client application for ssh. In order to connect to a remote server (like your VMs, Matrix, etc) they need to be running the SSH service. In this lab, you will learn how to run an SSH server in a VM, then confirm that you can connect into the server by using the ssh client application.

We can use the netstat utility as a trouble-shooting / confirmation tool to view the SSH service and determine which STATE the SSH service is performing:

• LISTENING (waiting for a ssh connection attempt)
• ESTABLISHED (connection established)
• CLOSED (connection is closed)
• WAITING (time between connection closed, and changing to LISTENING mode)

Part 1: Enabling the sshd service.

Perform the following steps:

1. Use your c7host machine to complete this section
2. OpenSSH should have been installed by default. Let's confirm this by issuing the command:
   rpm -qa | grep ssh
3. You should see a number of packages installed including openssh-clients and openssh-server
4. openssh-server installs a service called sshd. Confirm that this service is running by issuing the command:
   service sshd status
5. Also, you should confirm that the sshd service is enabled (to start automatically upon boot-up). To do this issuing the following command:
   systemctl list-unit-files | grep sshd
   
   NOTE: the netstat is a very useful command for anything to do with networking. Read the man pages and make sure you understand the netstat command's output.
   
   
6. Now that you know the service is running, investigate what port number and protocol sshd uses by issuing the command:
   netstat -atunp
7. What protocol and port is the sshd process using? What is the state of the port? Why would you think that UDP ports don't have a state?
8. Reissue the netstat command without the -n option. What is the difference?
9. How is the file /etc/services related to the difference?
10. Make sure the sshd service is running on all 3 of your VM's

Storing Fingerprints When a user connects to a host using ssh, the host sends a fingerprint or digital signature to the client to establish its identity. The first time a connection is established the identity must be stored for subsequent connections. The fingerprints are stored separately for each user in a file called ~/.ssh/known_hosts . From now on when you connect to that host the client will compare the received fingerprint against the list of known hosts before connecting. If the fingerprint does not match it could indicate somebody had setup a system to impersonate the computer you wish to connect to and you would receive a message like this

Part 2: Establishing a Safe SSH Connection: Public Key Authentication

As a system administrator, you have the ability to generate or create public and private keys to ensure safe and secure ssh connections. The system administer can generate these keys for the first time, of if they suspect that a hacker has compromised the server, can remove the existing keys and generate new keys. A common type of attack, Arp Poisoning (Man in the Middle Attack), can be used to redirect packets to a third party while maintaining the illusion that the connection is secure. Therefore, understanding about the generation and management of public/private keys are important to the security of servers.

Perform the following steps:

1. Use your centos2 VM to complete this section.
2. Open a terminal and run the netstat command (pipe to "grep sshd") to check the state of the connection. What is the state (i.e. LISTENING or ESTABLISHED)?
3. Open another terminal and establish an ssh connection to your centos3 VM using the command:
   ssh ops235@centos3
   (Where 'ops235' is the account on centos3 and 'centos3' is the hostname of the centos3 VM.)
   
   
4. You should receive a message similar to the following:
   
   

   The authenticity of host 'centos3 (192.168.235.13)' can't be established.
   

   RSA key fingerprint is 53:b4:ad:c8:51:17:99:4b:c9:08:ac:c1:b6:05:71:9b.
   

   Are you sure you want to continue connecting (yes/no)? yes
   

   Warning: Permanently added 'centos3' (RSA) to the list of known hosts.
   
   

5. Answer yes to add to the list of known hosts.
6. Issue the following command to confirm that you connected to your centos3 VM: hostname

If you receive a message like the one displayed above, you should investigate why it is happening as it could indicate a serious security issue, or it could just mean that something on the host has changed(i.e. the OS was reinstalled)

7. Switch back to the original terminal and re-run the netstat pipeline command again. Any change to the connection status?
8. Return to the second terminal, and logout of your ssh connection by typing exit.
9. Run the netstat command in the original terminal and check the state of the connection after logging out. Wait a few minutes and then check again. Record your observations.
10. Make certain to exit all connections, and remain in your centos2 VM. When using ssh to connect to other servers, it is very easy to forget which server you are currently using. Verify that you are in your centos2 VM by entering the command: hostname
11. Use the Internet to search for TCP 3 way handshake to see how TCP connections are established and closed.
    
    

So far, we have learned to establish an ssh connection to another host using a password to establish your identity. But passwords are not the only or even the best way of authenticating your identity. We can also use Public/Private key encryption.

Public Key authentication is a method of establishing identity using a pair of encryption keys that are designed to work together. One key is known as your private key (which as the name suggests should remain private and protected) and the other is known as the public key (which as the name suggests can be freely distributed) The keys are designed to work together to encrypt data asymmetrically, that is to say that when we encrypt data with one of the keys it can only be decrypted with the other key from the pair.

While it doesn't mean the message is secure as anybody could decrypt it with the public key, it does establish my identity, if the host can successfully decrypt the message then it must have come from the one person in possession of the private key.

12. We are now going to generate a new set of public/private keys.
<li>Students run into a lot of trouble when using ssh and generating key-pairs by performing these operations as root user by Mistake! Make certain that you are NOT logged in as root! (you have been warned!)
13. To generate a keypair (public/private keys), issue the following command:
    ssh-keygen
14. That should generate output similar to the following:
    
    

Generating public/private rsa key pair.

Enter file in which to save the key (/home/user1/.ssh/id_rsa):

Enter passphrase (empty for no passphrase):

Enter same passphrase again:

Your identification has been saved in /home/user1/.ssh/id_rsa.

Your public key has been saved in /home/user1/.ssh/id_rsa.pub.

The key fingerprint is:

93:58:20:56:72:d7:bd:14:86:9f:42:aa:82:3d:f8:e5 user1@centos2

16. After generating the keys it prompts you for the location to save the keys. The default is ~/.ssh Your private key will be saved as id_rsa and your public key will be saved as id_rsa.pub
17. You will then be prompted for a pass-phrase. The pass-phrase must be entered in order to use your private key. Pass-phrases are more secure than passwords and should be lengthy, hard to guess and easy to remember. For example one pass-phrase that meets this criteria might be "seneca students like fish at 4:00am". Avoid famous phrases such as "to be or not to be" as they are easy to guess. It is possible to leave the pass-phrase blank but this is dangerous. It means that if a hacker were able to get into your account they could then use your private key to access other systems you use.
    
    
18. Now issue the command ssh-copy-id -i ~/.ssh/id_rsa.pub ops235@centos3
19. Try using ssh to now log into your centos3 VM from your centos2 VM. What happens? Were you required to use your pass-phrase? Issue the hostname command to verify that you are successfully logged into your centos3 VM.
20. Make certain to logout of your centos3 system. Use the hostname command to verify you are back in your centos2 server.

Answer INVESTIGATION 1 observations / questions in your lab log book.

INVESTIGATION 2: USING SSH AND OTHER SECURE SHELL UTILITIES

Part 1: How do you use scp and sftp.

When you have created an SSH server, then users can take advantage of secure shell tools (including the scp and sftp utilities)

The ssh client utility also contains many options to provide useful features or options when establishing secure connections between servers. One of these features is referred to as tunnelling - this term refers to running programs on remote servers (i.e. running the program on a remote server, yet interacting and viewing program on your local server. Since X-windows in Linux is a support layer to transmit graphical information efficiently between servers, ssh tunnelling becomes more useful and important to allow organizations to work efficiently and securely in a user-friendly environment.

Perform the following steps:

1. Remain in your centos2 VM for this section.
2. To connect to a remote host type the command:
   sftp ops235@centos3
3. This will establish an interactive session after authentication.
4. Type help to see the list of sftp commands at any time.
5. The 2 main commands are put to copy a file from the local host to the remote host (upload) and get to copy a file from the remote host to the local host (download).
6. Try using sftp to transfer files back and forth between hosts.
7. Confirm that the files were transfer and then enter the command exit to quit your sftp session.

8. You can also use the scp command to copy files to and from remote hosts and even from one remote host to another.
9. Use scp to copy your services file to the centos3 host into the /tmp directory. (The path on a remote host follows the :) using the command:
   scp /etc/services ops235@centos3:/tmp
10. Here is a neat trick: You can run commands remotely using ssh by typing the command as an argument after the ssh command. Issue the following command in your centos2 VM:
    ssh ops235@centos3 ls /tmp
11. What happened when you issued that command? Where you able to successfully using scp to copy the /etc/services file to centos3's /tmp directory?
12. Experiment with scp to copy a file from centos3 directly to centos1.

You can use an SSH tunnel with options to allow running of applications on remote Linux servers.

Part 2: How do you use ssh to tunnel X.

You can also use ssh to tunnel window and bitmap information, allowing us to login to a remote desktop host and run a Xwindows application such as gedit or firefox and the application will run on the remote host but be displayed on the local host.

Perform the following steps:

1. For this section, you will be using your centos1 and centos2 VMs.
2. From your centos2 VM issue the ssh command to connect to your centos1 VM using the following command:
   ssh -X -C username@centos1   (where 'username' is your learn account on centos1)
   (The -X option enables the forwarding of X window information, and the -C option enables compression for better performance).
   
   
3. Once the connection is properly established, run the gedit application. (Gnome Text Editor)
4. The gedit window will display on your centos2 VM, but in reality, this application is running on your centos1 VM!
5. Enter some text and save a file with gedit.
6. Exit the gedit application.
7. In which VM was the file saved?
8. Experiment with running other GUI applications through ssh.

centos1 VM iptables and ssh service
You may need to adjust the firewall on your centos1 host to complete this section, and verify that the sshd service is running on that VM.

Answer INVESTIGATION 2 observations / questions in your lab log book.

INVESTIGATION 3: SECURING THE SSH CONNECTION

You can also use an ssh connection to tunnel other types of traffic. There could be different reasons for doing this. For example tunneling traffic for an unencrypted application/protocol through ssh can increase the security of that application (i.e. deceive potential hackers).

Alternatively you could use it to circumvent a firewall that is blocking traffic you wish to use but allows ssh traffic to pass through.

Part 1: How do you use ssh to tunnel other traffic.

To help harden (protect a server from attack or "penetration"), system or security administrators have the ability to "trick" or "mislead" a potential hacker in order to prevent system penetration.

In this part, you will learn to use a combination of SSH server configuration and iptables rules to redirect the SSH port to allow secure data traffic via another port (as opposed to the default port: 22), and use iptables to reject (better log) incoming tcp traffic via the default port.

What is the result of this? Simple. Permit the SSH service for the organization, and yet trick and confound the potential hacker into thinking that ssh traffic is used on a port that is no longer available (but they may not know this!)

Sneaky! >;)

Perform the following steps:

1. For this section, you will still be using your centos1 and centos2 VMs.
2. We will be bypassing a firewall that blocks http traffic.
3. In this investigation, centos1 will be your http server and centos2 will be your client.
4. Use the hostname command to verify that you are in your centos1 VM (as opposed to another VM by mistake via ssh!)
5. On the HTTP server, make sure that the Apache web server is installed by typing the command:
   rpm -q httpd
6. If this is not installed, make sure to install httpd.
7. When httpd is properly installed, check the configuration of the service to see if it is automatically started at any run-levels by using the command:
   systemctl list-unit-files | grep httpd
8. If it has not been started automatically start the service using the service command to start the httpd service.
9. It is also a good idea to enable the httpd service so it starts automatically from boot-up. To do this, issue the command:
   chkconfig httpd on
10. Issue a previous command to verify that the httpd service is enabled.
11. Confirm that httpd is listening to TCP/80 using the netstat command.
12. Create a small html document called /var/www/html/index.html that displays a short message. If you do not know how to use HTML markup language, just type a simple text message...
13. Restart your centos1 VM.
14. On your centos1 VM (i.e. the http server), confirm everything is working locally by using a browser to connect to http://localhost
15. Set the default firewall configuration on centos1 to REJECT incoming requests to http (TCP/80)
16. NOTE: centos1: if http://localhost stops working locally, add the following iptables rule to centos1, as root
    iptables -I INPUT -i lo -j ACCEPT
17. On centos2 confirm that the httpd service is stopped so it cannot interfere with your observations.
18. On centos2 confirm that you can't connect by using firefox to centos1 http://centos1/
19. The next step is to establish a tunnel. When you establish a tunnel you make an ssh connection to a remote host and open a new port on the local host. That local host port is then connected to a port on the remote host through the established tunnel. When you send requests to the local port it is forwarded through the tunnel to the remote port.
20. In a terminal in your centos2 VM, make certain you are NOT logged in as root!
21. Establish a tunnel using a local port on centos2 of 20808, that connects to the remote port on centos1 of 80, using the following command on centos2:
    ssh -L 20808:centos1:80 username@centos1
    
    Note:
    The -L option (which means Local port) takes one argument:
    <local-port>:<connect-to-host>:<connect-to-port>
    
    The command basically connects your local port of 20808 to the remote port of 80 on centos1.
    This means all requests to 20808 on the localhost (centos2) are actually tunnelled through your ssh connection
    to port 22 on centos1 and then delivered to port 80 on centos1, bypassing the firewall.
    
    
22. Once the tunnel is established use netstat to verify the port 20808 is listening on centos2
23. Now using the browser on centos2 connect to http://localhost:20808
24. You should see the index.html page on centos1.
25. Close the ssh connection and verify that the port 20808 is no longer listening.

Troubleshooting Tips for SSH Cannot connect via SSH? To fix issues with the ability to ssh, on both machines: Ensure ssh is running. Disable SELinux Flush iptables (iptables -F) For scp, use the access the option (eg. scp -P 2200 )

Part 2: Making sshd More Secure

Anytime you configure your computer to allow logins from the network you are leaving yourself vulnerable to potential unauthorized access by so called "hackers". Running the sshd service is a fairly common practice but care must be taken to make things more difficult for those hackers that attempt to use "brute force" attacks to gain access to your system. Hackers use their knowledge of your system and many password guesses to gain access. They know which port is likely open to attack (TCP:22), the administrative account name (root), all they need to do is to "guess" the password.

Making your root password (and all other accounts!) both quite complex but easy to remember is not hard.

The Linux system administrator can also configure the SSH server to make the SSH server more secure. Examples include not permitting root login, and change the default port number for ssh.

Perform the following steps:

1. For this section, you will still be using your centos1 and centos2 VMs.
2. Think of a good quality password and change your root passwords on all 3 VM's to be more secure. (It would be a good idea to do this for non-root accounts also)
3. The next change you can make is to prevent the root account from logging in to sshd altogether.
4. Change to your centos2 VM and open a terminal.
5. Edit the file /etc/ssh/sshd_config and look for the option PermitRootLogin. Un-comment the option (or add the option if it does not appear) and change the option value to "no".
   
   NOTE: Now any hacking attempt also has to guess an account name as well as the password. If you need to ssh with root access, ssh as a regular user and use su - to become root.
   
   
6. Even better, it is possible to restrict access to just specific users that require it.
7. Edit the file /etc/ssh/sshd_config and add a new option of "AllowUsers account" using your login account for account
8. In order for these changes to be effective, issue the following command to restart the sshd service:
   service sshd restart
9. Try sshing from your centos1 VM to your centos2 VM. Where you successful? Would it work if you let "AllowUsers account" without a username, or a non-existent username? Do not do this for your machine!
10. Next change the default port number that sshd uses (TCP:22).
11. Edit the /etc/ssh/sshd_config file again, un-comment the port option and change the port number it uses from 22 to 2200.
12. Restart the service.
13. Confirm the new port is being used with the netstat command.
14. Before we can use this new port we must change our firewall to allow traffic through the new port number and block access to port 22 by issuing the command:
    iptables -I INPUT -p tcp -s0/0 --dport 2200 -j ACCEPT
15. Next, we will drop any incoming traffic to port 22 by issuing the command:
    iptables -I INPUT -p tcp -s0/0 --dport 22 -j DROP
16. We have now possibly mislead a potential "hacker" to the true port for our ssh server's communication channel (port).
17. Switch to your centos1 VM.
18. Issue the commmand: ssh username@centos2. What happens? What port do you think that command is using by default?
19. Now issue the following command to ssh via port "2200": ssh -p 2200 username@centos2. Where you able to connect?

17. Finally, as a system administrator, you should periodically monitor your system logs for unauthorized login attempts.
18. On CentOS systems the log file that is used is /var/log/secure
19. It also logs all uses of the su and sudo commands.
20. Attempt to connect to all of your VM's as root and other users using both public key and password authentication. Use some su and sudo commands also.
21. Inspect the log to see what kind of information is logged.

Answer INVESTIGATION 3 observations / questions in your lab log book.

LAB 7 SIGN-OFF (SHOW INSTRUCTOR)

Time for a new backup!
If you have successfully completed this lab, make a new backup of your virtual machines as well as your host machine.

Perform the Following Steps:

1. Switch to your centos2 VM.
2. Issue the Linux command: wget http://matrix.senecac.on.ca/~murray.saul/ops235/lab7-check.bash
3. Give the lab7-check.bash file execute permissions (for the file owner).
4. Run the shell script and if any warnings, make fixes and re-run shell script until you receive "congratulations" message.
5. Arrange proof of the following on the screen:
   

   ✓ centos2 VM:

   • have tunneled Xwindows application from centos1 via ssh
   • have tunneled http through firewall using ssh (on web-browser
   • have secured ssh against root access
   • have logged into centos3 VM using public key authentication
   • Output from running the lab7-check.bash script with all OK messages

   ✓ c7host Machine:

   • Confirmation that sshd is running on host machine

   ✓ Lab7 log-book filled out.

Practice For Quizzes, Tests, Midterm & Final Exam

1. What port does sshd use by defaults?
2. What file is used to configure sshd?
3. What sftp commands are used to upload/download files?
4. What kind of files are stored in the "~/.ssh/" directory?
5. How do you determine whether the sshd service is running on your system or not?
6. What is the purpose of the ~/.ssh/known_hosts file?
7. What is the purpose of the ~/.ssh/authorized_keys file?
8. Which system log file records each use of the sudo command?
9. How do you stop the sshd service?
10. How do you tunnel XWindows applications?
11. What port is the default scp port?
12. What port(s) is/are used by httpd service?