OPS235 Lab 7

LAB PREPARATION

Purpose / Objectives of Lab 7

Protecting a computer network from unauthorized access is one of the many day-to-day operations for a Linux system administrator and/or security specialist

Setting up a computer network is very important, but there are many other operations that occur on a daily basis that can include trouble-shooting, repairing network connection issues as well as maintaining network security. System administrators need to protect or "harden" their computer networks from "penetration" from unauthorized computer users. Hardening a computer system can range from running an IDS (Intrusion Detection System) to monitoring and flagging suspicious activity to implementing screen savers on workstations.

In this lab, you will learn how to use ssh, scp, sftp commands to securely access and share data between authorized personnel. In addition, you will learn various methods of running and configuring an ssh server which include: using Public Key Authentication, setting up an SSH tunnel in order to securely run graphical applications safely among computers in the network, disabling root login, and changing the default ssh communication port to mislead potential penetration testers (also known as "pen-testers" or "hackers").

Main Objectives

• To set up, configure Secure Shell Services (ssh/sshd)
• To use the ssh, scp, and sftp clients to access, copy, or transfer data among Linux servers in a secure manner
• Use ssh to tunnel X applications
• To tunnel network traffic via other communication ports
• To customize sshd to create a more private, secure system

Minimum Required Materials			Linux Command Reference
Solid State Drive	USB key (for backups)	Lab7 Log Book	Networking Utilities ssh ssh-keygen ssh-copy-id scp sftp netstat ifconfig ping arp iptables	Additional Utilities hostname restorecon chkconfig Configuration Files ssh_config sshd_config	SSH Reference A good ssh tutorial A good HOW-TO to make ssh more secure

INVESTIGATION 1: CONFIGURING AND ESTABLISHING AN SSH CONNECTION

So far, you have learned to use the ssh utility to establish a secure connection to a remote server to perform tasks, administer the server, etc. For these common operations, you have issued the ssh command, which is the client application for ssh. In order to connect to a remote server (like your VMs, Matrix, etc) they need to be running the SSH service. In this lab, you will learn how to run an SSH server in a VM, then confirm that you can connect into the server by using the ssh client application.

Part 1: Enabling the sshd service.

Perform the following steps:

1. Launch ALL of your VMs.
2. Switch to your c7host VM.
3. OpenSSH should have been installed by default. Let's confirm this by issuing the command:
   rpm -qa | grep ssh
4. You should see a number of packages installed including openssh-clients and openssh-server
5. openssh-server installs a service called sshd. Confirm that this service is running by issuing the command:
   systemctl status sshd

6. Now that you know the service is running, investigate what port number and protocol sshd uses by issuing the command:
   netstat -atunp | grep sshd
7. What protocol and port is the sshd process using? What is the state of the port? Why would you think that UDP ports don't have a state?
8. Reissue the netstat command without the -n option. What is the difference?
9. Make sure the sshd service is running on all 3 of your VM's
10. Remember that you can view the /etc/services file in case you need to reference a service with a corresponding port number.

Storing Fingerprints When a user connects to a host using ssh, the host sends a fingerprint or digital signature to the client to establish its identity. The first time a connection is established the identity must be stored for subsequent connections. The fingerprints are stored separately for each user in a file called ~/.ssh/known_hosts . From now on when you connect to that host the client will compare the received fingerprint against the list of known hosts before connecting. If the fingerprint does not match it could indicate somebody had setup a system to impersonate the computer you wish to connect to and you would receive a message like this

Part 2: Establishing a Safe SSH Connection: Public Key Authentication

As a system administrator, you have the ability to generate or create public and private keys to ensure safe and secure ssh connections. The system administer can generate these keys for the first time, or if the system administor suspects that a hacker has compromised or trying to penetrate the server, they can remove the existing keys and generate new keys.

A common type of attack, Arp Poisoning (Man in the Middle Attack), can be used to redirect packets to a third party while maintaining the illusion that the connection is secure. Therefore, understanding about the generation and management of public/private keys are important to the security of servers.

Perform the following steps:

1. Switch to your centos3 VM.
   
   

We can use the netstat utility as a trouble-shooting / confirmation tool to view the SSH service and determine which STATE the SSH service is performing: LISTENING, ESTABLISHED, CLOSED , or WAITING

2. Run the netstat -atunp command (pipe to "grep sshd") to check the state of a possible ssh connection. What is the state (i.e. LISTENING or ESTABLISHED)?
3. While in your centos3 VM, issue the following command to connect to the same VM via ssh: ssh ops235@centos3
4. Enter yes at the prompt, and enter your OPS235 password.
   The output should appear similar as what is shown below:
   
   

   The authenticity of host 'centos3 (192.168.235.13)' can't be established.
   

   RSA key fingerprint is 53:b4:ad:c8:51:17:99:4b:c9:08:ac:c1:b6:05:71:9b.
   

   Are you sure you want to continue connecting (yes/no)? yes
   

   Warning: Permanently added 'centos3' (RSA) to the list of known hosts.
   
   

5. Issue the following command to confirm that you connected to your centos3 VM: hostname

If you receive a message like the one displayed above, you should investigate why it is happening as it could indicate a serious security issue, or it could just mean that something on the host has changed(i.e. the OS was reinstalled)

7. Re-run that same netstat pipeline command. Any change to the connection status?
8. Log-out of your ssh connection by typing exit.
9. Run that same netstat command again. Wait a few minutes and then check again. Record your observations.

So far, we have learned to establish an ssh connection to another host using a password to establish your identity. But passwords are not the only or even the best way of authenticating your identity. We can also use Public/Private key encryption.

Public Key authentication is a method of establishing identity using a pair of encryption keys that are designed to work together. One key is known as your private key (which as the name suggests should remain private and protected) and the other is known as the public key (which as the name suggests can be freely distributed) The keys are designed to work together to encrypt data asymmetrically, that is to say that when we encrypt data with one of the keys it can only be decrypted with the other key from the pair.

While it doesn't mean the message is secure as anybody could decrypt it with the public key, it does establish my identity, if the host can successfully decrypt the message then it must have come from the one person in possession of the private key.

10. Switch to your centos2 VM.
11. Confirm you are in your centos2 VM by entering the command: hostname
12. Make certain that you are in your centos you are NOT logged in as root! (you have been warned!)
13. To generate a keypair (public/private keys), issue the following command: ssh-keygen
14. Press ENTER to accept the default, then enter a pass-phrase used to establish your identity, and re-enter the pass-phrase to verify.
    The output should appear similar as what is shown below:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/ops235/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase):
Enter passphrase again:
Your public key has been saved in /home/ops235/.ssh/id_rsa.pub.
The key fingerprint is:
ef:de:31:67:f7:15:a4:43:39:15:5d:78:1b:e8:97:74 ops235@centos3
The key's randomart image is:
+--[ RSA 2048]----+
|              .+=|
|             .+oE|
|            .+.o=|
|            ..++ |
|        S    o.. |
|         .    . .|
|          . o o o|
|         . . = .o|
|         .o .   .|
+-----------------+

15. After generating the keys it prompts you for the location to save the keys. The default is ~/.ssh Your private key will be saved as id_rsa and your public key will be saved as id_rsa.pub
16. You will then be prompted for a pass-phrase. The pass-phrase must be entered in order to use your private key. Pass-phrases are more secure than passwords and should be lengthy, hard to guess and easy to remember. For example one pass-phrase that meets this criteria might be "seneca students like fish at 4:00am". Avoid famous phrases such as "to be or not to be" as they are easy to guess. It is possible to leave the pass-phrase blank but this is dangerous. It means that if a hacker were able to get into your account they could then use your private key to access other systems you use.
    
    
17. Now issue the command ssh-copy-id -i ~/.ssh/id_rsa.pub ops235@centos3
18. When prompted for password, enter OPS235's root password
19. Try using ssh to now log into your centos3 VM from your centos2 VM. What happens? Were you required to use your pass-phrase?
20. Issue the hostname command to verify that you are successfully logged into your centos3 VM.
21. Make certain to logout of your centos3 system. Use the hostname command to verify you are back in your centos2 server.

Answer INVESTIGATION 1 observations / questions in your lab log book.

INVESTIGATION 2: USING SSH AND OTHER SECURE SHELL UTILITIES

Part 1: How do you use scp and sftp.

When the SSH service is running, users can then take advantage of secure shell tools (including the scp and sftp utilities).

The ssh client utility also contains many options to provide useful features or options when establishing secure connections between servers. One of these features is referred to as tunnelling - this term refers to running programs on remote servers (i.e. running the program on a remote server, yet interacting and viewing program on your local server. Since X-windows in Linux is a support layer to transmit graphical information efficiently between servers, ssh tunnelling becomes more useful and important to allow organizations to work efficiently and securely in a user-friendly environment.

Perform the following steps:

1. Switch to your centos1 VM.
2. Open a terminal, and remain logged in as a regular user.
3. Establish an sftp session with your centos3 VM by issuing the following command:
   sftp ops235@centos3
4. Type help to see the list of sftp commands at any time.
   
   Remote Server (eg. centos3) commands: pwd, ls, cd
   Local Machine (eg. centos1) commands: !pwd, !ls, lcd
   
   
5. The basic commands to upload and download files between servers are put and get respectively.
6. Try using sftp to transfer files back and forth between hosts.
7. Confirm that the files were transfer and then enter the command exit to quit your sftp session.

8. You can also use the scp command to copy files to and from remote hosts and even from one remote host to another.
9. Use scp to copy your services file to the centos3 host into the /tmp directory. (The path on a remote host follows the :) using the command:
   scp /etc/services ops235@centos3:/tmp
10. Here is a neat trick: You can run commands remotely using ssh by typing the command as an argument after the ssh command. Issue the following command in your centos2 VM:
    ssh ops235@centos3 ls /tmp
11. What happened when you issued that command? Where you able to successfully using scp to copy the /etc/services file to centos3's /tmp directory?
    This neat trick allow your lab checking scripts to run commands from remote VMs to verify your work!
12. Experiment with scp to copy a file from centos3 directly to centos1.

You can use an SSH tunnel with options to allow running of applications on remote Linux servers.

Part 2: Tunneling to Run Graphical Applications on Remote VMs .

You can also use ssh to tunnel window and bitmap information, allowing us to login to a remote desktop host and run a Xwindows application such as gedit or firefox and the application will run on the remote host but be displayed on the local host.

Perform the following steps:

1. For this section, you will be using your c7host and centos1 VMs.
2. Switch to your c7host VM, open a terminal and remain logged in as a regular user.
3. Issue the following command to connect to your centos1 VM:
   ssh -X -C yourUserID@centos1   (where 'yourUserID' is your user account name on centos1)
   (The -X option enables the forwarding of X window information, and the -C option enables compression for better performance).
   
   
4. Once the connection is properly established, run the command gedit
5. The gedit window will display on your c7host VM, but in reality, this application is running on your centos1 VM!
6. Enter some text and save your editing session.
7. Exit the gedit application.
8. In which VM was the file saved? What does that tell you about the use of tunneling for this section?
9. Experiment with running other GUI applications through ssh.

Answer INVESTIGATION 2 observations / questions in your lab log book.

INVESTIGATION 3: SECURING THE SSH CONNECTION

You can also use an ssh connection to tunnel other types of traffic. There could be different reasons for doing this. For example tunneling traffic for an unencrypted application/protocol through ssh can increase the security of that application (i.e. deceive potential hackers).

Alternatively you could use it to circumvent a firewall that is blocking traffic you wish to use but allows ssh traffic to pass through.

Part 1: Deceiving Penetration Testers (Hackers)

To help harden (i.e. protect a server from attack or "penetration"), system or security administrators have the ability to "trick" or "mislead" a potential hacker in order to prevent system penetration. Just like in the movies, "spying" is not just about collecting information about the adversary, but also to deceive the adversary into making incorrect decisions.

In this section, you will learn a common technique that organization use to help harden their servers: Use a combination of SSH server configuration and iptables rules in order to trick a hacker into thinking that the SSH port is not working, when in fact, it is running quietly (tunneled) via another port... Sneaky, but effective... >;)

Perform the following steps:

1. Make certain that your c7host and centos1 VMs are running.
2. Switch to your c7host VM. This will be your web-server.
3. Make sure that the Apache web server is installed by typing the command:
   rpm -q httpd
   (If this is not installed, make sure to install httpd).
4. Issue the following commands to start and enable the httpd service:
   systemctl start httpd
systemctl enable httpd
5. Issue a command to verify that the httpd service has been started and has been enabled.
6. Confirm that httpd is LISTENING to tcp (port 80) using the netstat command.
7. Create a small html document called /var/www/html/index.html that displays a short message. If you do not know how to use HTML markup language, just type a simple text message...
8. Restart your c7host VM.
9. On your c7host VM click on the following link to open a web-browser view your html or text message: http://localhost
   
   
10. IMPORTANT: Before proceeding, flush the iptables rules for INPUT, OUTPUT, and FORWARD chains for both your c7host and centos1 VMs.
    
    
11. Issue an iptables command to verify that both the the iptables rules have no rules for those chains for your c7host and centos1 VM's
12. Issue an iptables command to set the default firewall policy to REJECT incoming requests to http ( via TCP, port:80) by issuing the command:
    iptables -P INPUT -p tcp --sport 80 REJECT
13. NOTE: c7host: if http://localhost stops working locally, add the following iptables rule to centos1, as root
    iptables -I INPUT -i lo -j ACCEPT
14. On centos1 confirm that the httpd service is stopped so it cannot interfere with your observations.
15. On centos1 confirm that you can't connect by using firefox to centos1 http://centos1/
16. The next step is to establish a tunnel. When you establish a tunnel you make an ssh connection to a remote host and open a new port on the local host. That local host port is then connected to a port on the remote host through the established tunnel. When you send requests to the local port it is forwarded through the tunnel to the remote port.
17. In a terminal in your centos2 VM, make certain you are NOT logged in as root!
18. Establish a tunnel using a local port on centos2 of 20808, that connects to the remote port on c7host of 80, using the following command on centos1:
    ssh -L 20808:centos1:80 username@centos1
    
    Note:
    The -L option (which means Local port) takes one argument:
    <local-port>:<connect-to-host>:<connect-to-port>
    
    The command basically connects your local port of 20808 to the remote port of 80 on c7host.
    This means all requests to 20808 on the localhost (centos1) are actually tunneled through your ssh connection
    to port 22 on c7host and then delivered to port 80 on c7host, bypassing the firewall.
    
    
19. Once the tunnel is established use netstat to verify the port 20808 is listening on centos1
20. Now using the browser on centos1 connect to http://localhost:20808
21. You should see the index.html page on c7host.
22. Close the ssh connection and verify that the port 20808 is no longer listening.

Troubleshooting Tips for SSH Cannot connect via SSH? To fix issues with the ability to ssh, on both machines: Ensure ssh is running. Disable SELinux Flush iptables (iptables -F) For scp, use the access the option (eg. scp -P 2200 )

Part 2: Additional SSH Server Security Configuration

Anytime you configure your computer to allow logins from the network you are leaving yourself vulnerable to potential unauthorized access by so called "hackers". Running the sshd service is a fairly common practice but care must be taken to make things more difficult for those hackers that attempt to use "brute force" attacks to gain access to your system. Hackers use their knowledge of your system and many password guesses to gain access. They know which port is likely open to attack (TCP:22), the administrative account name (root), all they need to do is to "guess" the password.

Making your root password (and all other accounts!) both quite complex but easy to remember is not hard.

The Linux system administrator can also configure the SSH server to make the SSH server more secure. Examples include not permitting root login, and change the default port number for ssh.

Perform the following steps:

1. For this section, you will still be using your c7host and centos1 VMs.
2. Think of a good quality password and change your root passwords on all 3 VM's to be more secure. (It would be a good idea to do this for non-root accounts also)
3. The next change you can make is to prevent the root account from logging in to sshd altogether.
4. Change to your centos1 VM and open a terminal.
5. Edit the file /etc/ssh/sshd_config and look for the option PermitRootLogin. Un-comment the option (or add the option if it does not appear) and change the option value to "no".
   
   NOTE: Now any hacking attempt also has to guess an account name as well as the password. If you need to ssh with root access, ssh as a regular user and use su - to become root.
   
   
6. Even better, it is possible to restrict access to just specific users that require it.
7. Edit the file /etc/ssh/sshd_config and add a new option of "AllowUsers account" using your login account for account
8. In order for these changes to be effective, issue the following command to restart the sshd service:
   service sshd restart
9. Try sshing from your c7host VM to your centos1 VM. Where you successful? Would it work if you let "AllowUsers account" without a username, or a non-existent username? Do not do this for your machine!
10. Next change the default port number that sshd uses (TCP:22).
11. Edit the /etc/ssh/sshd_config file again, un-comment the port option and change the port number it uses from 22 to 2200.
12. Restart the service.
13. Confirm the new port is being used with the netstat command.
14. Before we can use this new port we must change our firewall to allow traffic through the new port number and block access to port 22 by issuing the command:
    iptables -I INPUT -p tcp -s0/0 --dport 2200 -j ACCEPT
15. Next, we will drop any incoming traffic to port 22 by issuing the command:
    iptables -I INPUT -p tcp -s0/0 --dport 22 -j DROP
16. We have now possibly mislead a potential "hacker" to the true port for our ssh server's communication channel (port).
17. Switch to your centos1 VM.
18. Issue the commmand: ssh username@centos2. What happens? What port do you think that command is using by default?
19. Now issue the following command to ssh via port "2200": ssh -p 2200 username@centos1. Where you able to connect?

17. Finally, as a system administrator, you should periodically monitor your system logs for unauthorized login attempts.
18. On CentOS systems the log file that is used is /var/log/secure
19. It also logs all uses of the su and sudo commands.
20. Attempt to connect to all of your VM's as root and other users using both public key and password authentication. Use some su and sudo commands also.
21. Inspect the log to see what kind of information is logged.

Answer INVESTIGATION 3 observations / questions in your lab log book.

LAB 7 SIGN-OFF (SHOW INSTRUCTOR)

Time for a new backup!
If you have successfully completed this lab, make a new backup of your virtual machines as well as your host machine.

Perform the Following Steps:

1. Make certain ALL of your VMs are running.
2. Switch to your c7host VM and su - into root.
3. Change to the /root/bin directory.
4. Issue the Linux command: wget http://matrix.senecac.on.ca/~murray.saul/ops235/lab7-check.bash
5. Give the lab7-check.bash file execute permissions (for the file owner).
6. Run the shell script and if any warnings, make fixes and re-run shell script until you receive "congratulations" message.
7. Arrange proof of the following on the screen:
   

   ✓ centos2 VM:

   • have logged into centos3 VM using public key authentication (with a pass-phrase)

   ✓ c7host Machine:

   • have tunneled Xwindows application from centos1 via ssh
   • have tunneled http through firewall using ssh (on web-browser
   • Run the lab7-check.bash script in front of your instructor (must have all  OK  messages)

   ✓ Lab7 log-book filled out.

Practice For Quizzes, Tests, Midterm & Final Exam

1. What port does sshd use by defaults?
2. What file is used to configure sshd?
3. What sftp commands are used to upload/download files?
4. What kind of files are stored in the "~/.ssh/" directory?
5. How do you determine whether the sshd service is running on your system or not?
6. What is the purpose of the ~/.ssh/known_hosts file?
7. What is the purpose of the ~/.ssh/authorized_keys file?
8. Which system log file records each use of the sudo command?
9. How do you stop the sshd service?
10. How do you tunnel XWindows applications?
11. What port is the default scp port?
12. What port(s) is/are used by httpd service?