View source for SEC520/labs/Lab 2

<h1> <span class="mw-headline">Information Gathering</span></h1>
<h2> <span class="mw-headline">Introduction</span></h2>
<dl><dd><ul><li>This lab teaches various methods of <b>gathering information</b> from a <b>targeted computer system</b>. Normally, an individual or a company can be hired to perform <b>Penetration Testing</b> in order to detect weaknesses in an organization's computer system. The first phase (called the <b>"reconnaissance phase"</b>
 is considered to be a "harmless activity", where a person can simply gather information to be used later in other aspects of penetration 
testing (network <i>scanning</i> and <i>enumeration</i>).
</li></ul>
</dd></dl>
<dl><dd><ul><li>Students will first learn how to gather various 
documents / information via a web-browser in order to obtain information
 regarding the <b>structure</b>, <b>relationships</b> and <b>policies</b>
 of a target company, as well as partners or servers that are associated
 with that target company (with emphasis on IP addresses). Once the 
relevant information has been collected, the student will then utilize 
open-source applications in order to perform <b>link analysis</b> to make connections between various IP addresses. 
</li></ul>
</dd></dl>
<dl><dd><ul><li>Students will then learn how to use Interent-Based tools and technolgies to <b>mine data</b>
 that pertains more to the internal structure of the targeted 
organization's server(s), as well as it's specific IP Address ranges 
(subnets).
</li></ul>
</dd></dl>
<dl><dd><ul><li>Students will also learn how to use tools to gather information of the <b>users</b> of a targeted server, as well as  <b>verifying</b> the targetted IP Addresses immediately prior to the <i>scanning</i> and <i>enumeration</i> phases.
</li></ul></dd></dl>


<br><br>
<h2> <span class="mw-headline">Objectives</span></h2>
<ol><li>Use the <b>search engine website (google.ca)</b> to obtain computer system information (including IP address).
</li><li>Use various open-source applications to perform IP address associations with IP address (<b>Link Analysis</b>).
</li><li>Understand the basic concepts of <b>"footprintng"</b> a targeted server with respect to the following open-source technologies:<ul><li>DNS Lookup</li><li>WHOIS (Website Service)</li><li>Domain Name Expansion</li><li>HOST</li><li>SMTP</li></ul>
</li><li>Using open-source tools in order to focus on <i>technical aspects</i> of the server, in order to be more successful in the <i>scanning</i> and <i>enumeration</i> phase.
</li><li>Use tools to gather user information such as e-mail addresses or other information via social networking sites.
</li><li>Verify (confirm and narrow-down) valid IP Addresses (and 
ranges) to help reduce the time during the scanning and enumeration 
phases.
</li><li>Practice skills learned in this lab to gather information of an educational penetration-testing server at Seneca College (<b>tank.senecac.on.ca</b>).
</li></ol>




<p><br>
</p>
<h2> <span class="mw-headline">Required Materials (Bring to All Labs)</span></h2>
<ul>
<li> <b>SATA Hard Disk</b> (in removable disk tray).
</li><li> <b>Lab Logbook (Lab2 Reference Sheet)</b> (to make notes and observations).
</li></ul>
<p><br>
</p>
<h2> <span class="mw-headline">Prerequisites</span></h2>
<ul><li> [https://wiki.cdot.senecacollege.ca/wiki/SEC520/labs/Lab_1 SEC520 Lab 1]
</li></ul>
<p><br>
</p>
<h2> <span class="mw-headline">Online Tools and References</span></h2>


<table cellpadding="12">
 <tr valign="top">
   <td><b>Information Gathering</b></td>
   <td><b>Foot-printing</b></td>
   <td><b>User Information</b></td>
   <td><b>Verification</b></td>
   
 </tr>
 <tr valign="top">
   <td>
<ul>
 <li>[http://www.google.ca/ Google Search Engine] (site, filetype, link)</li>
 <li>[http://news.netcraft.com/ Netcraft]</li>
 <li>[http://github.com/sensepost/BiLE-suite BiLE Utilities]</li>
</ul>
   </td>
   <td>
<ul>
 <li>WHOIS Online Proxies:<br>
     ([http://whois.domaintools.com/ whois.domaintools.com])
 </li>
   </ul></td>
   <td>
   <ul>
 <li>[http://www.ehacking.net/2011/08/theharvester-backtrack-5-information.html theHarvester.py]</li>
 <li>[http://www.ehacking.net/2011/12/metagoofil-backtrack-5-tutorial.html Metagoofil.py]</li>
   </ul>
   </td>
   <td>
<ul>
 <li>[http://www.bing.com/ www.bing.com]</li>
 <li>[http://www.computerhope.com/unix/unslooku.htm nslookup]</li>
 <li>[http://www.ehacking.net/2011/02/dnsmap-dns-network-mapper.html dnsmap]</li>
</ul>
   </td>
   <td>
     <ul>
     </ul>
   </td>
  </tr>
</table>
<p><br>
</p>
<h2> <span class="mw-headline">Course Notes</span></h2>
<ul>
 <li>[http://cs.senecac.on.ca/%7Efac/sec520/slides/sec520_w1_l2.odp odp]| [http://cs.senecac.on.ca/%7Efac/sec520/slides/sec520_w1_l2.pdf pdf]| [http://cs.senecac.on.ca/%7Efac/sec520/slides/sec520_w1_l2.ppt ppt](Slides: Reconnaissance)</li>
</ul>

<p><br>
</p>
<h1> <span class="mw-headline">Performing Lab 2</span></h1>
<h2> <span class="mw-headline">Task #1: Using Search Engines to Obtain Target Server Information </span></h2>
<p>With the "information gathering" phase of penetration testing, it is 
recommended to obtain as much data regarding a targeted organization. 
This would include viewing the website, noting contacts, following-up 
information from social media sites (eg. facebook, etc). In addition to 
the above-mentioned techniques, there are other techniques and tools to 
help gather useful server information of a targeted organization.</p>
{{Admon/tip|sensepost.com|This is a website that is dedicated to internet security, and provides a platform to help gather information regarding a server. In fact, 
examples from the textbook: <b>Penetration Tester's Open Source Toolkit</b> use examples from this website. We will be using this site for the majority of lab2...}}
<br>
INSTRUCTIONS:
<ol>
 <li>Boot your  <b>Kali Linux</b> (host) system, and start a graphical session.</li>
 <li>Open a web-browser and go to the Google website ( <b>http://www.google.ca/</b> )</li>
 <li>Type in the following URL in the Google search box: <b>sensepost.com</b></li>
 <li>Note the type of links that are associated with this type of 
search (i.e total number of links at the top of the search results), and record the total number of links for this type of search in 
your lab logbook.</li>
 <li>Now, enter the following directive in the Google search box: <b>site:sensepost.com</b><br>{{Admon/important|Enter Site Directive in Google Search Textbox|Don't enter the "site" directive in the URL textbox at the top of the web-browser - enter this directive in the Google SEARCH text; otherwise, the directive will not work. Also make certain remain in the google web-page when performing this operation...}}</li>
 <li>You should notice a change in the display of links. How does this 
search method differ from the previous search method using only the 
text: "sensepost.com"? Record your findings (including the new total 
number of links) in your lab log-book.</li>
 <li>We will now be narrowing our search in the <b>sensepost.com</b> website for specific types of files for <b>pdf</b> with the filename keyword <b>hacking</b><br>Enter the following directive in the Google search box: <b>site:sensepost.com filetype:pdf  hacking</b> </li>
 <li>What are the total amount of links? Are all of the links contained in sensepost.com? Record your findings in your log lab-book.</li>
 <li>Issue directives to search for links in the <i>sensepost.com</i> website that contains MS Word documents (<b>doc</b>), and MS Word PowerPoint Presentations (<b>ppt</b>) that contain the pattern hacking. Record these findings in your lab log-book.</li>
 <li>Finally, the <b>link</b> directive is used to display links that 
are associated with a target website. In order to display all websites 
that link to the <i>sensepost.com</i> website, issue the following directive in the Google searchbox: <b>link:sensepost.com</b></li>
 <li>Record the total number of links in your lab log-book. Are there any other links outside the sensepost.com domain that are associated? How do you think this is useful in terms of penetration testing?</li>
 <li>How do you think that you could use this information that you have 
just collected during this lab for penetration testing? (Record your 
answer in your lab log-book)</li>
 <li>Repeat the information-gathering process for the following URL: <b>linux.senecac.on.ca</b> for practice.<br>{{Admon/tip|Gathering Information in your Own Server at Home|Just for Interest, it is not that difficult to obtain SOME information regarding your own computer system at home. First, determine your IP address by using the <b>ifconfig</b> command for Linux, or the <b>ipconfig</b> command in windows. One very quick way to determine your IP Address is to simply type <b>IP Address</b> in the URL Window of your web-browser. Knowing your own IP Address at home is useful during the <b>link analysis</b> and <b>domain name expansion</b> steps in the next task...}}</li>
 <li>Proceed to Task #2<br><br></li>
 </ol>
<p><b>Answer the Task #1 observations / questions in your lab log book.</b>
</p>
<br><br>
<h2> <span class="mw-headline">Task #2: Server Detection, Link Analysis &amp; Domain Name Expansion</span></h2>

<p>In this section, we will use the site information (obtained from <i>Task #1</i>)
 to gain more detailed information regarding the targeted 
organization's server (eg. IP address, Type of operating system, History
 of uptimes, name server information , Related IP addresses of other 
servers).</p><br>

INSTRUCTIONS:
<ol>
 <li>Assuming that your web-browser is still running, click on the following link (which should open in another browser window): <b>[http://www.netcraft.com http://www.netcraft.com]</b>.<br /><b>NOTE:</b>  Do not worry if you are redirected to another URL (eg. news.netcraft.com) - it will provides the same information we require.<br /><br /></li>
 <li>Let's find out additional information regarding the <b>sensepost.com</b> website. In the <b>What's that site running?</b> box, enter the following:<br><b>sensepost.com</b></li>
 <li>Record the following server information for "sensepost.com" (and record in your lab log-book):<ul><li>IP Address</li><li>Type of Operating System</li><li>Name Server</li><li>Country Origin</li><li>Date First Noticed (Tracked)</li><li>Frequency of Uptimes</li></ul></li>
<br />
<li>The next step in the reconnassaince phase involves <b>Linux Analysis</b>, which will list and 
categorize relationships between other websites, and the <i>"target"</i> website 
called <b>"sensepost.com"</b>. You will be downloading, installing and running 
serveral open-source tools (a series of packages packaged as <b>BiLE</b> (which stands for: <i>"Bi-directional Link Extraction"</i> tools) to asssist in obtaining this information.<br><br>
 {{Admon/important|Installing Dependencies for BiLE.pl, BiLE-Weigh.pl|You may need to download the <b>BiLE</b> Utilities, consisting of useful Perl Scripts. Your Kali Linux distribution most likely comes with Perl already loaded. On the other hand, prior to running these Perl Scripts, you may be required to first 
install the application called <b>HTTrack</b>. You can do this by installing "httrack" via "apt-get" or use a graphical application (such as <b>Synaptic Package Manager</b>)}}
<br /></li>
 <li>Issue the command: <b>which httrack</b> to confirm that this dependent application has been installed (refer to warning message above).</li>
 <li>In a web-browser, go to the following website (which will open in a separate browser window): <b>[http://github.com/sensepost/BiLE-suite http://github.com/sensepost/BiLE-suite]</b></li>
 <li>Download the <i>Perl Scripts</i> called <b>BiLE.pl</b>, <b>BiLE-Weigh.pl</b>, and <b>tld-expand.pl</b> to your Kali Linux system.<br><br>{{Admon/important|Perl Scripts Containing Errors When Executed|If errors occur, <b>check to see if that Perl Scripts were 
properly downloaded. If they contain HTML code, an alternative to 
downloading is to display the Perl Script in the web-browser, copying and pasting the code to the file on your computer</b> (<i>as opposed to right-clicking link and saving to your computer</i>). }}<br></li>
 <li>Run the following command: <b>perl  BiLE.pl  sensepost.com  output.sensepost.com</b>  (assuming BiLE.pl is located in the current directory).<br><br>Note: This process may take serveral minutes to complete.<br><br></li>
 <li>When the process has completed, a report called "<b>output.sensepost.com.mine</b>"
 (contained in the current directory) will be created that display 
associated links with the sensepost.com website. Using a text editor, 
view the contents of that file. Write in your lab log-book the number of
 lines in the file "output.sensepost.com.mine".</li>
 <li>If there is not enough information in this file, run the <b>BiLE.pl</b> script for the URL: <b>linux.senecac.on.ca</b> to be sorted in the file called <b>output.linux.senecac.on.ca</b></li>
 <li>Another Perl Script called <b>BiLE-weigh.pl</b> is used to rank the 
significance (relevance) of the related links with higher ranking links 
near the bottom of the file. This Perl Script requires the URL of the 
target website, as well as the output-file (generated by the BiLE.pl 
Perl Script.</li>
 <li>Issue the following command: <b> perl BiLE-weigh.pl sensepost.com output.sensepost.com.mine</b> (Assuming BiLE.pl Script and "output.sensepost.com" are contained in the current directory).<br><br> {{Admon/important|Error: Sort: open failed: +1: No such file or directory|If you run the <b>BiLE-Weigh.pl</b> command, and encounter the above error, then make the following editing changes for this script:<br><br><b>change following line:</b><nowiki> 'cat temp | sort -r -t ";" +1 -n &gt; @ARGV[1].sorted';</nowiki><br><br><b>to read:</b><nowiki> `cat temp | sort -r -t ":" -k 2 -n > @ARGV[1].sorted`;</nowiki><br><br>(Note: ` in this case is "Left-Tick" representing command substitution - not to be confused with a single-quote.<br /><br />}}<br><br></li>
 <li>View the contents of the file "output.sensepost.com.sorted" in your
 current directory. Notice the ranking of the relavance of links 
associated with "sensepost.com" website. Record the number of lines in 
this file in your lab log-book. What conclusions can you draw in terms of link analysis? Write this information down in your lab log-book.</li>
 <li>Run the <b>BiLE-weigh.pl</b> perl script for the URL: <b>linux.senecac.on.ca</b> and using the file: <b>output.linux.senecac.on.ca.mine</b></li>
<li>The final step in the information gathering process is to perform <b>Domain Name Expansion</b>. There are two parts to this process:<br /><br /><ul><li>Variations in the DNS Name (use <b>host</b> command)</li><li>Variations in the Top Level Domain (use <b>tld-expand.pl</b> Perl Script)</li></ul><br></li>
<li>Open a shell terminal, and type the following command: <b>host -t ns sensepost.com</b> (If there is a long list of variations, you can redirect stdout to a text file).</li>
<li>Record the various name servers that are listed in your lab log-book.</li>
<li>Create an input file called <b>sensepost.com.domains.txt</b>, and place any domain names that you have discovered, and then save and exit editing session.</li>
<li> Issue the command: <b>perl tld-expand.pl sensepost.com.domains.txt  sensepost.com.domains.variations.txt</b>. What do these variations represent in terms of reconnaissance? Record your finds in your lab log-book.</li>
<li>Proceed to Task #3<br><br></li>
</ol>

<p><b>Answer Task #2 observations / questions in your lab log book.</b>
</p><p><br>
</p>

<h2> <span class="mw-headline">Task #3: Foot-printing</span></h2>
<br>
As opposed to the Information Gathering phase (that collects information such as IP Addresses), the Foot-printing phase tends to gain a “clearer picture” of the structure of the organization's computer system. This can include <b>relationships among servers</b>, as well as noting <b>IP Address ranges</b>.
<br><br>
Footprinting (in simpler terms) means <b>Network Mapping</b>.
<br><br>
<b>Note:</b> You will be using information that you gathered from the server: <b>sensepost.com</b> in order to assist you with this lab.
<br><br>
INSTRUCTIONS:
<ol>
 <li>In a shell window, issue the following command: <b>host  sensepost.com</b></li>
 <li>Record the results in your lab log-book.</li>
 <li>Issue the same command with following options: <b>host -t ns  sensepost.com</b></li>
 <li>Record the results in your lab log-book.</li>
 <li>Issue the following command: <b>nslookup sensepost.com</b></li>
 <li>How does this information differ from the other 2 commands previously issued?</li>
 <li>Issue the following command: <b>whois sensepost.com</b></li>
 <li>List the additional general information that is provided from your all three previous commands.</li>
 <li>How do you think that this recently collected information can help you "map" the target computer's network?</li>
 <li>Proceed to Task #4</li>
 </ol>
<br>
<p><b>Answer the Task #3 observations / questions in your lab log book.</b>
</p>

<br />
<h2> <span class="mw-headline">Task #4: Obtaining User Information</span></h2>
<br>
You will be using the information collected in Task #1 to assist with obtaining User information in this task.
<br>
{{Admon/important|Install metagoofil program|<br><br> The harvester program is already installed in your Kali system, but you will need to install the program metagoofil. Issue the command (as root):<br><br><b>apt-get install metagoofil</b><br><br>}}<br><br>


<p>

INSTRUCTIONS:
</p>
<ol>
 <li>Issue the command <b>theharvester --help</b>, to learn how to run this script again with the following options:<ul><li>Domain: <b>sensepost.com</b></li><li>Number of limited results: <b>100</b></li><li>Data Source: <b>google</b></li><li>Output filename: <b>~/sensepost.user</b><br><br></li></ul></li>
 <li>Record any user information that you consider relevant (for penetration testing) in your lab log-book.</li>
 <li>For user information collected so far, use this information to see 
if you can access their profiles or other information on social media 
sites (eg. <i>Facebook</i>, <i>Classmates</i>, <i>MySpace</i>, <i>Twitter</i>, etc.).</li>
 <li>Finally, we will be obtaining documents from the targeted network 
(via Google) that may help provide more information regarding the users.</li>
 <li>Issue the following command: <b>metagoofil --help</b> to learn how to run this script again with the following options:<ul><li>Domain: <b>sensepost.com</b></li><li>Number of limited results: <b>10</b></li><li>Number of files to download (<i>-n option</i>): <b>10</b></li><li>Filetype: <b>pdf,ppt</b></li><li>output directory: <b>sensepost.docs</b></li></ul></li>
 <li>Check to see if any files were downloaded. If so, write the filenames in your lab log-book.</li>
<li>Proceed to Task #5</li>
 </ol>
<br>
<p><b>Answer Task #4 observations / questions in your lab log book.</b>
</p><p><br>
</p>

<h2> <span class="mw-headline">Task #5: Verification / The "Tank" Server</span></h2>
<br>
{{Admon/important|Location of dnsmap Utility in Kali Linux|The <b>dnsmap</b> utility is a time-saving method of determining reverse dns lookups in a batch mode involving an input file of collected dns entries.<br><br>This utility is contained in your Kali Linux boot media under the file pathname: <b>/pentest/enumeration/dns/dnsmap</b>}}
<br>
It is important to "double-check" the validity of your collected information - in particular, your IP addresses. If any servers are no longer running, this can waste a tremendous amount of time during the scanning process. Remember: the longer a scan takes to execute, the more vulnerable you are as the penetration tester to detection.
<br><br>

INSTRUCTIONS:
<ol>
 <li>Open a web-broswer and go to the website: <b>www.bing.com</b></li>
 <li>Enter the IP addresses that you have gathered during your reconnaisance phase for <b>sensepost.com</b>. Verify that each IP address is valid, and it currently operational.</li>
<li>For each of the related IP address information you have gathered regarding sensepost.com, use the <b>nslookup</b> command to verify it's existence.</li>
 <li>Change to the directory that contains <b>dnsmap</b> utility.</li>
<li>Run the <b>dnsmap</b> utility with an input file containing your collected IP_ADDRESSES.</li>
<li>Seneca College has a special server (called <b>tank</b>) that is used for penetration testing. No only is this server intended for educational purposeses only, but students are <b>NOT</b>
 allowed to perform penetration testing unless that have completed a 
form that is distributed and collected by your instructor to permit 
students to perform testing on that server for the semester!<br><br><b>Once
 you have signed and given the tank server consent form your your 
instructor, try gathering information regarding this server called 
"tank", and record your findings in your lab log-book.</b><br><br></li>
<li>Proceed to "Completing the Lab"</li>
</ol>
<br>
<p><b>Answer Task #5 observations / questions in your lab log book.</b>
<br><br>

</p><h1> <span class="mw-headline"> Completing the Lab </span></h1>
<p><b>Arrange evidence for each of these items on your screen, then ask 
your instructor to review them and sign off on the lab's completion:</b>
</p>
<ol>
 <li>Reconnaissance Information from sensepost.com:
 <ul>
 <li>Basic information from <b>sensepost.com</b> website via the Netcraft utility site.</li>
 <li>Reports from running <i>BiLE.pl</i> and <i>BiLE-Weigh.pl</i> for <i>sensepost.com</i></li>
 <li>Main DNS information (Footprint) for <i>sensepost.com</i></li>
 <li>User information (e-mail addresses) for the <i>sensepost.com</i> site.</li>
 <li>Verification of DNS information for <i>sensepost.com</i> site.<br><br></li>
 </ul>
 </li>
 <li>Completed Lab 2 notes (including common commands, etc).</li>
</ol>
<p><br>
</p>
<h1> <span class="mw-headline"> Preparing for Quizzes </span></h1>

<ol>
 <li>List the major phases contained in penetration testing.</li>
 <li>Explain the difference between <b>reconnaissance</b> and <b>footprinting</b>.</li>
 <li>List 3 open-source tools to assist in the <b>Footprinting</b> phase of penetration testing.</li>
 <li>Briefly describe the process to obtain key documents from a server using <b>google.ca</b></li>
 <li>Briefly describe the steps to obtain IP, operating system information from a website called <b>linux.senecac.on.ca</b>. Indicate how this information might be useful in future stages of penetration testing.</li>
 <li>Define the term <b>link analysis</b>. What open-source tools can be used to perform <i>link analysis</i>?</li>
 <li>Define the term <b>Footprinting</b> as it relates to penetration testing.</li>
 <li>List the steps (using open source tools) to obtrain user account 
information of a targeted server. Indicate how this information might be
 usedful in future stages of penetration testing.</li>
 <li>Why do you think that verification of gathered information (such as
 IP address (IP address ranges) is critical prior to proceeding to the 
scanning and enumeration phases?</li>
</ol>