Changes

Jump to: navigation, search

Tutorial9: Regular Expressions

565 bytes added, 20:36, 4 September 2023
no edit summary
{{Admon/caution|DO NOT USE THIS VERSION OF THE LAB. This page will no longer be updated.|'''New version here:''' https://seneca-ictoer.github.io/ULI101/A-Tutorials/tutorial9<br />'''Andrew's students please go here:''' http://wiki.littlesvr.ca/wiki/OPS145_Lab_8}}
=USING REGULAR EXPRESSIONS=
<br>
|- valign="top" style="padding-left:15px;"
|colspan="2" |Course Notes'''Slides:'''<ul><li>Week 9 Lecture 1 Notes:<br>[https://ictwiki.cdot.senecacollege.ca/~murrayuli101/slides/ULI101-9.1.pdf PDF] | [https://wiki.saulcdot.senecacollege.ca/uli101/slides/ULI101-Week99.1.pptx PPTX]</li><li>Week 9 Lecture 2 Notes:<br> [https://wiki.cdot.senecacollege.ca/uli101/slides/ULI101-9.2.pdf PDF] | [https://ictwiki.cdot.senecacollege.ca/~murray.saululi101/uli101slides/ULI101-Week99.2.pptx PPTX]<br></li></ul>
| style="padding-left:15px;" |'''Regular Expressions:'''
* [https://techterms.com/definition/regular_expression#:~:text=A%20regular%20expression%20(or%20%22regex,wildcards%2C%20and%20ranges%20of%20characters.&text=A%20regular%20expression%20can%20be,%2C%20such%20as%20%22app%22. Definition]
* [https://en.wikipedia.org/wiki/Regular_expression#:~:text=Regular%20expressions%20are%20used%20in,built%2Din%20or%20via%20libraries. Purpose (WIKI)]<br><br>
| style="padding-left:15px;"|'''Linux Commands:'''
* [https://ss64.com/bash/egrep.html egrep]
* [https://www.man7.org/linux/man-pages/man1/man.1.html man]
* [https://linux.die.net/man/1/wget wget]
|colspan="1" style="padding-left:15px;" width="30%"|'''Brauer Instructional Videos:'''<ul><li>[https://www.youtube.com/watch?v=-2pwLHcvCsU&list=PLU1b1f-2Oe90TuYfifnWulINjMv_Wr16N&index=12 Using grep Command with Regular Expressions]</li></ul>
|}
=INVESTIGATION 1: SIMPLE &amp; COMPLEX REGULAR EXPRESSIONS=
<span style="color:red;">'''ATTENTION''': This online tutorial will be required to be completed by '''Friday in week 10 by midnight''' to obtain a grade of '''2%''' towards this course</span><br><br> 
In this investigation, you will learn how to use the '''grep''' command with '''simple and complex regular expressions'''<br>to help search for ''patterns'' contained in text files.
# '''Login''' to your matrix account.<br><br>
# Issue a Linux command to '''confirm''' you are located in your '''home''' directory.<br><br>The '''wget''' command is used to download files from the Internet to your shell.<br>This will be useful to download '''text files''' and '''data files''' that we will be using in this tutorial.<br><br># Issue the following linux Linux command to '''downloadcopy''' a text file to ''your '' '''home''' directory from the ULI101 home directory:<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/cp ~murray.saululi101/uli101tutorialfiles/textfile1.txt<~/nowiki></span><br><br>
# View the contents of the '''textfile1.txt''' file using the '''more''' command see what data is contained in this file.<br><br>Although there are several Linux commands that use regular expressions,<br>we will be using the '''grep''' command for this investigation.<br><br>[[Image:regexps-1.png|thumb|right|250px|Output of '''grep''' command matching simple regular expression "'''the'''" (only lowercase). Notice the pattern matches larger words like "'''their'''" or "'''them'''".]]
#Issue the following Linux command to match the pattern '''the''' within '''textfile1.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "the" textfile1.txt</span><br><br>Take a few moments to view the output and observe the matched patterns.<br><br>
# Issue the following Linux command to create the '''regexps''' directory: <span style="color:blue;font-weight:bold;font-family:courier;">mkdir ~/regexps</span><br><br>
# Change to the '''regexps''' directory and confirm that you have moved to this directory.<br><br>
# First, issue the following Linux command to download copy another data file called '''numbers1.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/cp ~murray.saululi101/uli101tutorialfiles/numbers1.dat<~/nowiki>regexps</span><br><br>
# View the contents of the '''numbers.dat''' file using the '''more''' command and quickly view the contents of this file.<br>You should notice '''valid''' and '''invalid''' numbers contained in this file. When finished, exit the more command.<br><br>
# Issue the following linux pipeline command to display only '''whole''' numbers (i.e. no '''+''' or '''-''' sign):<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9]*$" numbers1.dat | tee faulty.txt</span><br><br>You may have noticed that the command '''does not entirely work'''. You may notice an '''empty line'''<br>(which is NOT a whole number). This occurs since the * regular expression symbol represents<br>ZERO or MORE occurrences of a number. You can use an additional numeric character class<br>with the * regular expression symbol to search for one or more occurrences of a number.<br><br>
# Issue the following Linux pipeline command to display <u>only</u> '''signed''' integers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-][0-9][0-9]*$" numbers1.dat | tee signed.txt</span><br><br>What did you notice? Positive and negative numbers display, not '''unsigned''' numbers.<br><br>[[Image:regexps-8.png|thumb|right|300px|Simultaneous '''anchoring''' of regular expressions using '''character class''' and '''zero or more occurrences''' to display '''signed''' and '''unsigned''' integers.]]
# Issue the following Linux pipeline command to display '''signed''' or '''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]*[0-9][0-9]*$" numbers1.dat | tee all.txt</span><br><br>Did this command work?<br><br>
# Issue the following command to check that you created those hard links: <br><span style="color:blue;font-weight:bold;font-family:courier;">bash /home/murray.saul/myscripts~uli101/week9-check-1</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script. If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>You can also use the '''grep''' command using ''regular expression'' as a '''filter''' in pipeline commands.<br><br>
# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | grep "[0-9].*dat$"</span><br><br>What did this pipeline display?<br><br>
# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | grep "[a-z].*txt$"</span><br><br>What did this pipeline display?<br><br>
<br>
In this sectioninvestigation, you will learn how to use '''extended regular expressions''' with the '''egrep''' command<br>to further refine your search patterns.
# Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account.<br><br>
# Issue the following Linux command to download copy another data file called '''numbers2.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/cp ~murray.saululi101/uli101tutorialfiles/numbers2.dat<~/nowiki></span><br><br>
# View the contents of the '''numbers2.dat''' file using the '''more''' command and quickly view the contents of this file.<br>You should notice ''valid'' and ''invalid'' numbers contained in this file. When finished, exit the more command.<br><br>[[Image:eregexps-1.png|thumb|right|300px|'''Weakness''' of '''complex''' regular expressions that do not '''limit''' the number of '''positive''' or '''negative''' signs.]]
# Issue the following Linux command to display '''signed''' or '''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]*[0-9][0-9]*$" numbers2.dat</span><br><br>You should notice '''multiple''' '''+''' or '''-''' '''signs''' appear <u>prior</u> to some numbers.<br>This occurs since you are searching or one or MORE occurrences of a + or - sign.<br><br>Using '''extended regular expression''' symbols to specify '''minimum''' and '''maximum''' repetitions: '''{min,max}''' can solve that problem.<br><br>
# Issue the following Linux command (using extended regular expression symbols)<br>to display '''signed''' or '''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]{0.,1}[0-9]{1,}$" numbers2.dat</span><br><br>'''NOTE: No output will be displayed! Why?'''<br><br>This is due to the fact that the '''grep command was NOT issued correctly to use extended regular expression symbols'''.<br>You would need to issue either '''grep -E''', or just issue the '''egrep''' command. The egrep command works with<br>'''all''' regular expression symbols, and should be used in the future <u>instead</u> of the older grep command.<br><br>We will use '''pipeline commands''' to both display stdout to the screen and save to files<br>for confirmation of running these pipeline commands when run a '''checking-script''' later in this investigation.<br><br>
# Issue the following Linux pipeline command using '''egrep''' instead of ''grep'':<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]{0,1}[0-9]{1,}$" numbers2.dat | tee better-number1.txt</span><br><br>[[Image:eregexps-2.png|thumb|right|300px|Using '''extended''' regular expression symbols (such as '''repetition''') to refine matches of ''signed'' and ''unsigned'' integers.]]You should have noticed that the command worked correctly this time because you used the '''egrep''' command.<br><br>'''NOTE:''' With extended regular expressions, the '''?''' symbol can be used to represent the '''{0,1}''' repetition symbols and the '''+''' symbol can be used to represent the '''{1,}''' repetition symbols<br><br>
# Issue the following Linux pipeline command using the repetition shortcuts <span style="font-weight:bold;font-family:courier;">"+"</span> and <span style="font-weight:bold;font-family:courier;">"?"</span>:<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]?[0-9]+$" numbers2.dat | tee better-number2.txt</span><br><br>You should have seen the '''same results''', but less typing was required.<br><br>
# Issue the following Linux pipeline command to display '''signed''', '''unsigned''', '''whole''', and '''decimal''' numbers:<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]{0,1}[0-9]{1,}[.]{0,1}[0-9]*$" numbers2.dat | tee better-number3.txt</span><br><br>Were all signed and unsigned intergers and decimal numbers displayed?<br><br>
# Issue the follwoing command to check that you correctly issued<br>those ''Linux pipeline commands'': <br><span style="color:blue;font-weight:bold;font-family:courier;">bash /home/murray.saul/myscripts~uli101/week9-check-2</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.<br>If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>You can also use extended regular expression symbols for '''grouping'''.<br>For example, you can search for repetitions of GROUPS of characters (like a word)<br>as opposed to just a single character or a GROUP of numbers as opposed to a single digit.<br><br># Issue the following linux pipeline command to download copy another data file called '''words.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/cp ~murray.saululi101/uli101tutorialfiles/words.dat<~/nowiki></span><br><br>
# View the contents of the '''words.dat''' file using the '''more''' command and quickly view the contents of this file.<br>Within this file, you should notice some lines that contain repetitions of words. When finished, exit the more command.<br><br>
# Issue the following linux pipeline command to display '''two or more occurrences''' of the word "the":<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the){2,}" words.dat | tee word-search1.txt more</span><br><br>'''NOTE: No output is displayed! Why?'''<br><br>This is due to the fact that a <u>space</u> should be included at the end of the word "'''the'''".<br>Usually words are separated by spaces; therefore, there were no matches since there were not occurrences<br>of "thethe" as opposed to "'''the the'''" (i.e. no space after repetition of the pattern).<br><br>
# Reissue the previous pipeline command with the word the followed by a '''space''' within the brackets:<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the ){2,}" words.dat | tee word-search2.txt</span><br><br>[[Image:eregexps-3.png|thumb|right|330px|Using '''extended''' regular expression symbols (such as '''grouping''') to refine matches of repetition of '''words''' (as opposed to ''characters'').]]The <span style="font-weight:bold;font-family:courier;">"|"</span> (or) symbol (same symbol as "pipe") can be used within the grouping symbols to allow matching of additional groups of characters.<br>Again, it is important to follow the character groupings with the space character<br><br>
# Issue the following linux pipeline command to search for '''two or more occurrences''' of the word "'''the '''" <u>or</u> '''two or more occurrences''' of the word "'''and '''":<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the |and ){2,}" words.dat | tee word-search3.txt</span><br><br>
# Issue the following Linux command to check that you correctly issued<br>those ''Linux pipeline commands'' using the '''tee''' command to create those text files:<br><span style="color:blue;font-weight:bold;font-family:courier;">bash /home/murray.saul/myscripts~uli101/week9-check-3</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.<br>If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>Let's issue a Linux '''pipeline''' command using the '''egrep''' command as a '''filter'''<br>using <u>both</u> '''complex''' and '''extended''' regular expressions.<br><br>
# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | egrep "[a-z]{1,}.*[0-9]"</span><br><br>What did this Linux pipeline command display?<br><br>
: The '''grep''' and '''egrep''' Linux commands are NOT the only Linux commands that use regular expressions.<br>In the next investigation, you will apply regular expressions to a number of Linux commands<br>that you already learned in this course.
=INVESTIGATION 3: OTHER COMMANDS USING REGULAR EXPRESSIONS =
<br>
In this sectioninvestigation, you will see commands other than '''grep''' or '''egrep''' that can use regular expressions.
# Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account.<br><br>
# Let's look at using regular expressions with the '''man''' command.<br>Issue the following linux command :<br><span style="color:blue;font-weight:bold;font-family:courier;">man ls</span><br><br>[[Image:other-re-1.png|thumb|right|300px|Entering '''/sort''' in the '''man''' command can search for the string "'''sort'''".]]
# We want to search for an option that can sort the file listing.<br>Type the following regular expression below and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/sort</span><br><br>'''FYI:''' The '''grep''' and '''egrep''' Linux commands contain the regular expressions within quotes, but '''most''' other Linux commands specify regular expressions using <br>'''forward slashes ''' (e.g. <span style="font-weight:bold;font-family:courier;">/regular expression</span> &nbsp; or &nbsp; <span style="font-weight:bold;font-family:courier;">/regular expression/</span>).<br><br>
# Scroll throughout the man pages for the ls command to view matches for the pattern "'''sort'''"<br>(You can press '''SPACE''' or key combination '''alt-b''' to move forward and backwards one screen respectively).<br><br>
# Press the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span> to '''exit''' the ''man'' pages for '''ls'''.<br><br>Let's use regular expressions with the '''moreless''' command.<br><br># Issue the following Linux command to download copy another data file called '''large-file.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/cp ~murray.saululi101/uli101tutorialfiles/large-file.txt<~/nowiki></span><br><br># View the contents of the '''large-file.txt''' file using the '''more''' command and quickly view the contents of this file.<br><br>[[Image:other-re-2.png|thumb|right|300px|Entering '''/uli101''' in the '''less''' command can display all matches of "'''uli101'''" throughout the text file.]]
# Issue the following Linux command to view the contents of the '''large-file.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">less large-file.txt</span><br><br>
#We want to search for a pattern '''uli101''' within this text file.<br>Type the following regular expression and press ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">/uli101</span><br><br>You should see the pattern "uli101" throughout the text file.<br><br>
# Press the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span> to exit the '''moreless''' command.<br><br>
# Try the same search techniques with the '''more''' command.<br><br>Does it work the same for the ''less'' command?<br><br>[[Image:other-re-3.png|thumb|right|300px|Entering '''/uli101''' in the '''vi''' command can search for the string "'''uli101'''".]]Let's learn how to perform a simple '''search and replace''' within the '''vi''' utility<br>by using regular expressions.<br><br>
# Issue the following Linux command to edit the '''large-file.txt''' file:<br><span style="color:blue;font-weight:bold;font-family:courier;">vi large-file.txt</span><br><br>Let's first perform a simple search within this text file.<br><br>
# Press the '''ESC''' key to make certain you are in '''COMMAND''' mode.<br><br>
# Type the following and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/uli101</span><br><br>You should notice the pattern "'''uli101'''" highlighted for ALL occurrences in this text file.<br><br>Let's '''search''' for the '''uli101''' pattern, and '''replace''' it in capitals (i.e '''ULI101''').<br><br>In vi, to issue a command, you need to enter '''LAST LINE''' MODE then issue a command.<br>Let's issue a command from '''LAST LINE''' MODE to search and replace '''uli101''' to '''ULI101'''.<br><br>[[Image:other-re-4.png|thumb|right|500px|In l'''ast line''' MODE in the '''vi''' text editor, issuing a command using regular expressions to convert '''uli101''' to '''ULI101'''.]]
# Making certain that you are '''COMMAND''' MODE in vi,<br>type the following and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">:%s/uli101/ULI101/g</span><br><br>'''NOTE:''' The letter '''g''' after the replace regular expression represents "'''global'''" and will replace ALL occurrences of uli101 in the text document (as opposed to replacing the first occurrence for every line).<br><br># Type the following (in uppercase letters) and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/ULI101</span><br><br>You should notice the pattern "'''ULI101'''" highlighted for ALL occurrences in this text file.<br><br># Navigate throughout the text file to confirm that ALL occurrences of '''uli101 ''' have been <u>replaced </u> with '''ULI101'''.<br><br>
# Save changes to your vi editing session and exit by typing the following and pressing ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">:x</span><br><br>
: After you complete the '''Review Questions''' section to get additional practice, then work on your<br>'''online assignment 3'''.
<br><br>
= LINUX PRACTICE QUESTIONS =
simulate a quiz:
https://ictwiki.cdot.senecacollege.ca/~murray.saululi101/uli101files/uli101_command_practice_9a.docx
Your instructor may take-up these questions during class. It is up to the student to attend classes in order to obtain the answers to the following questions. Your instructor will NOT provide these answers in any other form (eg. e-mail, etc).
'''Part A: Display Results from Linux Commands using Simple &amp; Complex Regular Expressions'''
Note the contents from the following tab-delimited file called '''~murray.saul/uli101/cars''':
<pre>
Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.
1. <span style="font-family:courier;font-weight:bold">grep plym ~murray.saul/uli101/cars</span><br>2. <span style="font-family:courier;font-weight:bold">grep -i fury ~murray.saul/uli101/cars</span><br>3. <span style="font-family:courier;font-weight:bold">grep “^[m-z]” ~murray.saul/uli101/cars</span><br>4. <span style="font-family:courier;font-weight:bold">grep -i “^[m-z]” ~murray.saul/uli101/cars</span><br>5. <span style="font-family:courier;font-weight:bold">grep “3$” ~murray.saul/uli101/cars</span><br>6. <span style="font-family:courier;font-weight:bold">grep -i “c.*5$” ~murray.saul/uli101/cars</span><br>
simulate a quiz:
https://ictwiki.cdot.senecacollege.ca/~murray.saululi101/uli101files/uli101_command_practice_9b.docx
Your instructor may take-up these questions during class. It is up to the student to attend classes in order to obtain the answers to the following questions. Your instructor will NOT provide these answers in any other form (eg. e-mail, etc).
'''Part A: Display Results from Linux Commands using Regular Expressions'''
Note the contents from the following tab-delimited file called '''~murray.saul/uli101/numbers.txt''':
<pre>
Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.
1. <span style="font-family:courier;font-weight:bold">grep "^[-+]" ~murray.saul/uli101/numbers.txt</span><br>2. <span style="font-family:courier;font-weight:bold">grep "^[-+]*.[0-9]" ~murray.saul/uli101/numbers.txt</span><br>3. <span style="font-family:courier;font-weight:bold">grep "^[+-]?[0-9]" ~murray.saul/uli101/numbers.txt</span><br> &nbsp; &nbsp;(Why?)<br>4. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]" ~murray.saul/uli101/numbers.txt</span><br>5. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]+$" ~murray.saul/uli101/numbers.txt</span><br>6. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]+[.]?[0-9]+$" ~murray.saul/uli101/numbers.txt</span><br>
_________________________________________________________________________________ Author: Murray Saul License: LGPL version 3Link: https://www.gnu.org/licenses/lgpl.html _________________________________________________________________________________ 
[[Category:ULI101]]

Navigation menu