Changes

Jump to: navigation, search

Tutorial9: Regular Expressions

4,462 bytes added, 20:36, 4 September 2023
no edit summary
{{Admon/caution|DO NOT USE THIS VERSION OF THE LAB. This page will no longer be updated.|'''New version here:''' https://seneca-ictoer.github.io/ULI101/A-Tutorials/tutorial9<br />'''Andrew's students please go here:''' http://wiki.littlesvr.ca/wiki/OPS145_Lab_8}}
=USING REGULAR EXPRESSIONS=
<br>
===Main Objectives of this Practice Tutorial===
:* Explain Define the purpose of term '''Regular Expressions'''
:* Explain the difference between '''Regular Expressions''' and '''Filename Expansion'''
:* Understand and use command symbols for '''Extended''' Regular Expressions and their purpose
:* List various several Linux commands (other than '''grep''' or '''egrep''') that can use regular expressions<br>
===Tutorial Reference Material===
|- valign="top" style="padding-left:15px;"
|colspan="2" |Course Notes'''Slides:'''<ul><li>Week 9 Lecture 1 Notes:<br>[https://ictwiki.cdot.senecacollege.ca/~murrayuli101/slides/ULI101-9.1.pdf PDF] | [https://wiki.saulcdot.senecacollege.ca/uli101/slides/ULI101-Week99.1.pptx PPTX]</li><li>Week 9 Lecture 2 Notes:<br> [https://wiki.cdot.senecacollege.ca/uli101/slides/ULI101-9.2.pdf PDF] | [https://ictwiki.cdot.senecacollege.ca/~murray.saululi101/uli101slides/ULI101-Week99.2.pptx PPTX]<br></li></ul>
| style="padding-left:15px;" |'''Regular Expressions:'''
* [https://techterms.com/definition/regular_expression#:~:text=A%20regular%20expression%20(or%20%22regex,wildcards%2C%20and%20ranges%20of%20characters.&text=A%20regular%20expression%20can%20be,%2C%20such%20as%20%22app%22. Definition]
* [https://en.wikipedia.org/wiki/Regular_expression#:~:text=Regular%20expressions%20are%20used%20in,built%2Din%20or%20via%20libraries. Purpose (WIKI)]<br><br>
| style="padding-left:15px;"|'''Linux Commands:'''
* [https://ss64.com/bash/egrep.html egrep]
* [https://www.man7.org/linux/man-pages/man1/man.1.html man]
* [https://linux.die.net/man/1/wget wget]
|colspan="1" style="padding-left:15px;" width="30%"|'''Brauer Instructional Videos:'''<ul><li>[https://www.youtube.com/watch?v=-2pwLHcvCsU&list=PLU1b1f-2Oe90TuYfifnWulINjMv_Wr16N&index=12 Using grep Command with Regular Expressions]</li></ul>
|}
= KEY CONCEPTS =
<i>A '''regular expression''' … is a sequence of characters that define a search pattern.  Usually such patterns are used by string searching algorithms for "find" or "find and replace" operations on strings, or for input validation. </i><br>Reference: https://en.wikipedia.org/wiki/Regular_expression ===Regular Expressions vs. Filename Expansion===
<table align="right"><tr valign="top"><td>[[Image:re-1.png|thumb|right|350px|'''Filename expansion symbols''' allow the Linux shell to expand filenames as arguments (referred to as “globbing”).]]</td><td>[[Image:re-2.png|thumb|right|250px|'''Concept''' of matching a simple pattern of text contained within a text file.]]</td></table>In a previous lesson, you learned that '''filename expansion symbols''' allow the Linux shell to expand filenames as arguments (referred to as “globbing”) for file management commands. This is very useful for managing multiple files sharing similar characteristics such as the same file extension.  ''Filename Expansion symbols'' are used to search, edit and manipulate text and are used with Linux file management commands such as '''ls''', '''rm''', '''mv''', '''cp''', '''cat''', '''less''', '''more''', '''head''', '''tail''', '''sort''', '''uniq''', '''cut''', '''tr''', and '''wc'''.<br><br> <i>A '''regular expression''' is a combination of two types of characters: '''literals''' and '''special characters'''.<br>In combination, these characters define a logical pattern. Strings of text can be compared to this pattern<br>to see if they fit the pattern defined by the expressionthere is a match.</i> Reference: https://www.whoishostingthis.com/resources/regex/
This usually refers to text that is <u>contained</u> inside a '''file''' or text as a result<br>of issuing Linux commands using a '''Linux pipeline command'''.
<br><br>
===Literal (Simple) Regular Expressions===
[[Image:re-3.png|thumb|right|250px200px|The simplest A '''simple''' ('''literal''') regular expression is a series of letters and numbers, possibly including white space (tabs or space charactersspaces).]]The simplest regular expression is a series of letters and numbers, possibly including white space (tabs or space charactersspaces), that have no special meaning. Such a <br>A '''simple''' ('''literal''') regular expression consists of ``literals''; that is, normal letterscharacters, which used to match only an identical letter in the data being searchedpatterns.<br><br>For example:"This is a Although there are many Linux commands that use regular expression"<br><br><i>When an editor searches for a expressions, the '''literal regular expressiongrep''', it can only score command is a ``hit'' if it finds exactly that sequence useful command to learn how to display matches of patterns of characters in the data it is searchingstrings within text files.</ibr> <br>ReferenceFor example:<span style="color:blue;font-weight: httpbold;font-family://osr507doccourier;">grep Linux document.xinuos.comtxt</en/OSUserG/_Literal_chars_in_regexps.htmlspan><br><br>
===Complex / Extended Regular Expressions=== The problem with using literals or simple regular expressions is that only <u>simple</u> or <u>general</u> patterns are matched.
For example, the pattern “'''theComplex Regular Expressions'''” would be matched for larger word containing that pattern like <br><br>The problem with just using '''theresimple''', ('''theyliteral''', '''either''', '''them''', '''their''', etc.) regular expressions is that only <u>simple<br/u>Another problem is that you may want to search for pattern at a specific location within the string of text (like at the beginning or end)<u>general</u> patterns are matched.
There are other regular expression tools ''Complex Regular Expressions'' use symbols to provide help match text for more <u>precise matches</u> (complex) patterns. These tools <br>The most common complex regular expression symbols are displayed below:<br><br>:'''Anchors: ''' <span style="color:blue;font-family:courier;font-weight:bold;">^</span> , <span style="color:blue;font-family:courier;font-weight:bold;">$</span><br>Match lines the begin (^) or end ($) with a pattern.<br>:'''Single Character:'''complex&nbsp; <span style="color:blue;font-family:courier;font-weight:bold;">.</span><br>Represents a single character that can be any type of character.<br>:'''Character Class:'' and ' <span style="color:blue;font-family:courier;font-weight:bold;">[ ]</span> , <span style="color:blue;font-family:courier;font-weight:bold;">[^ ]</span><br>Represents a single character but with restrictions.<br>:''extended'Zero or More Occurrence:'' regular expressions' <span style="color:blue;font-family:courier;font-weight:bold;">*</span><br>Zero or more occurrences of previous character. <br><br>
:Examples of '''Complex Regular Expressionscomplex regular expressions''' use symbols to help match text for more <u>precise</u> (complex) patterns.<br>The most common complex regular expression symbols are displayed below:
:*'''Anchors: ''' <span styletable align="font-family:courier;font-weight:bold;left">^</span> , <span styletr valign="font-family:courier;font-weight:bold;top">$</spantd><br>Match lines the begin (^) or end ($) with a pattern[[Image:re-4.png|thumb|right|200px|Example of using '''anchors'''.]]<br/td><brtd>[[Image:*re-5.png|thumb|right|175px|Example of matching by '''Single Character:character(s)''' &nbsp; <span style="font-family:courier;font-weight:bold;">.]]</spantd><brtd>Represents a single character that can be any type [[Image:re-6.png|thumb|right|220px|Example of using '''characterclass'''.]]<br/td><brtd>[[Image:*re-7.png|thumb|right|200px|Example of matching '''Character Class:zero or more occurrence of preceding character''' .]]</td><span style="font-family:courier;font-weight:bold;"/tr>[ ]</spantable> , <span style="font-family:courier;font-weight:bold;"br>[^ ]</spanbr><br>Represents a single character but with restrictions.<br><br>:*'''Zero or More Occurrence:''' <span style="font-family:courier;font-weight:bold;"br>*</spanbr><br>Zero or more occurrences of previous character.<br><br>
Examples of how to use these complex regular expressions with the grep command are displayed below: '''Extended Regular Expressions'''
<table align="left"><tr valign="top"><td>[[Image:re-4.png|thumb|right|220px|Example of using ''Extended Regular Expressions'anchors'''.]]consist of additional special characters to “extend”</tdbr><td>[[Image:re-5the capability of regular expressions.png|thumb|right|200px|Example of matching by You must use the '''character(s)egrep'''.]]</td><td>[[Image:re-6.png|thumb|right|220px|Example of using or '''character class'''.]]</td><td>[[Image:regrep -7.png|thumb|right|200px|Example of matching E'''zero or more occurrence of preceding character'''.]]</td></tr></table><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>commands<br>in order to properly use extended regular expressions.
===Extended Regular Expressions===
'''Extended Regular Expressions''' consist of additional special characters<br> to “extend” the capability of regular expressions:
:'''Repetition:''' <span style="color:blue;font-family:courier;font-weight:bold;">{min,max}</span><br>Allows for more precise repetitions. Using braces, you can specify<br>the '''minimum''' and/or '''maximum''' number of repetitions.
:*'''RepetitionGroups: {min,max}'''<span style="color:blue;font-family:courier;font-weight:bold;">( )</span><br>Allows you to search for more precise repetitions. Using bracesrepetition for a '''group of characters''', a '''word''', you can specify<br>the or a '''minimumphase''' and.<br>You enclose them within brackets <span style="font-family:courier;font-weight:bold;">( )</or span> to specify a '''maximumgroup''' number of repetitions.
:*'''Groupingor Condition: ( )''' <span style="color:blue;font-family:courier;font-weight:bold;">|</span><br>Perhaps you want Can be used with '''groups''' to search for repetition for match a group variety of characters, a wordcharacter(s), words or a phasephases.<br>You can enclose them within brackets The | symbol is used to separate the variety of character( s) to specify within a ''group''.<br><br>
:*Examples of how to use '''or Condition: |extended regular expressions'''<br>Can be used with grouping to match a variety of character(s), words or phases.<br>The | symbol is used to separate the variety of character(s) within a group.'''egrep''' command are displayed below:<br><br>
Examples <table align="left"><tr valign="top"><td>[[Image:re-8.png|thumb|right|280px|Example of using '''repetition'''.]]</td><td>[[Image:re-9.png|thumb|right|250px|Example of using '''groups'''.]]</td><td>[[Image:re-10.png|thumb|right|250px|Example of how to use these complex regular expressions using '''or''' condition with the '''egrepgroups''' command are displayed below:.]]</td></tr></table><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>
<table align="left"><tr valignINVESTIGATION 1: SIMPLE &amp; COMPLEX REGULAR EXPRESSIONS="top"><td>[[Image:re-8.png|thumb|right|280px|Example of using '''repetition'''.]]</td><td>[[Image:re-9.png|thumb|right|250px|Example of using '''grouping'''.]]</td><td>[[Image:re-10.png|thumb|right|250px|Example of using '''or condition with grouping'''.]]</td></tr></table><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>
<span style=INVESTIGATION 1"color: SIMPLE &ampred; COMPLEX REGULAR EXPRESSIONS=">'''ATTENTION''': This online tutorial will be required to be completed by '''Friday in week 10 by midnight''' to obtain a grade of '''2%''' towards this course</span><br><br>
<br>In this sectioninvestigation, you will learn how to use the '''grep''' command with '''simple and complex regular expressions'''<br>to help search for ''patterns'' contained in text files.
# '''Login''' to your matrix account.<br><br>
# Issue a Linux command to '''confirm''' you are located in your '''home''' directory.<br><br>The # Issue the following linux Linux command to '''wgetcopy''' command is used to download files from the Internet a text file to ''your shell.<br>This will be useful to download '''text files''home' and '''data files''' that we will be using for this tutorial.directory from the ULI101 home directory:<br><br># Issue the following linux command to download a text file to your home directory:<br><span style="color:span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https:cp ~uli101/tutorialfiles/icttextfile1.senecacollege.ca/txt ~murray.saul/uli101/textfile1.txt</nowiki></span><br><br># Issue the '''ls''' command to confirm that the text file was downloaded.<br><br># View the contents of the '''textfile1.txt''' file using the '''more''' command and quickly view the contents of see what data is contained in this file.<br><br>Although there are several Linux commands that use regular expressions,<br>we will only be using the '''grep''' command for this investigation.<br><br>#Issue the following linux pipeline command to match the pattern the within [[Image:regexps-1.png|thumb|right|250px|Output of '''textfile1.txtgrep''' command matching simple regular expression "'''the''':<br><span style="color:blue;font-weight:bold;font-family:courier;(only lowercase). Notice the pattern matches larger words like ">grep '''their'''"theor "'''them'''" textfile1.txt</span><br><br>Take a few moments ]]#Issue the following Linux command to view match the output and observe the matched pattern.<br><br># Now, issue the grep linux pipeline command with '''the '''-iwithin '''textfile1.txt''' option to ignore case sensitively:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -i "the" textfile1.txt</span><br><br>What do you notice is different with this pipeline command?Take a few moments to view the output and observe the matched patterns.<br><br>You will notice that # Issue the pattern "'''grep Linux command with the'''<span style=" is matched including larger words like "'''them'''" and "'''their'''font-weight:bold;font-family:courier;".<br>You can issue the '''grep''' command with the -w -i</span> option to only match the pattern as a '''word'''.<br><br># Issue the following linux pipeline command:<brignore case sensitively:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "the" textfile1.txt</span><br><br>You should now see only strings of text What do you notice is different when issuing this command?<br><br>You will notice that match the word pattern "'''"the"''' (upper or lower case).<br><br>Matching literal or simple regular expressions can be useful, but are " is matched including larger words like "'''limitedthem'''<br>in what they can assist with pattern matching. For example, you may want to<br>search for a pattern located at the " and "'''beginningtheir''' or '''end''' of the string".<br><br>There are other regular expression symbols that provide more You can issue the '''precisegrep''' pattern matches.command with the <span style="font-weight:bold;font-family:courier;">-w<br/span>These special characters are known option to only match the pattern as a '''complex''' and '''extendedword''' regular expressions symbols.<br> In this section, we will focus on '''complex regular expressions''' and then discuss<br>'''extended regular expressions''' in '''investigation 2'''.<br><br# Issue the following Linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;"># Issue grep -w -i "the following Linux pipeline command:" textfile1.txt</span><br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "^br>You should now see only strings of text that match the" textfile1word '''the''' (upper or lower case).txt</span><br><br>The Matching literal or simple regular expressions can be useful, but are '''^limited''' symbol is referred <br>in what pattens they can match. For example, you may want to as an <br>search for a pattern located at the '''anchorbeginning'''. In this case, it only matches<br>the word "or '''theend'''" (both upper or lowercase) at of the string.<ubr>beginning</u> of the string.<br>The There are other regular expression symbols that provide more '''$precise''' symbol is used to anchor patterns at the end of stringssearch pattern matching.<br>These special characters are known as '''complex''' and '''extended''' regular expressions symbols.<br><br># Issue For the following Linux pipeline command:remainder of this investigation, we will focus on '''complex regular expressions''' and then<br><span stylefocus on ''extended regular expressions'' in INVESTIGATION 2.<br><br><table align="right"><tr valign="colortop"><td>[[Image:blue;fontregexps-weight:bold;font-family:courier;">grep -w -i "2.png|thumb|right|280px|Anchoring regular expressions at the$" textfile1.txt'''beginning''' of text.]]</spantd><brtd>[[Image:regexps-3.png|thumb|right|250px|Anchoring regular expressions at the '''ending''' of text.]]<br/td>What do you notice?<br/tr><br/table># Issue the following Linux pipeline command to anchor the '''word''' "the" simultaneously:<br>at the <u>beginning</u> and <u>end</u> of the string:<br><span stylespan style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "^the$" textfile1.txt </span><br><br>What do you notice?The '''^''' symbol is referred to as an '''anchor'''.<br>In this case, it only matches<br>Anchoring patterns at both the word "'''the'''" (both upper or lowercase) at the <u>beginning</u> and of the string.<ubr>ending</ubr> of strings can greatly assist# Issue the following Linux command:<br>for more robust search patterns.<br><br>We will now be demonstrating span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "the usefulness of '''simultaneous anchoring'''<br>with other complex regular expressions symbols$" textfile1.txt<br/span><br># Issue the following command to match strings that begin with 3 characters:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^..." textfile1.txtThe '''$''' symbol is used to anchor patterns at the <u>end</spanu>of the string.<br><br>What do you notice? Can lines that contain # Issue the following Linux command to anchor the <u>word</u> "'''less than 3 charactersthe''' be displayed?"<br><br'''simultaneously''' at the <u>beginning</u># Issue the following command to match strings that begin and end with 3 characters:<bru>end</u> of the string:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "^...the$" textfile1.txt</span><br><br>What do you notice compared to the previous command?<br><br># Issue Anchoring patterns at both the following command to match <u>beginning</u> and <u>ending</u> of strings that begin with 3 digits:can greatly assist<br><span style="color:blue;font-weight:bold;font-family:courier;"for more '''precise''' search pattern matching.<br><br>grep "^[0-9][0-9][0-9]" textfile1.txtWe will now be demonstrate the '''effectiveness''' of <u>combining</spanu><br>'''anchors''' with <bru>What did you notice?other<br/u>complex regular expressions symbols.<br># Issue the following command to match strings that end with 3 uppercase letters:<br><span styletable align="right"><tr valign="color:blue;font-weight:bold;font-family:courier;top">grep "<td>[A-Z][A-Z][AImage:regexps-Z]$" textfile14.txtpng|thumb|right|280px|Anchoring regular expressions using '''period''' symbols at the '''beginning''' of text.]]</spantd><brtd><br>Did any lines match this pattern?<br><br># Issue the following command to match strings that consist of only 3 digits[[Image:<br><span style="color:blue;font-weight:bold;font-family:courier;"regexps-5.png|thumb|right|250px|Anchoring regular expressions using '''period''' symbols simultaneously at the '''beginning''' and '''ending''' of text.]]</td></tr>grep "^[0-9][0-9][0-9]$" textfile1.txt</span><br><br>What did you notice?<brtable># Issue the following Linux command to match strings that '''begin with 3 characters''':<br>The <span style="color:blue;font-weight:bold;font-family:courier;">grep "*^..."textfile1.txt</span> complex regular expression symbol is often confused with filename expansion.<br>In other words, it does NOT represent zero or more of <br>What do you notice? Can lines that contain '''any characterless than 3 characters''', but zero or more occurrencesbe displayed?<br><br>of the character # Issue the following Linux command to match strings that comes '''beforebegin <u>and</u> end with 3 characters''' the "*" symbol.:<br><br># To demonstration, issue the following command to display zero or more occurrences of the letter "'''x'''span style="color:<br><span style="color:blueblue;font-weight:bold;font-family:courier;">grep "x*^...$" textfile1.txt</span><br><br>You will most likely What do you notice most lines of compared to the file is displayed.previous command?<br><br># Let's issue a Issue the following Linux command to display match strings that contain more than one occurrence of the letter "x"'''begin with 3 digits''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "xx*^[0-9][0-9][0-9]" textfile1.txt</span><br><br>Why What did this workyou notice? because the pattern indicates one occurrence of the letter "x",<br>followed by zero or MORE occurrences of <br># Issue the letter "x".<following Linux command to match strings that '''end with 3 uppercase letters''':<br><br>If you combine the complex regular expression symbols span style=".*color:blue;font-weight:bold;font-family:courier;" it will act like<br>zero or more occurrences of any character (i.e. like grep "[A-Z][A-Z][A-Z]$"*" did in filename expansion)textfile1.txt<br/span><br><br><table align="right"><tr valign="top"><td>[[Image:regexps-6.png|thumb|right|220px|Anchoring '''3 digits''' at the '''beginning''' and '''ending''' of text.]]</td><td>[[Image:regexps-7.png|thumb|right|250px|Anchoring '''3 alpha-numeric characters''' at the '''beginning''' and '''ending''' of text.]]</td></tr></table>What type of strings match this pattern?<br><br># Issue the following Linux command to match strings that '''consist of only 3 digits''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9][0-9]$" textfile1.txt</span><br><br>What did you notice?<br><br># Issue the following Linux command to match strings that '''consist of only 3 alphanumeric digits''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[a-zA-Z0-9][a-zA-Z0-9][a-zA-Z0-9]$" textfile1.txt</span><br><br>What did you notice?<br><br>The <span style="font-weight:bold;font-family:courier;">"*"</span> complex regular expression symbol is often confused with the "*" '''filename expansion''' symbol.<br>In other words, it does NOT represent zero or more of '''any character''', but zero or more '''occurrences'''<br>of the character that comes '''before''' the <span style="font-weight:bold;font-family:courier;">"*"</span> symbol.<br><br># To demonstrate, issue the following Linux command to display '''zero or more occurrences''' of the letter "'''x'''":<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "x*" textfile1.txt</span><br><br>You will most likely notice most lines of the file is displayed.<br><br># Let's issue a Linux command to display strings that contain '''more than one occurrence''' of the letter "'''x'''":<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "xx*" textfile1.txt</span><br><br>Why did this work? because the pattern indicates one occurrence of the letter "x",<br>followed by '''zero or MORE occurrences''' of the <u>next</u> letter "x".<br><br>If you combine the complex regular expression symbols <span style="font-weight:bold;font-family:courier;">".*"</span> it will act like<br>zero or more occurrences of <u>any</u> character (i.e. like <span style="font-weight:bold;font-family:courier;">"*"</span> did in filename expansion).<br><br># Issue the following Linux command to match strings begin and end with a number with nothing or anything inbetween:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9].*[0-9]$" textfile1.txt</span><br><br># Issue the following command to match strings begin and end Using '''simultaneous anchors''' combined with a number with nothing or anything inbetween:<br>the <span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9].*[0-9]$.*" textfile1.txt</span><br><br>Using '''simultaneous anchors''' combined with the ".*" symbol(s) can help you to refine your search patterns of strings.<br><br># Issue the following linux pipeline Linux command to display strings that begin with a capital letter,<br>end with a number, and contains a capital X somewhere inbetween:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[A-Z].*X.*[0-9]$" textfile1.txt</span><br><br>Let's look at another series of examples involving searching for strings that only contain '''valid numbers'''.<br>We will use '''pipeline commands''' to both display stdout to the screen and save to files<br>for confirmation of running these pipeline commands when run a '''checking-script''' later in this investigation.<br><br>
# Issue the following Linux command to create the '''regexps''' directory: <span style="color:blue;font-weight:bold;font-family:courier;">mkdir ~/regexps</span><br><br>
# Change to the '''regexps''' directory and confirm that you have moved to this directory.<br><br>
# First, issue the following linux Linux command to download copy another data file called '''numbers1.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/cp ~murray.saululi101/uli101tutorialfiles/numbers1.dat<~/nowiki>regexps</span><br><br># View the contents of the '''numbers.dat''' file using the '''more''' command and quickly view the contents of this file.<br>You should notice '''valid ''' and '''invalid ''' numbers contained in this file. When finished, exit the more command.<br><br># Issue the following linux pipeline command to display only '''whole ''' numbers(i.e. no '''+''' or '''-''' sign):<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9]*$" numbers1.dat | tee faulty.txt</span><br><br>You may have noticed that the command '''does not entirely work'''. You may notice an '''empty line'''<br>(which is NOT a whole number). This occurs since the * regular expression symbol represents<br>ZERO or MORE occurrences of a number. You can use an additional numeric character class<br>with the * regular expression symbol to search for one or more occurrences of a number.<br><br># Issue the following linux Linux pipeline command to display only whole numbers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9]*$" numbers1.dat | tee whole.txt</span><br><br>You should see that this now works.<br><br># Issue the following linux Linux pipeline command to display whole positive or negative <u>only</u> '''signed''' integers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-][0-9][0-9]*$" numbers1.dat | tee signed.txt</span><br><br>What did you notice?Positive and negative numbers display, not '''unsigned''' numbers.<br><br>[[Image:regexps-8.png|thumb|right|300px|Simultaneous '''anchoring''' of regular expressions using '''character class''' and '''zero or more occurrences''' to display '''signed''' and '''unsigned''' integers.]]# Issue the following linux Linux pipeline command to display only whole numbers (with '''signed''' or without a positive or negative sign)'''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]*[0-9][0-9]*$" numbers1.dat | tee all.txt</span><br><br>Did this command work?<br><br># Issue the following Linux command to check that you created those hard links: <br><span style="color:blue;font-weight:bold;font-family:courier;">bash /home/murray.saul/scripts~uli101/week9-check-1</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.<br>If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>You can also use the '''grep''' command using ''regular expression'' as a '''filter''' in pipeline commands.<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | grep "[0-9].*dat$"</span><br><br>What did this pipeline display?<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | grep "[a-z].*txt$"</span><br><br>What did this pipeline display?<br><br> : Although very useful, '''complex''' regular expressions do NOT <u>entirely</u> solve our problem of displaying<br> '''valid''' unsigned and signed numbers (not to mention displaying decimal numbers).<br><br>In the next investigation, you will learn how to use '''extended''' regular expressions that will completely solve this issue.<br>
:Proceed You can proceed to INVESTIGATION 2.<br>
=INVESTIGATION 2: EXTENDED REGULAR EXPRESSIONS =
<br>In this sectioninvestigation, you will learn how to use '''extended regular expressions''' with the '''egrep''' command<br>to help further refine your search when using regular expressionspatterns.
# Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account.<br><br>
# Issue the following linux pipeline Linux command to download copy another data file called '''numbers2.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/cp ~murray.saululi101/uli101tutorialfiles/numbers2.dat<~/nowiki></span><br><br># View the contents of the '''numbers2.dat''' file using the '''more''' command and quickly view the contents of this file.<br>You should notice ''valid '' and more ''invalid '' numbers contained in this file. When finished, exit the more command.<br><br># Issue [[Image:eregexps-1.png|thumb|right|300px|'''Weakness''' of '''complex''' regular expressions that do not '''limit''' the number of '''positive''' or '''negative''' signs.]]# Issue the following linux pipeline Linux command to display only whole numbers (with '''signed''' or without a positive or negative sign)'''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]*[0-9][0-9]*$" numbers2.dat | more</span><br><br>You should notice '''multiple''' '''+''' or '''-''' ''' signs ''' appear prior <u>prior</u> to some numbers.<br>This occurs since you are searching or one or MORE occurrences of a + or - sign.<br><br>Using '''extended regular expression''' symbols to specify '''minimum''' and '''maximum''' repetitions: '''{min,max}''' can solve that problem.<br><br># Issue the following linux pipeline Linux command (using extended regular expression symbols) <br>to display only whole numbers (with or without a positive '''signed''' or negative sign)'''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]{0.,1}[0-9]{1,}$" numbers2.dat | more</span><br><br>'''NOTE:No output will be displayed! Why?''' most likely, there were '''NO results'''. <br><br>This is due to the fact that the '''grep command was NOT issued correctly to use extended regular expression symbols'''. <br>You would need to issue either '''grep -E''' (, or more simply) just issue the '''egrep''' command. The egrep command works with <br>'''all ''' regular expression symbols, and should be used in the future <u>instead </u> of the older grep command.<br><br># Reissue the above We will use '''pipeline command using commands'''egrepto both display stdout to the screen and save to files<br>for confirmation of running these pipeline commands when run a ''' instead of checking-script''grep'':later in this investigation.<br><br># Issue the following Linux pipeline command using '''egrep''' instead of ''grep'':<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]{0,1}[0-9]{1,}$" numbers2.dat | tee better-number1.txt | more</span><br><br>You should have noticed that the command worked correctly this time because you used the [[Image:eregexps-2.png|thumb|right|300px|Using '''egrepextended''' command.<br><br>regular expression symbols (such as '''NOTE:repetition''' In extended regular expressions, the ) to refine matches of ''signed''and '?'unsigned'' symbol can be integers.]]You should have noticed that the command worked correctly this time because you used to represent the '''{0,1}egrep''' repetition symbolscommand.<br>and <br>'''NOTE:''' With extended regular expressions, the '''+?''' symbol can be used to represent the '''{0,1,}''' repetition symbols<br><br># Reissue and the above pipeline command using '''egrep+'''symbol can be used to represent the ' instead of ''grep{1,}''':repetition symbols<br><span style="color:blue;br># Issue the following Linux pipeline command using the repetition shortcuts <span style="font-weight:bold;font-family:courier;">egrep "^[+-]?[0-9]+$" numbers2.dat | tee better-number2.txt | more<</span>and <br><br>You should have seen the same results, but the extended regular expression required less typing.span style="font-weight:bold;font-family:courier;">"?"<br/span>:<br># Issue the following linux pipeline command to display signed, unsigned, whole, and decimal numbers:<br><<span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-]{0,1}?[0-9]{1,}[.]{0,1}[0-9]*+$" numbers2.dat | tee better-number3number2.txt | more</span><br><br># Issue You should have seen the following Linux command to check that you created those hard links: '''same results''', but less typing was required.<br><span style="color:blue;font-weight:bold;font-family:courier;"br># Issue the following Linux pipeline command to display '''signed''', '''unsigned''', '''whole''', and '''decimal''' numbers:<br>bash /home/murray.saul/scripts/week9<span style="color:blue;font-checkweight:bold;font-2</spanfamily:courier;"><br><br>If you encounter errorsegrep "^[+-]{0, then view the feedback to make corrections, and then re1}[0-9]{1,}[.]{0,1}[0-run the checking script9]*$" numbers2.<brdat | tee better-number3.txt</span>If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>You can also use extended regular expression symbols for '''grouping'''. For example, Were all signed and unsigned intergers and decimal numbers displayed?<br><br># Issue the follwoing command to check that you can search for repetitions of GROUPS of characterscorrectly issued<br>(like a word) as opposed to just a single character or a GROUP of numbers as opposed to a single digit.those ''Linux pipeline commands'': <br><br># Issue the following linux pipeline command to download another data file called '''words.dat''':<br><span style="color:span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/~uli101/words.dat</nowiki>week9-check-2</span><br><br># View If you encounter errors, then view the contents of feedback to make corrections, and then re-run the '''numbers2checking script.dat''' file using the '''more''' command and quickly view the contents of <br>If you receive a congratulation message that there are no errors, then proceed with this filetutorial.<br><br>You should notice valid and more invalid numbers contained in this filecan also use extended regular expression symbols for '''grouping'''. When finished<br>For example, exit the more command.you can search for repetitions of GROUPS of characters (like a word)<br><br># as opposed to just a single character or a GROUP of numbers as opposed to a single digit.<br><br># Issue the following linux pipeline command to display two or more occurrences of the word "the"copy another data file called '''words.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the){2,}" cp ~uli101/tutorialfiles/words.dat | tee word-search1.txt more~/</span><br><br># View the contents of the '''NOTE:words.dat''' You should NOT see any output due to file using the fact that a <u>space</u> should be included at the end of the word "'''themore'''". Usually words are separated by spaces; thereforecommand and quickly view the contents of this file.<br>Within this file, there were no matches since there were not occurrences of "thethe" as opposed to "'''you should notice some lines that contain repetitions of words. When finished, exit the the'''" (imore command.e. no space after repetition of the pattern).<br<br><br># Reissue Issue the previous following linux pipeline command including a space in brackets:<brto display '''two or more occurrences''' of the word "the":<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the ){2,}" words.dat | tee word-search2search1.txt | more</span><br><br>The "|" (or) symbol can be used within the grouping symbols to allow matching of additional groups of characters.'''NOTE: No output is displayed! Why?'''<br><br>Again, it This is important due to follow the character groupings with the space characterfact that a <bru>space<br/u># Issue should be included at the end of the following linux pipeline command to search for 2 or more occurrences of the word word "'''the'''" .<ubr>or</u> the word "Usually words are separated by spaces; therefore, there were no matches since there were not occurrences<br>of "thethe" as opposed to "'''andthe the'''":<br>(i.e. no space after repetition of the pattern).<br><span style="color:blue;font-weightbr># Reissue the previous pipeline command with the word the followed by a '''space''' within the brackets:<br><span style="color:boldblue;font-weight:bold;font-family:courier;">egrep -i "(the |and ){2,}" words.dat | tee word-search3search2.txt | more</span><br><br># Issue the following Linux command to check that you created those hard links[[Image: eregexps-3.png|thumb|right|330px|Using '''extended''' regular expression symbols (such as '''grouping''') to refine matches of repetition of '''words''' (as opposed to ''characters'').]]The <br><span style="color:blue;font-weight:bold;font-family:courier;">bash /home/murray.saul/scripts/week9-check-3</span><br>style="font-weight:bold;font-family:courier;">"|"<br/span>If you encounter errors, then view (or) symbol (same symbol as "pipe") can be used within the feedback grouping symbols to make correctionsallow matching of additional groups of characters.<br>Again, and then it is important to follow the character groupings with the space character<br><br># Issue the following linux pipeline command to search for '''two or more occurrences''' of the word "'''the '''" <u>or</u> '''two or more occurrences''' of the word "'''and '''":<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the |and ){2,}" words.dat | tee word-search3.txt</span><br><br># Issue the following Linux command to check that you correctly issued<br>those ''Linux pipeline commands'' using the '''tee''' command to create those text files:<br><span style="color:blue;font-weight:bold;font-family:courier;">~uli101/week9-check-3</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.<br>If you receive a congratulation message that there are no errors, then proceed with -run the checking script.<br>If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>Let's issue a Linux '''pipeline''' command using the '''egrep''' command as a '''filter'''<br>using <u>both</u> '''complex''' and '''extended''' regular expressions.<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | egrep "[a-z]{1,}.*[0-9]"</span><br><br>What did this Linux pipeline command display?<br><br>: The '''grep''' and '''egrep''' Linux commands are NOT the only Linux commands that use regular expressions.<br>In the next investigation, you will apply regular expressions to a number of Linux commands<br>that you already learned in this tutorialcourse.<br><br>
:Proceed You can proceed to Investigation INVESTIGATION 3<br><br>
=INVESTIGATION 3: OTHER COMMANDS THAT USE USING REGULAR EXPRESSIONS =<br>In this sectioninvestigation, you will see how regular expressions can be used with other Linux utilities commands other than '''grep''' or '''egrep'''that can use regular expressions.
'''Perform the Following Steps:'''
# Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account.<br><br># Let's look at using regular expressions with the '''man ''' command.<br>Issue the following linux command :<br><span style="color:blue;font-weight:bold;font-family:courier;">man ls</span><br><br>[[Image:other-re-1.png|thumb|right|300px|Entering '''/sort''' in the '''man''' command can search for the string "'''sort'''".]]# We want to search for an option that can sort the file listing.<br>Type the following regular expression below and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/sort</span><br><br>'''FYI:''' The '''grep''' and '''egrep''' linux Linux commands contain the regular expressions within quotes,<br>but '''most ''' other Linux commands specify regular expressions using forward slashes<br>'''forward slashes''' (e.g. '''<span style="font-weight:bold;font-family:courier;">/regular expression</''' span> &nbsp; or '''&nbsp; <span style="font-weight:bold;font-family:courier;">/regular expression'''/</span>).<br><br># Scroll throughout the man pages for the ls command to view matches for the pattern "'''sort'''"<br>(You can press '''SPACE''' and or key combination '''&lt;ALT&gt;&lt;alt-b&gt;''' to move forward and backwards per page one screen respectively).<br><br># Press the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span> to '''exit ''' the ''man '' pages for '''ls'''.<br><br>#Let's use regular expressions with the '''moreless''' command.<br><br># Issue the following linux Linux command to view the contents of the copy another data file called '''textfile1large-file.txt''' (downloaded in a previous section):<br><span style="color:blue;font-weight:bold;font-family:courier;">more textfile1cp ~uli101/tutorialfiles/large-file.txt~/</span><br><br>#We want to search for a pattern [[Image:other-re-2.png|thumb|right|300px|Entering '''/uli101''' in the '''less''' command can display all matches of "'''uli101''' within this " throughout the text file.<br>Type ]]# Issue the following regular expression and press ENTERLinux command to view the contents of the '''large-file.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/uli101less large-file.txt</span><br><br>What did you notice?<br><br>#Search We want to search for the next occurrence of the a pattern '''uli101''' by '''re-typing''' within this text file.<br>Type the following regular expression and pressing press ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">/uli101</span><br><br>you You should now see the '''second occurrence''' of this pattern within "uli101" throughout the text file.<br><br># Press the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span> to exit the '''moreless''' command.<br><br>#Try the same search techniques with the '''more''' command.<br><br>Does it work the same for the ''less'' command?<br><br>[[Image:other-re-3.png|thumb|right|300px|Entering '''/uli101''' in the '''vi''' command can search for the string "'''uli101'''".]]Let's learn how to perform a simple '''search and replace''' within the '''vi''' utility <br>by using regular expressions.<br><br># Issue the following linux Linux command to edit the '''textfile1large-file.txt''' file:<br><span style="color:blue;font-weight:bold;font-family:courier;">vi textfile1large-file.txt</span><br><br>Let's first perform a simple search within this text file.<br><br># Press the '''ESC''' key to make certain you are in '''COMMAND''' mode.<br><br># Type the following and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/uli101</span><br><br>You should move to the '''first occurrence''' of notice the pattern: "'''uli101'''" highlighted for ALL occurrences in this text file.<br><br>Let's '''search ''' for the '''uli101''' pattern, but and '''replace ''' it in capitals (i.e '''ULI101''').<br><br>In vi, in order to perform and issue a command, you need to go into last line mode, and enter '''LAST LINE''' MODE then issue a command to apply to the entire text file, followed by .<br>Let's issue a regular expression command from '''LAST LINE''' MODE to search for and a regular expression replace '''uli101''' to replace (i.e. '''/search/replace/ULI101''').<br><br>[[Image:other-re-4.png|thumb|right|500px|In l'''ast line''' MODE in the '''vi''' text editor, issuing a command using regular expressions to convert '''uli101''' to '''ULI101'''.]]# Type Making certain that you are '''COMMAND''' MODE in vi,<br>type the following and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">:%s/uli101/ULI101/g</span><br><br>You should have noticed that the first occurrence of uli101 has been changed to ULI101.<br><br># Navigate throughout the text file to see if the other occurrences have been replaced.<br><br>You should notice they haven't for ''NOTE:''' The letter '''g''' after the other two occurrences. In order to replace for ALL occurrences, you need to add the letter g (meaning regular expression represents "'''global'''") at the end and will replace ALL occurrences of uli101 in the last forward slash text document (e.g /'''search/replace/g'''as opposed to replacing the first occurrence for every line).<br><br># Making certain that you are command mode in vi, type Type the following (in uppercase letters) and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">:%s/uli101/ULI101/g</span><br><br>You should notice the pattern "'''ULI101'''" highlighted for ALL occurrences in this text file.<br><br># Navigate throughout the text file to confirm that ALL occurrences of '''uli101 ''' have been <u>replaced </u> with '''ULI101'''.<br><br># Save changes to your vi editing session and exit by typing the following and pressing ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">:wxx</span><br><br>: After you complete the Review Questions sections to get additional practice, then work on your<br>'''online assignment 3'''.<br><br>
= LINUX PRACTICE QUESTIONS =
simulate a quiz:
https://ictwiki.cdot.senecacollege.ca/~murray.saululi101/uli101files/uli101_command_practice_9a.docx
Your instructor may take-up these questions during class. It is up to the student to attend classes in order to obtain the answers to the following questions. Your instructor will NOT provide these answers in any other form (eg. e-mail, etc).
'''Part A: Display Results from Linux Commands using Simple &amp; Complex Regular Expressions'''
Note the contents from the following tab-delimited file called '''~murray.saul/uli101/cars''':
<pre>
Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.
1. <span style="font-family:courier;font-weight:bold">grep plym ~murray.saul/uli101/cars</span><br>2. <span style="font-family:courier;font-weight:bold">grep -i fury ~murray.saul/uli101/cars</span><br>3. <span style="font-family:courier;font-weight:bold">grep “^[m-z]” ~murray.saul/uli101/cars</span><br>4. <span style="font-family:courier;font-weight:bold">grep -i “^[m-z]” ~murray.saul/uli101/cars</span><br>5. <span style="font-family:courier;font-weight:bold">grep “3$” ~murray.saul/uli101/cars</span><br>6. <span style="font-family:courier;font-weight:bold">grep -i “c.*5$” ~murray.saul/uli101/cars</span><br>
simulate a quiz:
https://ictwiki.cdot.senecacollege.ca/~murray.saululi101/uli101files/uli101_command_practice_9b.docx
Your instructor may take-up these questions during class. It is up to the student to attend classes in order to obtain the answers to the following questions. Your instructor will NOT provide these answers in any other form (eg. e-mail, etc).
'''Part A: Display Results from Linux Commands using Regular Expressions'''
Note the contents from the following tab-delimited file called '''~murray.saul/uli101/numbers.txt''':
<pre>
Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.
1. <span style="font-family:courier;font-weight:bold">grep "^[-+]" ~murray.saul/uli101/numbers.txt</span><br>2. <span style="font-family:courier;font-weight:bold">grep "^[-+]*.[0-9]" ~murray.saul/uli101/numbers.txt</span><br>3. <span style="font-family:courier;font-weight:bold">grep "^[+-]?[0-9]" ~murray.saul/uli101/numbers.txt</span><br> &nbsp; &nbsp;(Why?)<br>4. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]" ~murray.saul/uli101/numbers.txt</span><br>5. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]+$" ~murray.saul/uli101/numbers.txt</span><br>6. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]+[.]?[0-9]+$" ~murray.saul/uli101/numbers.txt</span><br>
_________________________________________________________________________________ Author: Murray Saul License: LGPL version 3Link: https://www.gnu.org/licenses/lgpl.html _________________________________________________________________________________ 
[[Category:ULI101]]

Navigation menu