Changes

:* Explain Define the purpose of term '''Regular Expressions'''

:* Understand Explain the difference between '''Regular Expressions ''' and '''Filename Expansion'''

:* Understand Explain the purpose of '''Literal (Simple) ''' Regular Expressions

:* Understand and use common symbols for '''Complex ''' Regular Expressions and their purpose

:* Understand and use command symbols for '''Extended ''' Regular Expressions and their purpose

:* Use various List several Linux commands that can use regular expressions<br>

|colspan="2" |Course Notes'''Slides:'''<ul><li>Week 9 Lecture 1 Notes:<br>[https://ictwiki.cdot.senecacollege.ca/~murrayuli101/slides/ULI101-9.1.pdf PDF] | [https://wiki.saulcdot.senecacollege.ca/uli101/slides/ULI101-Week99.1.pptx PPTX]</li><li>Week 9 Lecture 2 Notes:<br> [https://wiki.cdot.senecacollege.ca/uli101/slides/ULI101-9.2.pdf PDF] | [https://ictwiki.cdot.senecacollege.ca/~murray.saululi101/uli101slides/ULI101-Week99.2.pptx PPTX]<br></li></ul>

| style="padding-left:15px;" |'''Regular Expressions:'''* Simple[https://techterms.com/definition/regular_expression#:~:text=A%20regular%20expression%20(or%20%22regex,wildcards%2C%20and%20ranges%20of%20characters.&text=A%20regular%20expression%20can%20be,%2C%20such%20as%20%22app%22. Definition]* Complex* Extended[https://en.wikipedia.org/wiki/Regular_expression#:~:text=Regular%20expressions%20are%20used%20in,built%2Din%20or%20via%20libraries. Purpose (WIKI)]<br><br>

| style="padding-left:15px;"|Additional '''Linux Commands :'''* [https://ss64.com/bash/egrep.html egrep]

|colspan="1" style="padding-left:15px;" width="30%"|'''Brauer Instructional Videos:'''<ul><li>[https://www.youtube.com/watch?v=-2pwLHcvCsU&list=PLU1b1f-2Oe90TuYfifnWulINjMv_Wr16N&index=12 Using grep Command with Regular Expressions]</li></ul>

A '''regular expression''' … is a sequence of characters that define a search pattern.  Usually such patterns are used by string searching algorithms for "find" or "find and replace" operations on strings, or for input validation. <br>Reference: https://en.wikipedia.org/wiki/Regular_expression ===Regular Expressions vs. Filename Expansion===

<table align="right"i><tr valign="top"><td>[[ImageA '''regular expression''' is a combination of two types of characters:re-1.png|thumb|right|350px|Filename expansion symbols allow the Linux shell to expand filenames as arguments (referred to as “globbing”).]]</td><td>[[Image:re-2.png|thumb|right|250px|Output of the '''ls -lRliterals''' and '''special characters''' command to display directory structure.]]</tdbr></table>In a previous lesson, you learned that filename expansion symbols allow the Linux shell Strings of text can be compared to expand filenames as arguments (referred this pattern to as “globbing”) for file management commands. This see if there is very useful for managing multiple files sharing similar characteristics such as the same file extensiona match. </i>

Filename Expansion symbols are used This usually refers to search, edit and manipulate text and are used with Linux file management commands such as that is <u>contained</u> inside a '''ls''', '''rm''', '''mv''', '''cp''', '''cat''', '''less''' and ''file'more'''. This can represent text contained in files or text as a result <br>of issuing Linux commands using a pipeline. A '''regular expression''' is a combination of two types of characters: '''literals''' and '''special charactersLinux pipeline command'''.<br>In combination, these characters define a logical pattern. Strings of text can be compared to this pattern<br>to see if they fit the pattern defined by the expression. Reference: https://www.whoishostingthis.com/resources/regex/

[[Image:re-3.png|thumb|right|250px200px|The simplest A '''simple''' ('''literal''') regular expression is a series of letters and numbers, possibly including white space (tabs or space charactersspaces).]]The simplest regular expression is a series of letters and numbers, possibly including white space (tabs or space charactersspaces), that have no special meaning. Such a <br>A '''simple''' ('''literal''') regular expression consists of ``literals''; that is, normal letterscharacters, which used to match only an identical letter in patterns.<br><br>Although there are many Linux commands that use regular expressions, the data being searched'''grep''' command is a useful command to learn how to display matches of patterns of strings within text files.<br><br>For example:<span style="This is a regular expressioncolor:blue;font-weight:bold;font-family:courier;">grep Linux document.txt<br/span><br>When an editor searches for a literal regular expression, it can only score a ``hit'' if it finds exactly that sequence of characters in the data it is searching. <br>Reference: http://osr507doc.xinuos.com/en/OSUserG/_Literal_chars_in_regexps.html

===Complex / Extended Regular Expressions=== The problem with using literals or simple regular expressions is that general patterns are matched.

For example, the pattern “'''theComplex Regular Expressions'''” would be matched for larger word containing that pattern like <br><br>The problem with just using '''theresimple''', ('''theyliteral''', '''either''', '''them''', '''their''', etc. Another problem ) regular expressions is that you may want to search for pattern at a specific location within the string of text (like at the beginning only <u>simple</u> or end)<u>general</u> patterns are matched.

There are other regular expression tools ''Complex Regular Expressions'' use symbols to provide help match text for more <u>precise matches</u> (complex) patterns. These tools <br>The most common complex regular expression symbols are displayed below:<br><br>:'''Anchors: ''' <span style="color:blue;font-family:courier;font-weight:bold;">^</span> , <span style="color:blue;font-family:courier;font-weight:bold;">$</span><br>Match lines the begin (^) or end ($) with a pattern.<br>:'''Single Character:'''complex&nbsp; <span style="color:blue;font-family:courier;font-weight:bold;">.</span><br>Represents a single character that can be any type of character.<br>:'''Character Class:'' and ' <span style="color:blue;font-family:courier;font-weight:bold;">[ ]</span> , <span style="color:blue;font-family:courier;font-weight:bold;">[^ ]</span><br>Represents a single character but with restrictions.<br>:''extended'Zero or More Occurrence:'' regular expressions' <span style="color:blue;font-family:courier;font-weight:bold;">*</span><br>Zero or more occurrences of previous character. <br><br>

Complex Regular Expressions use symbols to help match text for more precise or complex patterns.The most common :Examples of '''complex regular expression symbols expressions''' are displayed below:

<table align="left"><tr valign="top"><td>[[Image:*re-4.png|thumb|right|200px|Example of using '''Anchorsanchors''' .]]</td><td>[[Image:re-5.png|thumb|right|175px|Example of matching by '''^character(s)''' , '''$'''.]]</td><td>[[Image:*'''Characters re-6.png|thumb|right|220px|Example of using ''':*character class'''Character Class .]]</td><td>[ ]''' , '''[^ ]'''Image:*re-7.png|thumb|right|200px|Example of matching '''Zero zero or More Occurrence *more occurrence of preceding character'''.]]</td></tr></table><br><br><br><br><br><br><br><br><br><br>

'''Extended Regular Expressions consist of additional special characters the “extend” the capability of regular expressions.'''

:'''Grouping ( )Groups:'''<span style="color:blue;font-family:courier;font-weight:bold;">( )</span><br>Perhaps Allows you want to search for repetition for a '''group of characters''', a '''word''', or a '''phase'''.<br>You can enclose them within brackets <span style="font-family:courier;font-weight:bold;">( ) </span> to specify a '''group'''.

:Examples of how to use '''extended regular expressions'''or Condition |with the '''egrep''' command are displayed below:<br><br>Can be used with grouping to match a variety of character(s), words or phases. The | symbol is used to separate the variety of character(s).

Examples of how to use these complex regular expressions with the grep command are displayed below: <table align="left"><tr valign="top"><td>[[Image:re-8.png|thumb|right|280px|Example of using '''repetition'''.]]</td><td>[[Image:re-9.png|thumb|right|250px|Example of using grouping'''groups'''.]]</td><td>[[Image:re-10.png|thumb|right|250px|Example of using '''or ''' condition with grouping'''groups'''.]]</td></tr></table>

<span style="color:red;">'''ATTENTION''': This online tutorial will be required to be completed by '''Friday in week 10 by midnight''' to obtain a grade of '''2%''' towards this course</span><br><br> In this sectioninvestigation, you will learn how to use the '''grep ''' command with '''simple and complex regular expressions '''<br>to help search for ''patterns '' contained in text files.

# '''Login''' to your matrix account.<br><br># Issue a Linux command to '''confirm''' you are located in your '''home ''' directory.<br><br>The # Issue the following linux Linux command to '''wgetcopy''' command is used to download files from the Internet a text file to ''your shell.<br>This will be useful to download text files that we will be using for this tutorial.<br><br># Issue the following linux command ('''copy and paste''' to save time)home''' directory from the ULI101 home directory:<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/cp ~murray.saululi101/uli101tutorialfiles/textfile1.txt<~/nowiki></span><br><br># Issue View the contents of the '''lstextfile1.txt''' command to confirm that the text file was downloaded.<br><br># View the contents of the '''textfile1.txt''' file using file using the '''more''' command and quickly view the contents of see what data is contained in this file. When finished, exit the more command.<br><br>Although there are several Linux commands that use regular expressions, <br>we will only be using the '''grep ''' command for this sectioninvestigation.<br><br>#Issue the following linux pipeline command to match [[Image:regexps-1.png|thumb|right|250px|Output of '''grep''' command matching simple regular expression "'''the pattern the within '''textfile1" (only lowercase).txtNotice the pattern matches larger words like "'''their''':<br><span style="color:blue;font-weight:bold;font-family:courier;or ">grep "the'''them'''" textfile1.txt | more<br><br>]]# Now, issue Issue the grep linux pipeline following Linux command with to match the pattern '''the '''-iwithin '''textfile1.txt''' option to ignore case sensitively:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -i "the" textfile1.txt | more</span><br><br>What do you notice is different with this pipeline command?Take a few moments to view the output and observe the matched patterns.<br><br>You will notice that # Issue the pattern "grep Linux command with the<span style=" is matched including larger words that contain the pattern font-weight:bold;font-family:courier;"the". You can use the >-w i</span> option with the grep command in order to just match only words for a pattern.<br><br># Issue the following linux pipeline commandignore case sensitively:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "the" textfile1.txt | more</span><br><br>What do you notice is different when issuing this command?<br><br>You should now see only strings of text will notice that match the word pattern "'''"the"'''.<br>" is matched including larger words like "'''them'''" and "'''their'''".<br>Just matching literal or simple regular expressions You can be useful, but are limited in what they can assist issue the '''grep''' command with pattern matching.the <brspan style="font-weight:bold;font-family:courier;">For Example, you may want -w</span> option to search for only match the pattern at as a specific location within '''word'''.<br><br># Issue the string of text (like at the beginning or end of the string).following Linux command:<br><br>There are other regular expression tools to provide more precise matchesspan style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "the" textfile1. These tools are '''complextxt</span><br><br>You should now see only strings of text that match the word ''' and '''extendedthe''' (upper or lower case).<br><br>Matching literal or simple regular expressions. We will now look at complex regular expression symbols nowcan be useful, and we will discuss but are ''extended regular expressions'limited'''''Italic text'''' in the next section of this tutorial.<br>in what pattens they can match. For example, you may want to<br># Issue search for a pattern located at the following Linux pipeline command:<br>'''beginning''' or '''end''' of the string.<span style="color:blue;font-weight:bold;font-family:courier;"br>grep -w -i "^the" textfile1.txt | <br>There are other regular expression symbols that provide more</span><br><br>The '''^precise''' symbol is an anchor. In this case, it only matches the <u>word</u> "the" (both upper or lowercase) at the beginning of stringssearch pattern matching.<br>The These special characters are known as '''complex''' and '''$extended''' symbol is used to anchor patterns at the end of stringsregular expressions symbols.<br><br># Issue For the following Linux pipeline command:remainder of this investigation, we will focus on '''complex regular expressions''' and then<br>focus on ''extended regular expressions'' in INVESTIGATION 2.<br><br><span styletable align="color:blue;font-weight:bold;font-family:courier;right"><tr valign="top">grep <td>[[Image:regexps-w -i "the$" textfile12.txt png|thumb|right| more</span><br>280px|Anchoring regular expressions at the '''beginning''' of text.]]<br/td>What do you notice?<brtd><br># [[Image:regexps-3.png|thumb|right|250px|Anchoring regular expressions at the '''ending''' of text.]]</td></tr></table># Issue the following Linux pipeline command to anchor the work "the" simultaneously at the beginning and the end of the string:<:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "^the$" textfile1.txt | more</span><br><br>What do you notice?<br>The '''^''' symbol is referred to as an '''anchor'''.<br>Anchoring patterns at both the In this case, it only matches<ubr>beginning</u> and the word "'''the'''" (both upper or lowercase) at the <u>endingbeginning</u> of strings can greatly assist for more complex search patternsthe string.<br>We will now be demonstrating simultaneous anchoring with other complex regular expressions symbols.<br><br># <br># Issue the following Linux command to match strings that begin with 3 characters:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "^...the$" textfile1.txt | more</span><br><br>What do you notice?The '''$''' symbol is used to anchor patterns at the <u>end</u> of the string.<br><br># Issue the following Linux command to match strings that begin and end with 3 characters:anchor the <bru>word<span style="color:blue;font-weight:bold;font-family:courier;"/u>grep "^...$'''the'''" textfile1.txt | more</spanbr>'''simultaneously''' at the <bru>beginning<br/u>What do you notice?and <bru>end<br/u># Issue of the following command to match strings that begin with 3 digitsstring:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep -w -i "^[0-9][0-9][0-9]" the$" textfile1.txt | more</span><br><br># Issue the following command to match strings that end with 3 uppercase letters:What do you notice?<br><span style="color:blue;font-weight:bold;font-family:courier;"br>Anchoring patterns at both the <u>beginning</u>grep "[A-Z][A-Z][A-Z]$" textfile1.txt | moreand </spanu>ending<br/u>of strings can greatly assist<br># Issue the following command to match strings that consist of only 3 digits:for more '''precise''' search pattern matching.<br><span style="color:blue;font-weight:bold;font-family:courier;"br>We will now be demonstrate the '''effectiveness''' of <u>combining</u><br>grep "^[0-9][0-9][0-9]$" textfile1.txt | more'''anchors''' with <u>other</spanu>complex regular expressions symbols.<br><br>The <table align="right"><tr valign="top"><td>[[Image:regexps-4.png|thumb|right|280px|Anchoring regular expressions using '''period''*'symbols at the '' complex regular expression symbol is often confused with filename expansion. In other words, it does NOT represent zero or more of 'beginning''any character''', but zero or or occurrences of the character that comes before the * symbol.text.]]<br/td><brtd># To demonstration, issue the following command to display zero or more occurrences of the letter x[[Image:regexps-5.png|thumb|right|250px|Anchoring regular expressions using '''period''' symbols simultaneously at the '''beginning''' and '''ending''' of text.]]<br><span /td></tr></table># Issue the following Linux command to match strings that '''begin with 3 characters''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "x*^..." textfile1.txt | more</span><br><br>You will most likely What do you notice most ? Can lines of the file is that contain '''less than 3 characters''' be displayed.?<br><br># Let's issue a Issue the following Linux command to display match strings that contain more than one occurrence of the letter x'''begin <u>and</u> end with 3 characters''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "xx*^...$" textfile1.txt | more</span><br><br>Why did this workWhat do you notice compared to the previous command? because <br><br># Issue the pattern indicates one occurrence of the letter x, followed by zero or MORE occurrences of the letter x.following Linux command to match strings that '''begin with 3 digits''':<br><br>If you combine the complex regular expression symbols .* it will act like zero or more occurrence of any character (like * did in filename expansion)span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9][0-9]" textfile1.txt</span><br><br># Issue What did you notice?<br><br># Issue the following Linux command to match strings begin and that '''end with a number with nothing or anything inbetween3 uppercase letters''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0A-9Z][A-Z].*[0A-9Z]$" textfile1.txt | more</span><br><br>Using simultaneous anchors combined with the <table align="right"><tr valign="top"><td>[[Image:regexps-6.* symbol(s) can help you to refine your search patterns of strings.<br><br># Issue png|thumb|right|220px|Anchoring '''3 digits''' at the following linux pipeline command to display strings that begin with a capital letter, ends with a number, '''beginning''' and contains a capital X somewhere inbetween:'''ending''' of text.]]<br/td><span style="color:blue;font-weight:bold;font-family:courier;"td>grep "^[A[Image:regexps-Z]7.*X.*[0-9]$" textfile1.txt png|thumb|right|250px| more</span><br><br>LetAnchoring '''s look at another series of examples involving 3 alpha-numeric characters''' at the '''beginning''' and '''filteringending''' with numbers so only strings containing valid numbers are displayedof text.]]<br/td><br/tr></table>What type of strings match this pattern?<br><br># First, issue Issue the following linux Linux command to download another data file called numbers1.datmatch strings that '''consist of only 3 digits''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget grep "^[0-9][0-9][0-9]$" textfile1.txt<nowiki/span>https://ict.senecacollege.ca/~murray.saul/uli101/numbers1.dat</nowiki<br></spanbr>What did you notice?<br><br># View the contents of Issue the following Linux command to match strings that '''numbers.datconsist of only 3 alphanumeric digits''' file using the '''more''' command and quickly view the contents of this file. You should notice valid and invalid numbers contained in this file. When finished, exit the more command.<br:<br><span style="color:blue;font-weight:bold;font-family:courier;"><grep "^[a-zA-Z0-9][a-zA-Z0-9][a-zA-Z0-9]$" textfile1.txt</span><br># Issue the following linux command to display only whole numbers:<br>What did you notice?<br><br>The <span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9]*$" numbers1.dat | more</span>complex regular expression symbol is often confused with the "*" '''filename expansion''' symbol.<br><br>You may have noticed that the command In other words, it does not entirely work. You may notice an empty line (which is NOT a whole number). This occurs since the * regular expression symbol represents ZERO represent zero or MORE occurrences more of a number. You can use an additional numeric '''any character class with the * regular expression symbol to search for one ''', but zero or more '''occurrences of a number.'''<br><br># Issue of the character that comes '''before''' the following linux command to display only whole numbers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9]*$" numbers1.dat | more*"</span><br><br>You should see that this workssymbol.<br><br># Issue To demonstrate, issue the following linux Linux command to display whole positive '''zero or negative integersmore occurrences''' of the letter "'''x'''":<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-][0-9][0-9]*$" numbers1.dat | more<x*" textfile1.txt</span><br><br>What did you You will most likely notice?most lines of the file is displayed.<br><br># Issue the following linux Let's issue a Linux command to display only whole numbers (with or without a positive or negative sign):<br>strings that contain '''more than one occurrence''' of the letter "'''x'''":<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]xx*[0-9]*$" numbers1textfile1.dat | moretxt</span><br><br>Why did this work? because the pattern indicates one occurrence of the letter "x",<br>followed by '''zero or MORE occurrences''' of the <u>next</u> letter "x".<br><br>If you combine the complex regular expression symbols <span style="font-weight:bold;font-family:courier;">".*"</span> it will act like<br>zero or more occurrences of <u>any</u> character (i.e. like <span style="font-weight:bold;font-family:courier;">"*"</span> did in filename expansion).<br><br># Issue the following Linux command to match strings begin and end with a number with nothing or anything inbetween:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9].*[0-9]$" textfile1.txt</span><br><br>Using '''simultaneous anchors''' combined with the <span style="font-weight:bold;font-family:courier;">".*"</span> symbol(s) can help you to refine your search patterns of strings.<br><br># Issue the following Linux command to display strings that begin with a capital letter,<br>end with a number, and contains a capital X somewhere inbetween:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[A-Z].*X.*[0-9]$" textfile1.txt</span><br><br>Let's look at another series of examples involving searching for strings that only contain '''valid numbers'''.<br>We will use '''pipeline commands''' to both display stdout to the screen and save to files<br>for confirmation of running these pipeline commands when run a '''checking-script''' later in this investigation.<br><br># Issue the following Linux command to create the '''regexps''' directory: <span style="color:blue;font-weight:bold;font-family:courier;">mkdir ~/regexps</span><br><br># Change to the '''regexps''' directory and confirm that you have moved to this directory.<br><br># First, issue the following Linux command to copy another data file called '''numbers1.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">cp ~uli101/tutorialfiles/numbers1.dat ~/regexps</span><br><br># View the contents of the '''numbers.dat''' file using the '''more''' command and quickly view the contents of this file.<br>You should notice '''valid''' and '''invalid''' numbers contained in this file. When finished, exit the more command.<br><br># Issue the following linux pipeline command to display only '''whole''' numbers (i.e. no '''+''' or '''-''' sign):<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9]*$" numbers1.dat | tee faulty.txt</span><br><br>You may have noticed that the command '''does not entirely work'''. You may notice an '''empty line'''<br>(which is NOT a whole number). This occurs since the * regular expression symbol represents<br>ZERO or MORE occurrences of a number. You can use an additional numeric character class<br>with the * regular expression symbol to search for one or more occurrences of a number.<br><br># Issue the following Linux pipeline command to display only whole numbers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[0-9][0-9]*$" numbers1.dat | tee whole.txt</span><br><br>You should see that this now works.<br><br># Issue the following Linux pipeline command to display <u>only</u> '''signed''' integers:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-][0-9][0-9]*$" numbers1.dat | tee signed.txt</span><br><br>What did you notice? Positive and negative numbers display, not '''unsigned''' numbers.<br><br>[[Image:regexps-8.png|thumb|right|300px|Simultaneous '''anchoring''' of regular expressions using '''character class''' and '''zero or more occurrences''' to display '''signed''' and '''unsigned''' integers.]]# Issue the following Linux pipeline command to display '''signed''' or '''unsigned integers''':<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-]*[0-9][0-9]*$" numbers1.dat | tee all.txt</span><br><br>Did this command work?<br><br># Issue the following command to check that you created those hard links: <br><span style="color:blue;font-weight:bold;font-family:courier;">~uli101/week9-check-1</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script. If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>You can also use the '''grep''' command using ''regular expression'' as a '''filter''' in pipeline commands.<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | grep "[0-9].*dat$"</span><br><br>What did this pipeline display?<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | grep "[a-z].*txt$"</span><br><br>What did this pipeline display?<br><br>

Proceed : Although very useful, '''complex''' regular expressions do NOT <u>entirely</u> solve our problem of displaying<br> '''valid''' unsigned and signed numbers (not to Investigation mention displaying decimal numbers).<br><br>In the next investigation, you will learn how to use '''extended''' regular expressions that will completely solve this issue.<br> : You can proceed to INVESTIGATION 2.<br>

<br>In this sectioninvestigation, you will learn how to use '''extended regular expressions ''' with the '''egrep''' command<br>to help further refine your search when using regular expressionspatterns.

# Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account.<br><br># Issue the following linux Linux command to download copy another data file called numbers2'''numbers2.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/cp ~murray.saululi101/uli101tutorialfiles/numbers2.dat<~/nowiki></span><br><br># View the contents of the '''numbers2.dat''' file using the '''more''' command and quickly view the contents of this file.<br>You should notice ''valid '' and more ''invalid '' numbers contained in this file. When finished, exit the more command.<br><br># Issue the following linux command to display only whole numbers (with or without a positive or negative sign)[[Image:<br><span style="color:blue;font-weight:bold;font-family:courier;">grep "^[+-eregexps-1.png|thumb|right|300px|'''Weakness''' of '''complex''' regular expressions that do not '''limit''' the number of '''positive''' or '''negative''' signs.]*[0-9][0-9]*$" numbers2.dat | more</span># Issue the following Linux command to display '''signed''' or '''unsigned integers''':<br><br>You should notice multiple + or span style="color:blue;font- signs are appearing as well. This occurs since you are searching or one or MORE occurrences of a + or weight:bold;font- sign.<brfamily:courier;">grep "^[+-]*[0-9][0-9]*$" numbers2.dat</span>Using '''extended regular expression''' symbols to specify minimum and maximum repetitions '''{min,max}''' can solve this problem.<br><br># Issue the following linux command (using extended regular expression symbols) to display only whole numbers (with You should notice '''multiple''' '''+''' or without a positive or negative sign):'''-''' '''signs''' appear <u>prior</u> to some numbers.<br><span style="color:blue;fontThis occurs since you are searching or one or MORE occurrences of a + or -weight:bold;font-family:courier;"sign.<br><br>grep "^[+-]{0.1Using '''extended regular expression''' symbols to specify '''minimum''' and '''maximum''' repetitions: '''{min,max}[0-9]{1,}$" numbers2''' can solve that problem.dat | more</span><br><br># Issue the following Linux command (using extended regular expression symbols)<br>to display '''signed''''NOTE:or ''' most likely, there were NO results. This is due to the fact that the grep command was NOT issued correctly to use extended regular expression symbols. You would need to issue either '''grep unsigned integers''':<br><span style="color:blue;font-E''' (or more simply) issue the '''egrep''' command. The egrep command works with all regular expression symbolsweight:bold;font-family:courier;">grep "^[+-]{0,1}[0-9]{1, and should be used in the future instead of the older grep command}$" numbers2.dat</span><br><br># Reissue the above command using '''egrep''' instead of 'NOTE: No output will be displayed! Why?'grep'':<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+br>This is due to the fact that the '''grep command was NOT issued correctly to use extended regular expression symbols'''.<br>You would need to issue either '''grep -]{0E''',1}[0-9]{1,}$" numbers2 or just issue the '''egrep''' command.dat | more</span>The egrep command works with<br><br>You '''all''' regular expression symbols, and should have noticed that the command worked correctly this time because you be used in the '''egrep''' command.future <bru>instead<br/u># Issue of the following linux older grep command to display signed, unsigned, whole, and decimal numbers:.<br><span style="color:blue;fontbr>We will use '''pipeline commands''' to both display stdout to the screen and save to files<br>for confirmation of running these pipeline commands when run a '''checking-weight:bold;font-family:courier;"script''' later in this investigation.<br><br>egrep "^[+-]{0,1}[0-9]{1,}[.]{0,1}[0-9]*$" numbers2.dat | more# Issue the following Linux pipeline command using '''egrep''' instead of ''grep'':</spanbr><brspan style="color:blue;font-weight:bold;font-family:courier;"><br>You can also use extended regular expression symbols for '''grouping'''. For exampleegrep "^[+-]{0, you can search for repetitions of GROUPS of characters1}[0-9]{1,}$" numbers2.dat | tee better-number1.txt<br/span>(like a word) as opposed to just a single character or a GROUP of numbers as opposed to a single digit.<br><br># Issue the following linux command to download another data file called words.dat:<br><span style="color[[Image:blue;fonteregexps-weight:bold;font-family:courier;">wget <nowiki>https://ict2.senecacollege.ca/~murray.saul/uli101/words.dat</nowiki></span><br><br># View the contents of the png|thumb|right|300px|Using '''extended''' regular expression symbols (such as 'numbers2.dat''repetition' file using the '') to refine matches of 'more'signed'' command and quickly view the contents of ''unsigned'' integers.]]You should have noticed that the command worked correctly this filetime because you used the '''egrep''' command.<br>You should notice valid and more invalid numbers contained in this file. When finished, exit the more command.<br><br># Issue '''NOTE:''' With extended regular expressions, the following linux command '''?''' symbol can be used to display two or more occurrences of represent the '''{0,1}''' repetition symbols and the word "the":<'''+''' symbol can be used to represent the '''{1,}''' repetition symbols<br><span style="color:blue;br># Issue the following Linux pipeline command using the repetition shortcuts <span style="font-weight:bold;font-family:courier;">egrep -i "(the){2,}+"</span> and <span style="font-weight:bold;font-family:courier;">"?" numbers2.dat | more</span>:<br><brspan style="color:blue;font-weight:bold;font-family:courier;">You should not see any output due to the fact that a space should be included at the end of the word egrep "the^[+-]?[0-9]+$"numbers2. Usually words are separated by spaces; therefore, there were no matches since there were not occurrences of "thethe" as opposed to "dat | tee better-number2.txt</span><br><br>You should have seen the the"'''same results''', but less typing was required.<br><br># Reissue Issue the previous following Linux pipeline command including a space in brackets:<br><span style="colorto display '''signed''', '''unsigned''', '''whole''', and '''decimal''' numbers:blue<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep "^[+-i "(the )]{20,1}" numbers2.[0-9]{1,}[.]{0,1}[0-9]*$" numbers2.dat | moretee better-number3.txt</span><br><br>The or symbol | can be used within Were all signed and unsigned intergers and decimal numbers displayed?<br><br># Issue the grouping regular expression symbol follwoing command to allow matching of additional groups of characters. Again, it is important to follow the character groupings with the space charactercheck that you correctly issued<br><those ''Linux pipeline commands'': <br># Issue the following linux command to search for 2 or more occurrences of the word <span style="the" or the word "and":<br><span style="colorcolor:blue;font-weight:bold;font-family:courier;">egrep -i "(the |and ){2,}" numbers2~uli101/week9-check-2</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.<br>If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>You can also use extended regular expression symbols for '''grouping'''.<br>For example, you can search for repetitions of GROUPS of characters (like a word)<br>as opposed to just a single character or a GROUP of numbers as opposed to a single digit.<br><br># Issue the following linux pipeline command to copy another data file called '''words.dat''':<br><span style="color:blue;font-weight:bold;font-family:courier;">cp ~uli101/tutorialfiles/words.dat ~/</span><br><br># View the contents of the '''words.dat''' file using the '''more''' command and quickly view the contents of this file.<br>Within this file, you should notice some lines that contain repetitions of words. When finished, exit the more command.<br><br># Issue the following linux pipeline command to display '''two or more occurrences''' of the word "the":<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the){2,}" words.dat | tee word-search1.txt more</span><br><br>'''NOTE: No output is displayed! Why?'''<br><br>This is due to the fact that a <u>space</u> should be included at the end of the word "'''the'''".<br>Usually words are separated by spaces; therefore, there were no matches since there were not occurrences<br>of "thethe" as opposed to "'''the the'''" (i.e. no space after repetition of the pattern).<br><br># Reissue the previous pipeline command with the word the followed by a '''space''' within the brackets:<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the ){2,}" words.dat | tee word-search2.txt</span><br><br>[[Image:eregexps-3.png|thumb|right|330px|Using '''extended''' regular expression symbols (such as '''grouping''') to refine matches of repetition of '''words''' (as opposed to ''characters'').]]The <span style="font-weight:bold;font-family:courier;">"|"</span> (or) symbol (same symbol as "pipe") can be used within the grouping symbols to allow matching of additional groups of characters.<br>Again, it is important to follow the character groupings with the space character<br><br># Issue the following linux pipeline command to search for '''two or more occurrences''' of the word "'''the '''" <u>or</u> '''two or more occurrences''' of the word "'''and '''":<br><span style="color:blue;font-weight:bold;font-family:courier;">egrep -i "(the |and ){2,}" words.dat | tee word-search3.txt</span><br><br># Issue the following Linux command to check that you correctly issued<br>those ''Linux pipeline commands'' using the '''tee''' command to create those text files:<br><span style="color:blue;font-weight:bold;font-family:courier;">~uli101/week9-check-3</span><br><br>If you encounter errors, then view the feedback to make corrections, and then re-run the checking script.<br>If you receive a congratulation message that there are no errors, then proceed with this tutorial.<br><br>Let's issue a Linux '''pipeline''' command using the '''egrep''' command as a '''filter'''<br>using <u>both</u> '''complex''' and '''extended''' regular expressions.<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | egrep "[a-z]{1,}.*[0-9]"</span><br><br>What did this Linux pipeline command display?<br><br>: The '''grep''' and '''egrep''' Linux commands are NOT the only Linux commands that use regular expressions.<br>In the next investigation, you will apply regular expressions to a number of Linux commands<br>that you already learned in this course.dat | more</span><br><br>

Proceed : You can proceed to Investigation INVESTIGATION 3<br><br>

=INVESTIGATION 3: OTHER COMMANDS THAT USE USING REGULAR EXPRESSIONS =<br>In this sectioninvestigation, you will see how regular expressions can be used with commands other Linux utilities than just '''grep ''' or '''egrep''' that can use regular expressions.

# Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account.<br><br># Let's look at using regular expressions with the '''man ''' command.<br>Issue the following linux command :<br><span style="color:blue;font-weight:bold;font-family:courier;">man ls</span><br><br>[[Image:other-re-1.png|thumb|right|300px|Entering '''/sort''' in the '''man''' command can search for the string "'''sort'''".]]# We want to search for an option that can sort the file listing.<br>Type the following regular expression below and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">/sort</span><br><br>'''FYI:''' The '''grep''' and '''egrep''' linux Linux commands contain the regular expressions within quotes, but '''most ''' other Linux commands specify regular expressions using <br>'''forward slashes ''' (e.g. <span style="font-weight:bold;font-family:courier;">/regular expression</ span> &nbsp; or &nbsp; <span style="font-weight:bold;font-family:courier;">/regular expression/</span>).<br><br># Scroll throughout the man pages for the ls command to view matches for the pattern "'''sort'''"<br>(You can press '''SPACE''' or key combination '''alt-b''' to move forward and backwards one screen respectively).<br><br># Press the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span> to '''exit''' the ''man'' pages for '''ls'''.<br><br>Let's use regular expressions with the '''less''' command.<br><br>#Issue the following linux Linux commandto copy another data file called '''large-file.txt''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https:cp ~uli101/tutorialfiles/ictlarge-file.senecacollege.catxt ~/</~murrayspan><br><br>[[Image:other-re-2.saulpng|thumb|right|300px|Entering '''/uli101/textfile1''' in the '''less''' command can display all matches of "'''uli101'''" throughout the text file.]]# Issue the following Linux command to view the contents of the '''large-file.txt''':<br></nowikispan style="color:blue;font-weight:bold;font-family:courier;">less large-file.txt</span><br><br>#Issue We want to search for a pattern '''uli101''' within this text file.<br>Type the following linux commandregular expression and press ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">wget /uli101</span><br><br>You should see the pattern "uli101" throughout the text file.<br><nowikibr>https# Press the letter <span style="color:blue;font-weight:bold;font-family:courier;">q<//ictspan> to exit the '''less''' command.senecacollege<br><br># Try the same search techniques with the '''more''' command.ca/~murray<br><br>Does it work the same for the ''less'' command?<br><br>[[Image:other-re-3.saulpng|thumb|right|300px|Entering '''/uli101/textfile1''' in the '''vi''' command can search for the string "'''uli101'''".]]Let's learn how to perform a simple '''search and replace''' within the '''vi''' utility<br>by using regular expressions.<br><br># Issue the following Linux command to edit the '''large-file.txt''' file:</nowikibr><span style="color:blue;font-weight:bold;font-family:courier;">vi large-file.txt</span><br><br>Let's first perform a simple search within this text file.<br><br># Press the '''ESC''' key to make certain you are in '''COMMAND''' mode.<br><br>#Issue Type the following linux commandand press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget /uli101</span><br><br>You should notice the pattern "'''uli101'''" highlighted for ALL occurrences in this text file.<br><br>Let's '''search''' for the '''uli101''' pattern, and '''replace''' it in capitals (i.e '''ULI101''').<br><br>In vi, to issue a command, you need to enter '''LAST LINE''' MODE then issue a command.<br>Let's issue a command from '''LAST LINE''' MODE to search and replace '''uli101''' to '''ULI101'''.<br><nowikibr>https[[Image://ictother-re-4.senecacollegepng|thumb|right|500px|In l'''ast line''' MODE in the '''vi''' text editor, issuing a command using regular expressions to convert '''uli101''' to '''ULI101'''.ca]]# Making certain that you are '''COMMAND''' MODE in vi,<br>type the following and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">:%s/~murray.saululi101/uli101ULI101/textfile1.txtg</nowikispan><br></spanbr>'''NOTE:''' The letter '''g''' after the replace regular expression represents "'''global'''" and will replace ALL occurrences of uli101 in the text document (as opposed to replacing the first occurrence for every line).<br><br>#Issue Type the following linux command(in uppercase letters) and press '''ENTER''':<br><span style="color:blue;font-weight:bold;font-family:courier;">wget /ULI101</span><nowikibr>https://ict<br>You should notice the pattern "'''ULI101'''" highlighted for ALL occurrences in this text file.senecacollege.ca/~murray.saul/<br><br># Navigate throughout the text file to confirm that ALL occurrences of '''uli101''' have been <u>replaced</textfile1u> with '''ULI101'''.txt</nowikibr><br># Save changes to your vi editing session and exit by typing the following and pressing ENTER:<br><span style="color:blue;font-weight:bold;font-family:courier;">:x</span><br><br>

https://ictwiki.cdot.senecacollege.ca/~murray.saululi101/uli101files/uli101_command_practice_9a.docx

'''Review QuestionsPart A:Display Results from Linux Commands using Simple &amp; Complex Regular Expressions'''

'''Part A: Display Results from Linux Commands using Regular Expressions''' Note the contents from the following tab-delimited file called '''~murray.saul/uli101/cars''':

Plym fury 77 73 2500chevy nova 79 60 3000 ford mustang 65 45 10003volvo gl 78 102 9850 ford ltd 83 15 10507 chevy nova 80 50 3503 fiat 600 65 115 450 honda accord 81 30 6000ford thundbd 84 10 17000toyota tercel 82 180 755chevy impala 65 85 1553ford bronco 83 25 9505

1. <span style="font-family:courier;font-weight:bold">grep plym ~murrayuli101/cars</span><br>2.saul<span style="font-family:courier;font-weight:bold">grep -i fury ~uli101/cars</span><br>3. <span style="font-family:courier;font-weight:bold">grep “^[m-z]” ~uli101/cars</span><br>4. <span style="font-family:courier;font-weight:bold">grep -i “^[m-z]” ~uli101/cars</span><br>5. <span style="font-family:courier;font-weight:bold">grep “3$” ~uli101/cars</span><br>6. <span style="font-family:courier;font-weight:bold">grep -i “c.*5$” ~uli101/cars</span><br>

7. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains the pattern:the<br>8. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains the word:the<br>9. Write a Linux command to display all lines in the file called '''~/text.txt''' that begin with a number.<br>10. Write a Linux command to display all lines in the file called '''~/text.txt''' that end with a letter(either upper or lowercase).<br>11. Write a Linux command to display all lines in the file called '''~/text.txt''' that begin and end with a number.<br>12. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains exactly 3characters that can be anything.<br>13. grep “^[m-z]” Write a Linux command to display all lines in the file called '''~murray/text.saultxt''' that contains exactly 3 numbers.<br>14. Write a Linux command to display all lines in the file called '''~/uli101/carstext.txt''' that contains 1 or more “C” characters.<br>

5. grep “3$” Note the contents from the following tab-delimited file called '''~murray.saul/uli101/carsnumbers.txt''':

61. <span style="font-family:courier;font-weight:bold">grep "^[-i “c+]" ~uli101/numbers.txt</span><br>2.<span style="font-family:courier;font-weight:bold">grep "^[-+]*.[0-9]" ~uli101/numbers.txt</span><br>3. <span style="font-family:courier;font-weight:bold">grep "^[+-]?[0-9]" ~uli101/numbers.txt</span><br> &nbsp; &nbsp;(Why?)<br>4. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]" ~uli101/numbers.txt</span><br>5. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]+$” " ~murrayuli101/numbers.saultxt</span><br>6. <span style="font-family:courier;font-weight:bold">egrep "^[+-]?[0-9]+[.]?[0-9]+$" ~uli101/cars numbers.txt</span><br>

 7. Write a single Linux command to display all lines in the file called ~/text.txt that contains the pattern:the   8. Write a Linux command to display all lines in the file called ~/text.txt that contains perform the word:specified tasks for each of the   9. Write a Linux command to display all lines in the file called ~/text.txt that begin with a number.    10. Write a Linux command to display all lines in the file called ~/text.txt that end with a letter (either upper or lowercase).    11. Write a Linux command to display all lines in the file called ~/text.txt that begin and end with a number.     12. Write a Linux command to display all lines in the file called ~/text.txt that contains exactly 3 characters that can be anythingfollowing questions.  

138. Write a Linux command to display all lines in the file called '''~/textdata.txt ''' that contains exactly ends with 3 numbers.or more occurrences of the number 6

If you wish additional practice using '''extended regular expressions''', here is a link to another MS Word documentLicense:LGPL version 3Link: https://ictwww.senecacollege.ca/~murraygnu.saulorg/uli101licenses/uli101_command_practice_9blgpl.docxhtml