Changes

← Older edit

Tutorial9: Regular Expressions

29,737 bytes added, 20:36, 4 September 2023

no edit summary

{{Admon/caution|DO NOT USE THIS VERSION OF THE LAB. This page will no longer be updated.|'''New version here:''' https://seneca-ictoer.github.io/ULI101/A-Tutorials/tutorial9 '''Andrew's students please go here:''' http://wiki.littlesvr.ca/wiki/OPS145_Lab_8}}

=USING REGULAR EXPRESSIONS=

===Main Objectives of this Practice Tutorial===

:* ~~Explain~~ Define the ~~purpose of~~ term '''Regular Expressions'''

:* ~~Understand~~ Explain the difference between '''Regular Expressions ''' and '''Filename Expansion'''

:* ~~Understand~~ Explain the purpose of '''Literal (Simple) ''' Regular Expressions

:* Understand and use common symbols for '''Complex ''' Regular Expressions and their purpose

:* Understand and use command symbols for '''Extended ''' Regular Expressions and their purpose

:* ~~Use various~~ List several Linux commands that can use regular expressions

===Tutorial Reference Material===

|- valign="top" style="padding-left:15px;"

|colspan="2" |~~Course Notes~~'''Slides:'''<ul><li>Week 9 Lecture 1 Notes: [https://~~ict~~wiki.cdot.senecacollege.ca/~~~murray~~uli101/slides/ULI101-9.1.pdf PDF] | [https://wiki.~~saul~~cdot.senecacollege.ca/uli101/slides/ULI101-~~Week9~~9.1.pptx PPTX]</li><li>Week 9 Lecture 2 Notes: [https://wiki.cdot.senecacollege.ca/uli101/slides/ULI101-9.2.pdf PDF] | [https://~~ict~~wiki.cdot.senecacollege.ca/~~~murray.saul~~uli101/~~uli101~~slides/ULI101-~~Week9~~9.2.pptx PPTX] </li></ul>

| style="padding-left:15px;" |'''Regular Expressions:'''* ~~Simple~~[https://techterms.com/definition/regular_expression#:~:text=A%20regular%20expression%20(or%20%22regex,wildcards%2C%20and%20ranges%20of%20characters.&text=A%20regular%20expression%20can%20be,%2C%20such%20as%20%22app%22. Definition]* ~~Complex~~* Extended[https://en.wikipedia.org/wiki/Regular_expression#:~:text=Regular%20expressions%20are%20used%20in,built%2Din%20or%20via%20libraries. Purpose (WIKI)]

| style="padding-left:15px;"|~~Additional~~ '''Linux Commands ~~that Use Regular Expressions~~:'''* [https://ss64.com/bash/egrep.html egrep]

* [https://www.man7.org/linux/man-pages/man1/man.1.html man]

* [https://man7.org/linux/man-pages/man1/more.1.html more] / [https://www.man7.org/linux/man-pages/man1/less.1.html less]

* [https://man7.org/linux/man-pages/man1/sed.1p.html sed]

* [https://man7.org/linux/man-pages/man1/awk.1p.html awk]

* [https://linux.die.net/man/1/wget wget]

|colspan="1" style="padding-left:15px;" width="30%"|'''Brauer Instructional Videos:'''<ul><li>[https://www.youtube.com/watch?v=-2pwLHcvCsU&list=PLU1b1f-2Oe90TuYfifnWulINjMv_Wr16N&index=12 Using grep Command with Regular Expressions]</li></ul>

|}

= KEY CONCEPTS =

~~A '''regular expression''' … is a sequence of characters that define a search pattern.~~ ===Regular Expressions===

~~Usually such patterns are used by string searching algorithms for "find" or "find~~ A '''regular expression''' is a combination of two types of characters: '''literals''' and ~~replace" operations on strings, or for input validation~~'''special characters'''. ~~Reference: https://en~~Strings of text can be compared to this pattern to see if there is a match.~~wikipedia.org~~</~~wiki/Regular_expression~~i>

This usually refers to text that is contained inside a '''file''' or text as a result of issuing Linux commands using a '''Linux pipeline command'''. ===Literal (Simple) Regular Expressions ~~vs. Filename Expansion~~===

~~<table align="right"><tr valign="top"><td>~~[[Image:re-13.png|thumb|right|~~350px~~200px|~~Filename expansion symbols allow the Linux shell to expand filenames as arguments~~ A '''simple''' ('''literal''') regular expression is a series of letters and numbers (~~referred to as “globbing”~~tabs or spaces).]]The simplest regular expression is a series of letters and numbers, (tabs or spaces).<~~/td~~br>A '''simple''' ('''literal''') regular expression consists of normal characters, which used to match patterns. <tdbr>~~[[Image:re-2.png|thumb|right|250px|Output of~~ Although there are many Linux commands that use regular expressions, the '''~~ls -lR~~grep''' command is a useful command to learn how to display ~~directory structure~~matches of patterns of strings within text files.]]<~~/td~~br><~~/table~~br>For example:~~In a previous lesson, you learned that filename expansion symbols allow the~~ grep Linux shell to expand filenames as arguments (referred to as “globbing”) for file management commands. This is very useful for managing multiple files sharing similar characteristics such as the same file extensiondocument. txt

Filename Expansion symbols are used to search, edit and manipulate text and are used with Linux file management commands such as '''ls''', '''rm''', '''mv''', '''cp''', '''cat''', '''less''' and '''more'''. This can represent text contained in files or text as a result of issuing Linux commands using a pipeline.=== Complex / Extended Regular Expressions ===

A '''~~regular expression~~Complex Regular Expressions''' ~~is a combination of two types of characters:~~ The problem with just using '''~~literals~~simple''' ~~and~~ ('''~~special characters~~literal'''.) regular expressions is that only simple<br/u> or ~~In combination, these characters define a logical pattern. Strings of text can be compared to this pattern~~general<br/u>~~to see if they fit the pattern defined by the expression~~patterns are matched.

~~Reference~~''Complex Regular Expressions'' use symbols to help match text for more precise (complex) patterns. The most common complex regular expression symbols are displayed below: :'''Anchors: ''' ^ , $</~~www~~span> Match lines the begin (^) or end ($) with a pattern.~~whoishostingthis~~ :'''Single Character:'''   .~~com~~</~~resources~~span> Represents a single character that can be any type of character. :'''Character Class:''' [ ]</~~regex~~span> , [^ ] Represents a single character but with restrictions. :'''Zero or More Occurrence:''' * Zero or more occurrences of previous character.

~~===Literal (Simple) Regular Expressions===~~:Examples of '''complex regular expressions''' are displayed below:

<table align="left"><tr valign="top"><td>[[Image:re-4.png|thumb|right|200px|Example of using '''anchors'''.]]</td><td>[[Image:re-35.png|thumb|right|~~250px~~175px|~~The simplest regular expression is a series~~ Example of ~~letters and numbers, possibly including white space~~ matching by '''character(~~tabs or space characters~~s)'''.]]~~The simplest regular expression is a series~~ </td><td>[[Image:re-6.png|thumb|right|220px|Example of ~~letters and numbers, possibly including white space (tabs~~ using '''character class'''.]]</td><td>[[Image:re-7.png|thumb|right|200px|Example of matching '''zero or ~~space characters), that have no special meaning. Such a regular expression consists~~ more occurrence of ~~``literals~~preceding character'''~~; that is, normal letters, which match only an identical letter in the data being searched~~.]]</td></tr></table> ~~For example:"This is a regular expression"~~ ~~When an editor searches for a literal regular expression, it can only score a ``hit'' if it finds exactly that sequence of characters in the data it is searching.~~ ~~Reference: http://osr507doc.xinuos.com/en/OSUserG/_Literal_chars_in_regexps.html~~

~~===Complex Regular Expressions===~~

~~The problem with using literals~~ '''Extended Regular Expressions''' ''Extended Regular Expressions'' consist of additional special characters to “extend” the capability of regular expressions. You must use the '''egrep''' or ~~simple~~ '''grep -E''' commands in order to properly use extended regular expressions . :'''Repetition:''' {min,max} Allows for more precise repetitions. Using braces, you can specify the '''minimum''' and/or '''maximum''' number of repetitions. :'''Groups:''' ( ) Allows you to search for repetition for a '''group of characters''', a '''word''', or a '''phase'''. You enclose them within brackets ( ) to specify a '''group'''. :'''or Condition:''' | Can be used with '''groups''' to match a variety of character(s), words or phases. The | symbol is ~~that general patterns are matched~~used to separate the variety of character(s) within a ''group''.

~~For example, the pattern “~~:Examples of how to use '''~~the~~extended regular expressions'''~~” would be matched for larger word containing that pattern like~~ with the '''~~there~~egrep'''~~, '''they''', '''either''', '''them''', '''their''', etc. Another problem is that you may want to search for pattern at a specific location within the string of text (like at the beginning or end).~~command are displayed below:

~~Complex Regular Expressions use symbols to help match text for more precise or complex patterns.The most common complex regular expression symbols are displayed below~~=INVESTIGATION 1:SIMPLE & COMPLEX REGULAR EXPRESSIONS=

'''~~Anchors~~ATTENTION''' ~~'''^''' , '''$''':*'''Characters .'''~~:*This online tutorial will be required to be completed by '''~~Character Class [ ]~~Friday in week 10 by midnight''' ~~, '''[^ ]'''~~:*to obtain a grade of '''Zero or More Occurrence *2%'''towards this course

~~Examples of~~ In this investigation, you will learn how to use ~~these~~ the '''grep''' command with '''simple and complex regular expressions ~~with the grep command are displayed below:~~''' to help search for ''patterns'' contained in text files.

~~ ~~

~~===Extended Regular Expressions===~~ ~~Extended Regular Expressions consist of additional special characters~~ '''Perform the ~~“extend” the capability of regular expressions.~~Following Steps:'''

# '''Login''' to your matrix account.

# Issue a Linux command to '''confirm''' you are located in your '''home''' directory.

# Issue the following linux Linux command to '''copy''' a text file to ''your'' '''home''' directory from the ULI101 home directory: cp ~uli101/tutorialfiles/textfile1.txt ~/

# View the contents of the '''textfile1.txt''' file using the '''more''' command see what data is contained in this file. Although there are several Linux commands that use regular expressions, we will be using the '''grep''' command for this investigation. [[Image:regexps-1.png|thumb|right|250px|Output of '''grep''' command matching simple regular expression "'''the'''" (only lowercase). Notice the pattern matches larger words like "'''their'''" or "'''them'''".]]

#Issue the following Linux command to match the pattern '''the''' within '''textfile1.txt''': grep "the" textfile1.txt Take a few moments to view the output and observe the matched patterns.

# Issue the grep Linux command with the -i option to ignore case sensitively: grep -i "the" textfile1.txt What do you notice is different when issuing this command? You will notice that the pattern "'''the'''" is matched including larger words like "'''them'''" and "'''their'''". You can issue the '''grep''' command with the -w option to only match the pattern as a '''word'''.

# Issue the following Linux command: grep -w -i "the" textfile1.txt You should now see only strings of text that match the word '''the''' (upper or lower case). Matching literal or simple regular expressions can be useful, but are '''limited''' in what pattens they can match. For example, you may want to search for a pattern located at the '''beginning''' or '''end''' of the string. There are other regular expression symbols that provide more '''precise''' search pattern matching. These special characters are known as '''complex''' and '''extended''' regular expressions symbols. For the remainder of this investigation, we will focus on '''complex regular expressions''' and then focus on ''extended regular expressions'' in INVESTIGATION 2. <table align="right"><tr valign="top"><td>[[Image:regexps-2.png|thumb|right|280px|Anchoring regular expressions at the '''beginning''' of text.]]</td><td>[[Image:regexps-3.png|thumb|right|250px|Anchoring regular expressions at the '''ending''' of text.]]</td></tr></table>

# Issue the following Linux command: grep -w -i "^the" textfile1.txt The '''^''' symbol is referred to as an '''anchor'''. In this case, it only matches the word "'''the'''" (both upper or lowercase) at the beginning of the string.

# Issue the following Linux command: grep -w -i "the$" textfile1.txt The '''$''' symbol is used to anchor patterns at the end of the string.

# Issue the following Linux command to anchor the word "'''the'''" '''simultaneously''' at the beginning and end of the string: grep -w -i "^the$" textfile1.txt What do you notice? Anchoring patterns at both the beginning and ending of strings can greatly assist for more '''precise''' search pattern matching. We will now be demonstrate the '''effectiveness''' of combining '''anchors''' with other complex regular expressions symbols. <table align="right"><tr valign="top"><td>[[Image:regexps-4.png|thumb|right|280px|Anchoring regular expressions using '''period''' symbols at the '''beginning''' of text.]]</td><td>[[Image:regexps-5.png|thumb|right|250px|Anchoring regular expressions using '''period''' symbols simultaneously at the '''beginning''' and '''ending''' of text.]]</td></tr></table>

# Issue the following Linux command to match strings that '''begin with 3 characters''': grep "^..." textfile1.txt What do you notice? Can lines that contain '''less than 3 characters''' be displayed?

# Issue the following Linux command to match strings that '''begin and end with 3 characters''': grep "^...$" textfile1.txt What do you notice compared to the previous command?

# Issue the following Linux command to match strings that '''begin with 3 digits''': grep "^[0-9][0-9][0-9]" textfile1.txt What did you notice?

# Issue the following Linux command to match strings that '''end with 3 uppercase letters''': grep "[A-Z][A-Z][A-Z]$" textfile1.txt <table align="right"><tr valign="top"><td>[[Image:regexps-6.png|thumb|right|220px|Anchoring '''3 digits''' at the '''beginning''' and '''ending''' of text.]]</td><td>[[Image:regexps-7.png|thumb|right|250px|Anchoring '''3 alpha-numeric characters''' at the '''beginning''' and '''ending''' of text.]]</td></tr></table>What type of strings match this pattern?

# Issue the following Linux command to match strings that '''consist of only 3 digits''': grep "^[0-9][0-9][0-9]$" textfile1.txt What did you notice?

# Issue the following Linux command to match strings that '''consist of only 3 alphanumeric digits''': grep "^[a-zA-Z0-9][a-zA-Z0-9][a-zA-Z0-9]$" textfile1.txt What did you notice? The "*" complex regular expression symbol is often confused with the "*" '''filename expansion''' symbol. In other words, it does NOT represent zero or more of '''any character''', but zero or more '''occurrences''' of the character that comes '''before''' the "*" symbol.

# To demonstrate, issue the following Linux command to display '''zero or more occurrences''' of the letter "'''x'''": grep "x*" textfile1.txt You will most likely notice most lines of the file is displayed.

# Let's issue a Linux command to display strings that contain '''more than one occurrence''' of the letter "'''x'''": grep "xx*" textfile1.txt Why did this work? because the pattern indicates one occurrence of the letter "x", followed by '''zero or MORE occurrences''' of the next letter "x". If you combine the complex regular expression symbols ".*" it will act like zero or more occurrences of any character (i.e. like "*" did in filename expansion).

# Issue the following Linux command to match strings begin and end with a number with nothing or anything inbetween: grep "^[0-9].*[0-9]$" textfile1.txt Using '''simultaneous anchors''' combined with the ".*" symbol(s) can help you to refine your search patterns of strings.

# Issue the following Linux command to display strings that begin with a capital letter, end with a number, and contains a capital X somewhere inbetween: grep "^[A-Z].*X.*[0-9]$" textfile1.txt Let's look at another series of examples involving searching for strings that only contain '''valid numbers'''. We will use '''pipeline commands''' to both display stdout to the screen and save to files for confirmation of running these pipeline commands when run a '''checking-script''' later in this investigation.

# Issue the following Linux command to create the '''regexps''' directory: mkdir ~/regexps

# Change to the '''regexps''' directory and confirm that you have moved to this directory.

# First, issue the following Linux command to copy another data file called '''numbers1.dat''': cp ~uli101/tutorialfiles/numbers1.dat ~/regexps

# View the contents of the '''numbers.dat''' file using the '''more''' command and quickly view the contents of this file. You should notice '''valid''' and '''invalid''' numbers contained in this file. When finished, exit the more command.

# Issue the following linux pipeline command to display only '''whole''' numbers (i.e. no '''+''' or '''-''' sign): grep "^[0-9]*$" numbers1.dat | tee faulty.txt You may have noticed that the command '''does not entirely work'''. You may notice an '''empty line''' (which is NOT a whole number). This occurs since the * regular expression symbol represents ZERO or MORE occurrences of a number. You can use an additional numeric character class with the * regular expression symbol to search for one or more occurrences of a number.

# Issue the following Linux pipeline command to display only whole numbers: grep "^[0-9][0-9]*$" numbers1.dat | tee whole.txt You should see that this now works.

# Issue the following Linux pipeline command to display only '''signed''' integers: grep "^[+-][0-9][0-9]*$" numbers1.dat | tee signed.txt What did you notice? Positive and negative numbers display, not '''unsigned''' numbers. [[Image:regexps-8.png|thumb|right|300px|Simultaneous '''anchoring''' of regular expressions using '''character class''' and '''zero or more occurrences''' to display '''signed''' and '''unsigned''' integers.]]

# Issue the following Linux pipeline command to display '''signed''' or '''unsigned integers''': grep "^[+-]*[0-9][0-9]*$" numbers1.dat | tee all.txt Did this command work?

# Issue the following command to check that you created those hard links: ~uli101/week9-check-1 If you encounter errors, then view the feedback to make corrections, and then re-run the checking script. If you receive a congratulation message that there are no errors, then proceed with this tutorial. You can also use the '''grep''' command using ''regular expression'' as a '''filter''' in pipeline commands.

# Issue the following Linux pipeline command: ls | grep "[0-9].*dat$" What did this pipeline display?

# Issue the following Linux pipeline command: ls | grep "[a-z].*txt$" What did this pipeline display?

: Although very useful, '''~~Repetition {min,max}~~complex''' , regular expressions do NOT entirely solve our problem of displaying '''?valid''' unsigned and signed numbers (not to mention displaying decimal numbers). In the next investigation, you will learn how to use '''+extended'''regular expressions that will completely solve this issue. ~~Allows for more precise repetitions. Using braces, you can specify the minimum and maximum number of repetitions.~~

: You can proceed to INVESTIGATION 2.

~~'''Grouping ( )''' Perhaps you want to search for repetition for a group of characters, a word, or a phase. You can enclose them within brackets ( ) to specify a group.~~=INVESTIGATION 2: EXTENDED REGULAR EXPRESSIONS =

In this investigation, you will learn how to use '''extended regular expressions''' with the '''egrep''' command to further refine your search patterns.

~~'''or Condition |''' ~~

~~Can be used with grouping to match a variety of character(s), words or phases. The | symbol is used to separate the variety of character(s).~~

~~Examples of how to use these complex regular expressions with~~ '''Perform the ~~grep command are displayed below~~Following Steps:'''

# Make certain that you are located in your '''~/regexps''' directory on your ''Matrix'' account. # Issue the following Linux command to copy another data file called '''numbers2.dat''':<~~table align~~br>cp ~uli101/tutorialfiles/numbers2.dat ~/ <~~tr valign="top"~~br># View the contents of the '''numbers2.dat''' file using the '''more''' command and quickly view the contents of this file. You should notice ''valid'' and ''invalid'' numbers contained in this file. When finished, exit the more command. <tdbr>[[Image:reeregexps-81.png|thumb|right|~~280px~~300px|~~Example~~ '''Weakness''' of '''complex''' regular expressions that do not '''limit''' the number of '''positive''' or '''negative''' signs.]]# Issue the following Linux command to display '''signed''' or '''unsigned integers''': grep "^[+-]*[0-9][0-9]*$" numbers2.dat You should notice '''multiple''' '''+''' or '''-''' '''signs''' appear prior to some numbers. This occurs since you are searching or one or MORE occurrences of a + or - sign. Using '''extended regular expression''' symbols to specify '''minimum''' and '''maximum''' repetitions: '''{min,max}''' can solve that problem. # Issue the following Linux command (using ~~repetition~~extended regular expression symbols) to display '''signed''' or '''unsigned integers''': grep "^[+-]{0,1}[0-9]{1,}$" numbers2.dat '''NOTE: No output will be displayed! Why?''' This is due to the fact that the '''grep command was NOT issued correctly to use extended regular expression symbols'''. You would need to issue either '''grep -E''', or just issue the '''egrep''' command. The egrep command works with '''all''' regular expression symbols, and should be used in the future instead of the older grep command. We will use '''pipeline commands''' to both display stdout to the screen and save to files for confirmation of running these pipeline commands when run a '''checking-script''' later in this investigation. # Issue the following Linux pipeline command using '''egrep''' instead of ''grep'': egrep "^[+-]{0,1}[0-9]{1,}$" numbers2.dat | tee better-number1.txt</tdspan> <tdbr>[[Image:reeregexps-92.png|thumb|right|~~250px~~300px|~~Example~~ Using '''extended''' regular expression symbols (such as '''repetition''') to refine matches of ''signed'' and ''unsigned'' integers.]]You should have noticed that the command worked correctly this time because you used the '''egrep''' command. '''NOTE:''' With extended regular expressions, the '''?''' symbol can be used to represent the '''{0,1}''' repetition symbols and the '''+''' symbol can be used to represent the '''{1,}''' repetition symbols # Issue the following Linux pipeline command using ~~grouping~~the repetition shortcuts "+" and "?": egrep "^[+-]?[0-9]+$" numbers2.dat | tee better-number2.txt You should have seen the '''same results''', but less typing was required. # Issue the following Linux pipeline command to display '''signed''', '''unsigned''', '''whole''', and '''decimal''' numbers: egrep "^[+-]{0,1}[0-9]{1,}[.]{0,1}[0-9]*$" numbers2.dat | tee better-number3.txt Were all signed and unsigned intergers and decimal numbers displayed? # Issue the follwoing command to check that you correctly issued those ''Linux pipeline commands'': ~uli101/week9-check-2 If you encounter errors, then view the feedback to make corrections, and then re-run the checking script. If you receive a congratulation message that there are no errors, then proceed with this tutorial. You can also use extended regular expression symbols for '''grouping'''. For example, you can search for repetitions of GROUPS of characters (like a word) as opposed to just a single character or a GROUP of numbers as opposed to a single digit. # Issue the following linux pipeline command to copy another data file called '''words.dat''': cp ~uli101/tutorialfiles/words.dat ~/ # View the contents of the '''words.dat''' file using the '''more''' command and quickly view the contents of this file. Within this file, you should notice some lines that contain repetitions of words. When finished, exit the more command. # Issue the following linux pipeline command to display '''two or more occurrences''' of the word "the": egrep -i "(the){2,}" words.dat | tee word-search1.txt more '''NOTE: No output is displayed! Why?''' This is due to the fact that a space should be included at the end of the word "'''the'''". Usually words are separated by spaces; therefore, there were no matches since there were not occurrences of "thethe" as opposed to "'''the the'''" (i.e. no space after repetition of the pattern). # Reissue the previous pipeline command with the word the followed by a '''space''' within the brackets: egrep -i "(the ){2,}" words.dat | tee word-search2.txt</tdspan><tdbr> [[Image:reeregexps-103.png|thumb|right|~~250px~~330px|~~Example~~ Using '''extended''' regular expression symbols (such as '''grouping''') to refine matches of repetition of ~~using or condition with grouping~~'''words''' (as opposed to ''characters'').]]The "|"</tdspan> (or) symbol (same symbol as "pipe") can be used within the grouping symbols to allow matching of additional groups of characters. Again, it is important to follow the character groupings with the space character # Issue the following linux pipeline command to search for '''two or more occurrences''' of the word "'''the '''" or</tru> '''two or more occurrences''' of the word "'''and '''": egrep -i "(the |and ){2,}" words.dat | tee word-search3.txt</~~table~~span> # Issue the following Linux command to check that you correctly issued those ''Linux pipeline commands'' using the '''tee''' command to create those text files: ~uli101/week9-check-3 If you encounter errors, then view the feedback to make corrections, and then re-run the checking script. If you receive a congratulation message that there are no errors, then proceed with this tutorial. Let's issue a Linux '''pipeline''' command using the '''egrep''' command as a '''filter''' using both '''complex''' and '''extended''' regular expressions. # Issue the following Linux pipeline command: <brspan style="color:blue;font-weight:bold;font-family:courier;">ls | egrep "[a-z]{1,}.*[0-9]" What did this Linux pipeline command display? : The '''grep''' and '''egrep''' Linux commands are NOT the only Linux commands that use regular expressions. In the next investigation, you will apply regular expressions to a number of Linux commands that you already learned in this course.

=: You can proceed to INVESTIGATION ~~1: SIMPLE & COMPLEX REGULAR EXPRESSIONS=~~3

=INVESTIGATION 3: OTHER COMMANDS USING REGULAR EXPRESSIONS =

In this ~~section~~investigation, you will ~~learn how to~~ see commands other than '''grep''' or '''egrep''' that can use ~~the grep command with simple and complex~~ regular expressions ~~to help search for patterns contained in text files~~.

'''Perform the Following Steps:'''

# Make certain that you are located in your '''~~Login~~~/regexps''' directory on your ~~matrix~~ ''Matrix'' account. # Let's look at using regular expressions with the '''man''' command. Issue a the following linux command to : man ls [[Image:other-re-1.png|thumb|right|300px|Entering '''~~confirm~~/sort''' ~~you are located~~ in ~~your home directory~~the '''man''' command can search for the string "'''sort'''".]]# We want to search for an option that can sort the file listing. Type the following regular expression below and press '''ENTER''': /sort '''FYI:''' The '''~~wget~~grep''' and '''egrep''' Linux commands contain the regular expressions within quotes, but '''most''' other Linux commands specify regular expressions using '''forward slashes''' (e.g. /regular expression   or   /regular expression/). # Scroll throughout the man pages for the ls command ~~is used~~ to ~~download files from~~ view matches for the ~~Internet~~ pattern "'''sort'''" (You can press '''SPACE''' or key combination '''alt-b''' to ~~your shell~~move forward and backwards one screen respectively). ~~This will be useful~~ # Press the letter q to ~~download text files that we will be using~~ '''exit''' the ''man'' pages for ~~this tutorial~~'''ls'''. Let's use regular expressions with the '''less''' command. # Issue the following ~~linux~~ Linux command (to copy another data file called '''~~copy and paste~~large-file.txt''' ~~to save time)~~: ~~wget~~ cp ~uli101/tutorialfiles/large-file.txt ~/<~~nowiki~~br>~~https~~ [[Image:other-re-2.png|thumb|right|300px|Entering '''/~~/ict~~uli101''' in the '''less''' command can display all matches of "'''uli101'''" throughout the text file.~~senecacollege~~]]# Issue the following Linux command to view the contents of the '''large-file.catxt''': less large-file.txt</~~~murray~~span> #We want to search for a pattern '''uli101''' within this text file.~~saul~~ Type the following regular expression and press ENTER: /uli101</~~textfile1~~span> You should see the pattern "uli101" throughout the text file.~~txt~~<~~/nowiki~~br> # Press the letter qto exit the '''less''' command. # ~~Issue~~ Try the same search techniques with the '''more''' command. Does it work the same for the ''less'' command? [[Image:other-re-3.png|thumb|right|300px|Entering '''/uli101''' in the '''lsvi''' command can search for the string "'''uli101'''".]]Let's learn how to ~~confirm that~~ perform a simple '''search and replace''' within the ~~text file was downloaded~~'''vi''' utility by using regular expressions. # ~~View~~ Issue the ~~contents of~~ following Linux command to edit the '''~~textfile1~~large-file.txt''' file ~~using~~ : vi large-file.txt Let's first perform a simple search within this text file. # Press the '''~~more~~ESC''' key to make certain you are in '''COMMAND''' ~~command~~ mode. # Type the following and ~~quickly view~~ press '''ENTER''': /uli101 You should notice the ~~contents of~~ pattern "'''uli101'''" highlighted for ALL occurrences in this text file. ~~When finished~~ Let's '''search''' for the '''uli101''' pattern, and '''replace''' it in capitals (i.e '''ULI101'''). In vi, to issue a command, ~~exit the more~~ you need to enter '''LAST LINE''' MODE then issue a command. Let's issue a commandfrom '''LAST LINE''' MODE to search and replace '''uli101''' to '''ULI101'''. ~~Although there are several Linux commands that use~~ [[Image:other-re-4.png|thumb|right|500px|In l'''ast line''' MODE in the '''vi''' text editor, issuing a command using regular expressionsto convert '''uli101''' to '''ULI101'''.]]# Making certain that you are '''COMMAND''' MODE in vi, we type the following and press '''ENTER''': :%s/uli101/ULI101/g '''NOTE:''' The letter '''g''' after the replace regular expression represents "'''global'''" and will ~~only be using~~ replace ALL occurrences of uli101 in the text document (as opposed to replacing the ~~grep command~~ first occurrence for ~~this section~~every line). #~~Issue~~ Type the following ~~linux pipeline command to match the pattern the within~~ (in uppercase letters) and press '''~~textfile1.txt~~ENTER''': ~~grep~~ /ULI101 You should notice the pattern "~~the~~'''ULI101'''" ~~textfile1~~highlighted for ALL occurrences in this text file.~~txt | more~~ # ~~Now, issue~~ Navigate throughout the ~~grep linux pipeline command~~ text file to confirm that ALL occurrences of '''uli101''' have been replaced with ~~the~~ '''-iULI101''' ~~option~~ . # Save changes to ~~ignore case sensitively~~your vi editing session and exit by typing the following and pressing ENTER: ~~grep -i "the" textfile1.txt | more~~:x ~~What do you notice~~ = LINUX PRACTICE QUESTIONS = The purpose of this section is ~~different~~ to obtain '''extra practice''' to help with ~~this pipeline command?~~'''quizzes''', your '''midterm''', and your '''final exam'''. #x==REVIEW QUESTIONS: SIMPLE & COMPLEX REGULAR EXPRESSIONS== Here is a link to the MS Word Document of ALL of the questions displayed below but with extra room to answer on the document tosimulate a quiz:

~~=INVESTIGATION 2~~https: ~~EXTENDED REGULAR EXPRESSIONS =~~//wiki.cdot.senecacollege.ca/uli101/files/uli101_command_practice_9a.docx

~~In this section, you will learn how~~ Your instructor may take-up these questions during class. It is up to the student to attend classes in order to obtain the answers to the following questions.Your instructor will NOT provide these answers in any other form (eg.e-mail, etc).

'''Part A: Display Results from Linux Commands using Simple & Complex Regular Expressions'''

Note the contents from the following tab-delimited file called '''~~Perform the Following Steps:~~~uli101/cars''':

~~# x~~<brpre>Plym fury 77 73 2500chevy nova 79 60 3000 ford mustang 65 45 10003volvo gl 78 102 9850 ford ltd 83 15 10507 chevy nova 80 50 3503 fiat 600 65 115 450 honda accord 81 30 6000ford thundbd 84 10 17000toyota tercel 82 180 755chevy impala 65 85 1553ford bronco 83 25 9505<br/pre>

In Write the ~~next investigation, you will ..~~results of each of the following Linux commands using regular expressions for the above-mentioned file.

1. grep plym ~uli101/cars 2. grep -i fury ~uli101/cars 3. grep “^[m-z]” ~uli101/cars 4. grep -i “^[m-z]” ~uli101/cars 5. grep “3$” ~uli101/cars 6. grep -i “c.*5$” ~uli101/cars

~~In this section, you will learn how to ...~~

'''Part B: Writing Linux Commands Using Regular Expressions'''

~~'''Perform~~ Write a single Linux command to perform the ~~Following Steps:'''~~specified tasks for each of the following questions.

~~# x~~7. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains the pattern:the 8. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains the word:the 9. Write a Linux command to display all lines in the file called '''~/text.txt''' that begin with a number. 10. Write a Linux command to display all lines in the file called '''~/text.txt''' that end with a letter(either upper or lowercase). 11. Write a Linux command to display all lines in the file called '''~/text.txt''' that begin and end with a number. 12. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains exactly 3 characters that can be anything. 13. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains exactly 3 numbers. 14. Write a Linux command to display all lines in the file called '''~/text.txt''' that contains 1 or more “C” characters.

~~= LINUX PRACTICE QUESTIONS =~~

~~The purpose of this section is to obtain '''extra practice''' to help with '''quizzes''', your '''midterm''', and your '''final exam'''.~~==REVIEW QUESTIONS: REGULAR EXPRESSIONS (INCLUDING EXTENDED REGULAR EXPRESSIONS)==

Here is a link to the MS Word Document of ALL of the questions displayed below but with extra room to answer on the document to

simulate a quiz:

https://~~ict~~wiki.cdot.senecacollege.ca/~~~murray.saul~~uli101/~~uli101~~files/~~uli101_week9_practice~~uli101_command_practice_9b.docx

Your instructor may take-up these questions during class. It is up to the student to attend classes in order to obtain the answers to the following questions. Your instructor will NOT provide these answers in any other form (eg. e-mail, etc).

'''Part A: Display Results from Linux Commands using Regular Expressions'''

Note the contents from the following tab-delimited file called '''~uli101/numbers.txt''':

<pre>

+123

---34

+++++++++++17

-45

45p8

25.6

11

</pre>

Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.

1. grep "^[-+]" ~uli101/numbers.txt

2. grep "^[-+]*.[0-9]" ~uli101/numbers.txt

3. grep "^[+-]?[0-9]" ~uli101/numbers.txt    (Why?)

4. egrep "^[+-]?[0-9]" ~uli101/numbers.txt

5. egrep "^[+-]?[0-9]+$" ~uli101/numbers.txt

6. egrep "^[+-]?[0-9]+[.]?[0-9]+$" ~uli101/numbers.txt

'''Part B: Writing Linux Commands Using Regular Expressions'''

Write a single Linux command to perform the specified tasks for each of the following questions.

7. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with 1 or more occurrences of an UPPERCASE letter.

8. Write a Linux command to display all lines in the file called '''~/data.txt''' that ends with 3 or more occurrences of the number 6

9. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with 2 or more occurrences of the word “the” (upper or lower case).

10. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with 2 or more occurrences       of the word “the” or the word “but” (upper or lower case).

11. Write a Linux command to display all lines in the file called '''~/data.txt''' that begins with a minimum of 2 occurrences       and a maximum of 4 occurrences of the word “the” or the word “but” (upper or lower case).

_________________________________________________________________________________

Author: Murray Saul

~~'''Review Questions~~License:~~'''~~LGPL version 3Link: https://www.gnu.org/licenses/lgpl.html

~~# x# x~~ ~~# x# x# x# x# x# x~~_________________________________________________________________________________

[[Category:ULI101]]

Andrew

Bureaucrats, Administrators

1,234

edits

CDOT Wiki β

Changes

Tutorial9: Regular Expressions

CDOT Wiki ^β