Changes

Tutorial 9 - Regular Expressions

13,065 bytes added, 20:28, 25 October 2021

→‎INVESTIGATION 1: SIMPLE & COMPLEX REGULAR EXPRESSIONS

= INVESTIGATION 1: SIMPLE & COMPLEX REGULAR EXPRESSIONS =

'''ATTENTION''': The due date for successfully completing this tutorial (i.e. tutorial 8) is by Friday, December 15 @ 11:59 PM (Week 14).

In this investigation, you will learn how to use the '''grep''' command with '''simple and complex regular expressions''' to help search for ''patterns'' contained in text files.

'''Perform the Following Steps:'''

# '''Login''' to your matrix account.

# Issue a Linux command to '''confirm''' you are located in your '''home''' directory. The '''wget''' command is used to download files from the Internet to your shell. This will be useful to download '''text files''' and '''data files''' that we will be using in this tutorial.

# Issue the following linux Linux command to '''download''' a text file to your '''home''' directory: wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/textfile1.txt</nowiki>

# View the contents of the '''textfile1.txt''' file using the '''more''' command see what data is contained in this file. Although there are several Linux commands that use regular expressions, we will be using the '''grep''' command for this investigation. [[Image:regexps-1.png|thumb|right|250px|Output of '''grep''' command matching simple regular expression "'''the'''" (only lowercase). Notice the pattern matches larger words like "'''their'''" or "'''them'''".]]

#Issue the following Linux command to match the pattern '''the''' within '''textfile1.txt''': grep "the" textfile1.txt Take a few moments to view the output and observe the matched patterns.

# Issue the grep Linux command with the -i option to ignore case sensitively: grep -i "the" textfile1.txt What do you notice is different when issuing this command? You will notice that the pattern "'''the'''" is matched including larger words like "'''them'''" and "'''their'''". You can issue the '''grep''' command with the -w option to only match the pattern as a '''word'''.

# Issue the following Linux command: grep -w -i "the" textfile1.txt You should now see only strings of text that match the word '''the''' (upper or lower case). Matching literal or simple regular expressions can be useful, but are '''limited''' in what pattens they can match. For example, you may want to search for a pattern located at the '''beginning''' or '''end''' of the string. There are other regular expression symbols that provide more '''precise''' search pattern matching. These special characters are known as '''complex''' and '''extended''' regular expressions symbols. For the remainder of this investigation, we will focus on '''complex regular expressions''' and then focus on ''extended regular expressions'' in INVESTIGATION 2. <table align="right"><tr valign="top"><td>[[Image:regexps-2.png|thumb|right|280px|Anchoring regular expressions at the '''beginning''' of text.]]</td><td>[[Image:regexps-3.png|thumb|right|250px|Anchoring regular expressions at the '''ending''' of text.]]</td></tr></table>

# Issue the following Linux command: grep -w -i "^the" textfile1.txt The '''^''' symbol is referred to as an '''anchor'''. In this case, it only matches the word "'''the'''" (both upper or lowercase) at the beginning of the string.

# Issue the following Linux command: grep -w -i "the$" textfile1.txt The '''$''' symbol is used to anchor patterns at the end of the string.

# Issue the following Linux command to anchor the word "'''the'''" '''simultaneously''' at the beginning and end of the string: grep -w -i "^the$" textfile1.txt What do you notice? Anchoring patterns at both the beginning and ending of strings can greatly assist for more '''precise''' search pattern matching. We will now be demonstrate the '''effectiveness''' of combining '''anchors''' with other complex regular expressions symbols. <table align="right"><tr valign="top"><td>[[Image:regexps-4.png|thumb|right|280px|Anchoring regular expressions using '''period''' symbols at the '''beginning''' of text.]]</td><td>[[Image:regexps-5.png|thumb|right|250px|Anchoring regular expressions using '''period''' symbols simultaneously at the '''beginning''' and '''ending''' of text.]]</td></tr></table>

# Issue the following Linux command to match strings that '''begin with 3 characters''': grep "^..." textfile1.txt What do you notice? Can lines that contain '''less than 3 characters''' be displayed?

# Issue the following Linux command to match strings that '''begin and end with 3 characters''': grep "^...$" textfile1.txt What do you notice compared to the previous command?

# Issue the following Linux command to match strings that '''begin with 3 digits''': grep "^[0-9][0-9][0-9]" textfile1.txt What did you notice?

# Issue the following Linux command to match strings that '''end with 3 uppercase letters''': grep "[A-Z][A-Z][A-Z]$" textfile1.txt <table align="right"><tr valign="top"><td>[[Image:regexps-6.png|thumb|right|220px|Anchoring '''3 digits''' at the '''beginning''' and '''ending''' of text.]]</td><td>[[Image:regexps-7.png|thumb|right|250px|Anchoring '''3 alpha-numeric characters''' at the '''beginning''' and '''ending''' of text.]]</td></tr></table>What type of strings match this pattern?

# Issue the following Linux command to match strings that '''consist of only 3 digits''': grep "^[0-9][0-9][0-9]$" textfile1.txt What did you notice?

# Issue the following Linux command to match strings that '''consist of only 3 alphanumeric digits''': grep "^[a-zA-Z0-9][a-zA-Z0-9][a-zA-Z0-9]$" textfile1.txt What did you notice? The "*" complex regular expression symbol is often confused with the "*" '''filename expansion''' symbol. In other words, it does NOT represent zero or more of '''any character''', but zero or more '''occurrences''' of the character that comes '''before''' the "*" symbol.

# To demonstrate, issue the following Linux command to display '''zero or more occurrences''' of the letter "'''x'''": grep "x*" textfile1.txt You will most likely notice most lines of the file is displayed.

# Let's issue a Linux command to display strings that contain '''more than one occurrence''' of the letter "'''x'''": grep "xx*" textfile1.txt Why did this work? because the pattern indicates one occurrence of the letter "x", followed by '''zero or MORE occurrences''' of the next letter "x". If you combine the complex regular expression symbols ".*" it will act like zero or more occurrences of any character (i.e. like "*" did in filename expansion).

# Issue the following Linux command to match strings begin and end with a number with nothing or anything inbetween: grep "^[0-9].*[0-9]$" textfile1.txt Using '''simultaneous anchors''' combined with the ".*" symbol(s) can help you to refine your search patterns of strings.

# Issue the following Linux command to display strings that begin with a capital letter, end with a number, and contains a capital X somewhere inbetween: grep "^[A-Z].*X.*[0-9]$" textfile1.txt Let's look at another series of examples involving searching for strings that only contain '''valid numbers'''. We will use '''pipeline commands''' to both display stdout to the screen and save to files for confirmation of running these pipeline commands when run a '''checking-script''' later in this investigation.

# Issue the following Linux command to create the '''regexps''' directory: mkdir ~/regexps

# Change to the '''regexps''' directory and confirm that you have moved to this directory.

# First, issue the following Linux command to download another data file called '''numbers1.dat''': wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/numbers1.dat</nowiki>

# View the contents of the '''numbers.dat''' file using the '''more''' command and quickly view the contents of this file. You should notice '''valid''' and '''invalid''' numbers contained in this file. When finished, exit the more command.

# Issue the following linux pipeline command to display only '''whole''' numbers (i.e. no '''+''' or '''-''' sign): grep "^[0-9]*$" numbers1.dat | tee faulty.txt You may have noticed that the command '''does not entirely work'''. You may notice an '''empty line''' (which is NOT a whole number). This occurs since the * regular expression symbol represents ZERO or MORE occurrences of a number. You can use an additional numeric character class with the * regular expression symbol to search for one or more occurrences of a number.

# Issue the following Linux pipeline command to display only whole numbers: grep "^[0-9][0-9]*$" numbers1.dat | tee whole.txt You should see that this now works.

# Issue the following Linux pipeline command to display only '''signed''' integers: grep "^[+-][0-9][0-9]*$" numbers1.dat | tee signed.txt What did you notice? Positive and negative numbers display, not '''unsigned''' numbers. [[Image:regexps-8.png|thumb|right|300px|Simultaneous '''anchoring''' of regular expressions using '''character class''' and '''zero or more occurrences''' to display '''signed''' and '''unsigned''' integers.]]

# Issue the following Linux pipeline command to display '''signed''' or '''unsigned integers''': grep "^[+-]*[0-9][0-9]*$" numbers1.dat | tee all.txt Did this command work?

# Issue the following command to check that you created those hard links: ~uli101/week9-check-1 If you encounter errors, then view the feedback to make corrections, and then re-run the checking script. If you receive a congratulation message that there are no errors, then proceed with this tutorial. You can also use the '''grep''' command using ''regular expression'' as a '''filter''' in pipeline commands.

# Issue the following Linux pipeline command: ls | grep "[0-9].*dat$" What did this pipeline display?

# Issue the following Linux pipeline command: ls | grep "[a-z].*txt$" What did this pipeline display?

: Although very useful, '''complex''' regular expressions do NOT entirely solve our problem of displaying '''valid''' unsigned and signed numbers (not to mention displaying decimal numbers). In the next investigation, you will learn how to use '''extended''' regular expressions that will completely solve this issue.

: You can proceed to INVESTIGATION 2.

= INVESTIGATION 2: EXTENDED REGULAR EXPRESSIONS =

Jason Michael Carman

715

edits

Changes

Tutorial 9 - Regular Expressions

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

get involved with CDOT

courses

course projects

links

Tools