Tutorial9: Regular Expressions

USING REGULAR EXPRESSIONS

Main Objectives of this Practice Tutorial

• Explain the purpose of Regular Expressions

• Understand the difference between Regular Expressions and Filename Expansion

• Understand the purpose of Literal (Simple) Regular Expressions

• Understand common symbols for Complex Regular Expressions and their purpose

• Understand command symbols for Extended Regular Expressions and their purpose

• Use various Linux commands that use regular expressions

Tutorial Reference Material

Course Notes		Linux Command/Shortcut Reference		YouTube Videos
Course Notes: PDF | PPTX		Regular Expressions Definition Purpose (WIKI)	Additional Linux Commands man more / less vi / vim sed awk wget	Brauer Instructional Videos: Using grep Command with Regular Expressions

KEY CONCEPTS

A regular expression … is a sequence of characters that define a search pattern.

Usually such patterns are used by string searching algorithms for "find" or "find and replace" operations on strings, or for input validation.
Reference: https://en.wikipedia.org/wiki/Regular_expression

Regular Expressions vs. Filename Expansion

Filename expansion symbols allow the Linux shell to expand filenames as arguments (referred to as “globbing”).

Concept of matching a simple pattern of text contained within a text file.

In a previous lesson, you learned that filename expansion symbols allow the Linux shell to expand filenames as arguments (referred to as “globbing”) for file management commands. This is very useful for managing multiple files sharing similar characteristics such as the same file extension.

Filename Expansion symbols are used to search, edit and manipulate text and are used with Linux file management commands such as ls, rm, mv, cp, cat, less and more. This can represent text contained in files or text as a result of issuing Linux commands using a pipeline.

A regular expression is a combination of two types of characters: literals and special characters.
In combination, these characters define a logical pattern. Strings of text can be compared to this pattern
to see if they fit the pattern defined by the expression.

Reference: https://www.whoishostingthis.com/resources/regex/

Literal (Simple) Regular Expressions

The simplest regular expression is a series of letters and numbers, possibly including white space (tabs or space characters).

The simplest regular expression is a series of letters and numbers, possibly including white space (tabs or space characters), that have no special meaning. Such a regular expression consists of ``literals; that is, normal letters, which match only an identical letter in the data being searched.

For example:

"This is a regular expression"

When an editor searches for a literal regular expression, it can only score a ``hit if it finds exactly that sequence of characters in the data it is searching. 
Reference: http://osr507doc.xinuos.com/en/OSUserG/_Literal_chars_in_regexps.html

Complex Regular Expressions

The problem with using literals or simple regular expressions is that general patterns are matched.

For example, the pattern “the” would be matched for larger word containing that pattern like there, they, either, them, their, etc.
Another problem is that you may want to search for pattern at a specific location within the string of text (like at the beginning or end).

There are other regular expression tools to provide more precise matches. These tools are complex and extended regular expressions.

Complex Regular Expressions use symbols to help match text for more precise (complex) patterns.
The most common complex regular expression symbols are displayed below:

• Anchors ^ , $
• Characters .
• Character Class [ ] , [^ ]
• Zero or More Occurrence *

Examples of how to use these complex regular expressions with the grep command are displayed below:

Example of using anchors.

Example of matching by character(s).

Example of using character class.

Example of matching zero or more occurrence of preceding character.

Extended Regular Expressions

Extended Regular Expressions consist of additional special characters the “extend” the capability of regular expressions.

Repetition {min,max}
Allows for more precise repetitions. Using braces, you can specify the minimum and maximum number of repetitions.

Grouping ( )
Perhaps you want to search for repetition for a group of characters, a word, or a phase.
You can enclose them within brackets ( ) to specify a group.

or Condition |
Can be used with grouping to match a variety of character(s), words or phases. The | symbol is used to separate the variety of character(s).

Examples of how to use these complex regular expressions with the grep command are displayed below:

Example of using repetition.

Example of using grouping.

Example of using or condition with grouping.

INVESTIGATION 1: SIMPLE & COMPLEX REGULAR EXPRESSIONS

In this section, you will learn how to use the grep command with simple and complex regular expressions to help search for patterns contained in text files.

Perform the Following Steps:

1. Login to your matrix account.
   
   
2. Issue a command to confirm you are located in your home directory.
   
   The wget command is used to download files from the Internet to your shell.
   This will be useful to download text files that we will be using for this tutorial.
   
   
3. Issue the following linux command (copy and paste to save time):
   wget https://ict.senecacollege.ca/~murray.saul/uli101/textfile1.txt
   
   
4. Issue the ls command to confirm that the text file was downloaded.
   
   
5. View the contents of the textfile1.txt file using the more command and quickly view the contents of this file. When finished, exit the more command.
   
   Although there are several Linux commands that use regular expressions, we will only be using the grep command for this section.
   
   
6. Issue the following linux pipeline command to match the pattern the within textfile1.txt:
   grep "the" textfile1.txt | more
   
   
7. Now, issue the grep linux pipeline command with the -i option to ignore case sensitively:
   grep -i "the" textfile1.txt | more
   
   What do you notice is different with this pipeline command?
   
   You will notice that the pattern "the" is matched including larger words that contain the pattern "the". You can use the -w option with the grep command in order to just match only words for a pattern.
   
   
8. Issue the following linux pipeline command:
   grep -w -i "the" textfile1.txt | more
   
   You should now see only strings of text that match the word "the".
   
   Just matching literal or simple regular expressions can be useful, but are limited in what they can assist with pattern matching.
   For Example, you may want to search for pattern at a specific location within the string of text (like at the beginning or end of the string).
   
   There are other regular expression tools to provide more precise matches. These tools are complex and extended regular expressions. We will now look at complex regular expression symbols now, and we will discuss extended regular expressions'Italic text' in the next section of this tutorial.
   
   
9. Issue the following Linux pipeline command:
   grep -w -i "^the" textfile1.txt | more
   
   The ^ symbol is an anchor. In this case, it only matches the word "the" (both upper or lowercase) at the beginning of strings.
   The $ symbol is used to anchor patterns at the end of strings.
   
   
10. Issue the following Linux pipeline command:
    grep -w -i "the$" textfile1.txt | more
    
    What do you notice?
    
    
11. Issue the following Linux pipeline command to anchor the work "the" simultaneously at the beginning and the end of the string:
    grep -w -i "^the$" textfile1.txt | more
    
    What do you notice?
    
    Anchoring patterns at both the beginning and ending of strings can greatly assist for more complex search patterns.
    We will now be demonstrating simultaneous anchoring with other complex regular expressions symbols.
    
    
12. Issue the following command to match strings that begin with 3 characters:
    grep "^..." textfile1.txt | more
    
    What do you notice?
    
    
13. Issue the following command to match strings that begin and end with 3 characters:
    grep "^...$" textfile1.txt | more
    
    What do you notice?
    
    
14. Issue the following command to match strings that begin with 3 digits:
    grep "^[0-9][0-9][0-9]" textfile1.txt | more
    
    
15. Issue the following command to match strings that end with 3 uppercase letters:
    grep "[A-Z][A-Z][A-Z]$" textfile1.txt | more
    
    
16. Issue the following command to match strings that consist of only 3 digits:
    grep "^[0-9][0-9][0-9]$" textfile1.txt | more
    
    The * complex regular expression symbol is often confused with filename expansion. In other words, it does NOT represent zero or more of any character, but zero or or occurrences of the character that comes before the * symbol.
    
    
17. To demonstration, issue the following command to display zero or more occurrences of the letter x:
    grep "x*" textfile1.txt | more
    
    You will most likely notice most lines of the file is displayed.
    
    
18. Let's issue a command to display strings that contain more than one occurrence of the letter x:
    grep "xx*" textfile1.txt | more
    
    Why did this work? because the pattern indicates one occurrence of the letter x, followed by zero or MORE occurrences of the letter x.
    
    If you combine the complex regular expression symbols .* it will act like zero or more occurrence of any character (like * did in filename expansion).
    
    
19. Issue the following command to match strings begin and end with a number with nothing or anything inbetween:
    grep "^[0-9].*[0-9]$" textfile1.txt | more
    
    Using simultaneous anchors combined with the .* symbol(s) can help you to refine your search patterns of strings.
    
    
20. Issue the following linux pipeline command to display strings that begin with a capital letter, ends with a number, and contains a capital X somewhere inbetween:
    grep "^[A-Z].*X.*[0-9]$" textfile1.txt | more
    
    Let's look at another series of examples involving filtering with numbers so only strings containing valid numbers are displayed.
    
    
21. First, issue the following linux command to download another data file called numbers1.dat:
    wget https://ict.senecacollege.ca/~murray.saul/uli101/numbers1.dat
    
    
22. View the contents of the numbers.dat file using the more command and quickly view the contents of this file. You should notice valid and invalid numbers contained in this file. When finished, exit the more command.
    
    
23. Issue the following linux command to display only whole numbers:
    grep "^[0-9]*$" numbers1.dat | more
    
    You may have noticed that the command does not entirely work. You may notice an empty line (which is NOT a whole number). This occurs since the * regular expression symbol represents ZERO or MORE occurrences of a number. You can use an additional numeric character class with the * regular expression symbol to search for one or more occurrences of a number.
    
    
24. Issue the following linux command to display only whole numbers:
    grep "^[0-9][0-9]*$" numbers1.dat | more
    
    You should see that this works.
    
    
25. Issue the following linux command to display whole positive or negative integers:
    grep "^[+-][0-9][0-9]*$" numbers1.dat | more
    
    What did you notice?
    
    
26. Issue the following linux command to display only whole numbers (with or without a positive or negative sign):
    grep "^[+-]*[0-9]*$" numbers1.dat | more
    
    

Proceed to Investigation 2.

INVESTIGATION 2: EXTENDED REGULAR EXPRESSIONS

In this section, you will learn how to use extended regular expressions to help refine your search when using regular expressions.

Perform the Following Steps:

1. Issue the following linux command to download another data file called numbers2.dat:
   wget https://ict.senecacollege.ca/~murray.saul/uli101/numbers2.dat
   
   
2. View the contents of the numbers2.dat file using the more command and quickly view the contents of this file.
   You should notice valid and more invalid numbers contained in this file. When finished, exit the more command.
   
   
3. Issue the following linux command to display only whole numbers (with or without a positive or negative sign):
   grep "^[+-]*[0-9][0-9]*$" numbers2.dat | more
   
   You should notice multiple + or - signs are appearing as well. This occurs since you are searching or one or MORE occurrences of a + or - sign.
   Using extended regular expression symbols to specify minimum and maximum repetitions {min,max} can solve this problem.
   
   
4. Issue the following linux command (using extended regular expression symbols) to display only whole numbers (with or without a positive or negative sign):
   grep "^[+-]{0.1}[0-9]{1,}$" numbers2.dat | more
   
   NOTE: most likely, there were NO results. This is due to the fact that the grep command was NOT issued correctly to use extended regular expression symbols. You would need to issue either grep -E (or more simply) issue the egrep command. The egrep command works with all regular expression symbols, and should be used in the future instead of the older grep command.
   
   
5. Reissue the above command using egrep instead of grep:
   egrep "^[+-]{0,1}[0-9]{1,}$" numbers2.dat | more
   
   You should have noticed that the command worked correctly this time because you used the egrep command.
   
   
6. Issue the following linux command to display signed, unsigned, whole, and decimal numbers:
   egrep "^[+-]{0,1}[0-9]{1,}[.]{0,1}[0-9]*$" numbers2.dat | more
   
   You can also use extended regular expression symbols for grouping. For example, you can search for repetitions of GROUPS of characters
   (like a word) as opposed to just a single character or a GROUP of numbers as opposed to a single digit.
   
   
7. Issue the following linux command to download another data file called words.dat:
   wget https://ict.senecacollege.ca/~murray.saul/uli101/words.dat
   
   
8. View the contents of the numbers2.dat file using the more command and quickly view the contents of this file.
   You should notice valid and more invalid numbers contained in this file. When finished, exit the more command.
   
   
9. Issue the following linux command to display two or more occurrences of the word "the":
   egrep -i "(the){2,}" words.dat | more
   
   NOTE: You should NOT see any output due to the fact that a space should be included at the end of the word "the". Usually words are separated by spaces; therefore, there were no matches since there were not occurrences of "thethe" as opposed to "the the"
   
   
10. Reissue the previous command including a space in brackets:
    egrep -i "(the ){2,}" words.dat | more
    
    The or symbol | can be used within the grouping regular expression symbol to allow matching of additional groups of characters. Again, it is important to follow the character groupings with the space character
    
    
11. Issue the following linux command to search for 2 or more occurrences of the word "the" or the word "and":
    egrep -i "(the |and ){2,}" words.dat | more
    
    

Proceed to Investigation 3

INVESTIGATION 3: OTHER COMMANDS THAT USE REGULAR EXPRESSIONS

In this section, you will see how regular expressions can be used with other Linux utilities than just grep or egrep.

Perform the Following Steps:

1. Let's look at using regular expressions with the man command.
   Issue the following linux command :
   man ls
   
   
2. We want to search for an option that can sort the file listing.
   Type the following regular expression below and press ENTER:
   /sort
   
   FYI: The grep and egrep linux commands contain the regular expressions within quotes,
   but most other Linux commands specify regular expressions using forward slashes
   (e.g. /regular expression/ or /regular expression).
   
   
3. Scroll throughout the man pages for the ls command to view matches for the pattern "sort"
   (You can press SPACE and <ALT><b> to move forward and backwards per page respectively).
   
   
4. Press the letter q to exit the man pages for ls.
   
   
5. Let's use regular expressions with the more command.
   Issue the following linux command to view the contents of the textfile1.txt (downloaded in a previous section):
   more textfile1.txt
   
   
6. We want to search for a pattern uli101 within this text file.
   Type the following regular expression and press ENTER:
   /uli101
   
   What did you notice?
   
   
7. Search for the next occurrence of the pattern uli101 by re-typing the following regular expression and pressing ENTER:
   /uli101
   
   you should now see the second occurrence of this pattern within the text file.
   
   
8. Press the letter q to exit the more command.
   
   
9. Let's learn how to perform a simple search and replace within the vi utility by using regular expressions.
   Issue the following linux command to edit the textfile1.txt file:
   vi textfile1.txt
   
   Let's first perform a simple search within this text file.
   
   
10. Type the following and press ENTER:
    /uli101
    
    You should move to the first occurrence of uli101.
    
    Let's search for the uli101 pattern, but replace it in capitals (i.e ULI101).
    
    In vi, in order to perform and command, you need to go into last line mode, and then issue a command to apply to the entire text file, followed by a regular expression to search for and a regular expression to replace (i.e. /search/replace/).
    
    
11. Type the following and press ENTER:
    :%s/uli101/ULI101
    
    You should have noticed that the first occurrence of uli101 has been changed to ULI101.
    
    
12. Navigate throughout the text file to see if the other occurrences have been replaced.
    
    You should notice they haven't for the other two occurrences. In order to replace for ALL occurrences, you need to add the letter g (meaning "global") at the end of the last forward slash (e.g /search/replace/g).
    
    
13. Making certain that you are command mode in vi, type the following and press ENTER:
    :%s/uli101/ULI101/g
    
    
14. Navigate throughout the text file to confirm that ALL occurrences of uli101 have been replaced with ULI101.
    
    
15. Save changes to your vi editing session and exit by typing the following and pressing ENTER:
    :wx
16. After you complete the Review Questions sections to get additional practice, then work on your
    online assignment 3.

LINUX PRACTICE QUESTIONS

The purpose of this section is to obtain extra practice to help with quizzes, your midterm, and your final exam.

Here is a link to the MS Word Document of ALL of the questions displayed below but with extra room to answer on the document to simulate a quiz:

https://ict.senecacollege.ca/~murray.saul/uli101/uli101_command_practice_9a.docx

Your instructor may take-up these questions during class. It is up to the student to attend classes in order to obtain the answers to the following questions. Your instructor will NOT provide these answers in any other form (eg. e-mail, etc).

Review Questions:

Part A: Display Results from Linux Commands using Regular Expressions

Note the contents from the following tab-delimited file called ~murray.saul/uli101/cars:

Plym	fury		77	73	2500
chevy	nova		79	60	3000
ford	mustang	65	45	10003
volvo	gl		78	102	9850
ford	ltd		83	15	10507
chevy	nova		80	50	3503
fiat	600		65	115	450
honda	accord	81	30	6000
ford	thundbd	84	10	17000
toyota	tercel	82	180	755
chevy	impala	65	85	1553
ford	bronco	83	25	9505

Write the results of each of the following Linux commands using regular expressions for the above-mentioned file.

1. grep plym ~murray.saul/uli101/cars

2. grep -i fury ~murray.saul/uli101/cars

3. grep “^[m-z]” ~murray.saul/uli101/cars

4. grep -i “^[m-z]” ~murray.saul/uli101/cars

5. grep “3$” ~murray.saul/uli101/cars

6. grep -i “c.*5$” ~murray.saul/uli101/cars

Part B: Writing Linux Commands Using Regular Expressions

7. Write a Linux command to display all lines in the file called ~/text.txt that contains the pattern: the

8. Write a Linux command to display all lines in the file called ~/text.txt that contains the word: the

9. Write a Linux command to display all lines in the file called ~/text.txt that begin with a number.

10. Write a Linux command to display all lines in the file called ~/text.txt that end with a letter (either upper or lowercase).

11. Write a Linux command to display all lines in the file called ~/text.txt that begin and end with a number.

12. Write a Linux command to display all lines in the file called ~/text.txt that contains exactly 3 characters that can be anything.

13. Write a Linux command to display all lines in the file called ~/text.txt that contains exactly 3 numbers.

14. Write a Linux command to display all lines in the file called ~/text.txt that contains 1 or more “C” characters.

If you wish additional practice using extended regular expressions, here is a link to another MS Word document: https://ict.senecacollege.ca/~murray.saul/uli101/uli101_command_practice_9b.docx