Tutorial 9 - Regular Expressions

Content under development

USING REGULAR EXPRESSIONS

Main Objectives of this Practice Tutorial

• Define the term Regular Expressions

• Explain the difference between Regular Expressions and Filename Expansion

• Explain the purpose of Literal (Simple) Regular Expressions

• Understand and use common symbols for Complex Regular Expressions and their purpose

• Understand and use command symbols for Extended Regular Expressions and their purpose

• List several Linux commands that can use regular expressions

Tutorial Reference Material

Course Notes		Linux Command/Shortcut Reference
Course Notes: PDF | PPTX		Regular Expressions Definition Purpose (WIKI)	Linux Commands egrep man more / less vi / vim sed awk wget

KEY CONCEPTS

Regular Expressions

A regular expression is a combination of two types of characters: literals and special characters.
Strings of text can be compared to this pattern to see if there is a match.

This usually refers to text that is contained inside a file or text as a result
of issuing Linux commands using a Linux pipeline command.

Literal (Simple) Regular Expressions

A simple (literal) regular expression is a series of letters and numbers (tabs or spaces).

The simplest regular expression is a series of letters and numbers, (tabs or spaces).
A simple (literal) regular expression consists of normal characters, which used to match patterns.

Although there are many Linux commands that use regular expressions, the grep command is a useful command to learn how to display matches of patterns of strings within text files.

For example: grep Linux document.txt

Complex / Extended Regular Expressions

Complex Regular Expressions

The problem with just using simple (literal) regular expressions is that only simple or general patterns are matched.

Complex Regular Expressions use symbols to help match text for more precise (complex) patterns.
The most common complex regular expression symbols are displayed below:

Anchors: ^ , $
Match lines the begin (^) or end ($) with a pattern.

Single Character:   .
Represents a single character that can be any type of character.

Character Class: [ ] , [^ ]
Represents a single character but with restrictions.

Zero or More Occurrence: *
Zero or more occurrences of previous character.

Examples of complex regular expressions are displayed below:

Example of using anchors.

Example of matching by character(s).

Example of using character class.

Example of matching zero or more occurrence of preceding character.

Extended Regular Expressions

Extended Regular Expressions consist of additional special characters to “extend”
the capability of regular expressions. You must use the egrep or grep -E commands
in order to properly use extended regular expressions.

Repetition: {min,max}
Allows for more precise repetitions. Using braces, you can specify
the minimum and/or maximum number of repetitions.

Groups: ( )
Allows you to search for repetition for a group of characters, a word, or a phase.
You enclose them within brackets ( ) to specify a group.

or Condition: |
Can be used with groups to match a variety of character(s), words or phases.
The | symbol is used to separate the variety of character(s) within a group.

Examples of how to use extended regular expressions with the egrep command are displayed below:

Example of using repetition.

Example of using groups.

Example of using or condition with groups.

INVESTIGATION 1: SIMPLE & COMPLEX REGULAR EXPRESSIONS

ATTENTION: The due date for successfully completing this tutorial (i.e. tutorial 8) is by Friday, December 15 @ 11:59 PM (Week 14).

In this investigation, you will learn how to use the grep command with simple and complex regular expressions
to help search for patterns contained in text files.

Perform the Following Steps:

1. Login to your matrix account.
   
   
2. Issue a Linux command to confirm you are located in your home directory.
   
   The wget command is used to download files from the Internet to your shell.
   This will be useful to download text files and data files that we will be using in this tutorial.
   
   
3. Issue the following linux Linux command to download a text file to your home directory:
   wget https://ict.senecacollege.ca/~murray.saul/uli101/textfile1.txt
   
   
4. View the contents of the textfile1.txt file using the more command see what data is contained in this file.
   
   Although there are several Linux commands that use regular expressions,
   we will be using the grep command for this investigation.
   
   
   

   Output of grep command matching simple regular expression "the" (only lowercase). Notice the pattern matches larger words like "their" or "them".
5. Issue the following Linux command to match the pattern the within textfile1.txt:
   grep "the" textfile1.txt
   
   Take a few moments to view the output and observe the matched patterns.
   
   
6. Issue the grep Linux command with the -i option to ignore case sensitively:
   grep -i "the" textfile1.txt
   
   What do you notice is different when issuing this command?
   
   You will notice that the pattern "the" is matched including larger words like "them" and "their".
   You can issue the grep command with the -w option to only match the pattern as a word.
   
   
7. Issue the following Linux command:
   grep -w -i "the" textfile1.txt
   
   You should now see only strings of text that match the word the (upper or lower case).
   
   Matching literal or simple regular expressions can be useful, but are limited
   in what pattens they can match. For example, you may want to
   search for a pattern located at the beginning or end of the string.
   
   There are other regular expression symbols that provide more precise search pattern matching.
   These special characters are known as complex and extended regular expressions symbols.
   
   For the remainder of this investigation, we will focus on complex regular expressions and then
   focus on extended regular expressions in INVESTIGATION 2.
   
   

   Anchoring regular expressions at the beginning of text.

   Anchoring regular expressions at the ending of text.

8. Issue the following Linux command:
   grep -w -i "^the" textfile1.txt
   
   The ^ symbol is referred to as an anchor.
   In this case, it only matches
   the word "the" (both upper or lowercase) at the beginning of the string.
   
   
9. Issue the following Linux command:
   grep -w -i "the$" textfile1.txt
   
   The $ symbol is used to anchor patterns at the end of the string.
   
   
10. Issue the following Linux command to anchor the word "the"
    simultaneously at the beginning and end of the string:
    grep -w -i "^the$" textfile1.txt
    
    What do you notice?
    
    Anchoring patterns at both the beginning and ending of strings can greatly assist
    for more precise search pattern matching.
    
    We will now be demonstrate the effectiveness of combining
    anchors with other complex regular expressions symbols.
    
    

    Anchoring regular expressions using period symbols at the beginning of text.

    Anchoring regular expressions using period symbols simultaneously at the beginning and ending of text.

11. Issue the following Linux command to match strings that begin with 3 characters:
    grep "^..." textfile1.txt
    
    What do you notice? Can lines that contain less than 3 characters be displayed?
    
    
12. Issue the following Linux command to match strings that begin and end with 3 characters:
    grep "^...$" textfile1.txt
    
    What do you notice compared to the previous command?
    
    
13. Issue the following Linux command to match strings that begin with 3 digits:
    grep "^[0-9][0-9][0-9]" textfile1.txt
    
    What did you notice?
    
    
14. Issue the following Linux command to match strings that end with 3 uppercase letters:
    grep "[A-Z][A-Z][A-Z]$" textfile1.txt
    
    

    Anchoring 3 digits at the beginning and ending of text.

    Anchoring 3 alpha-numeric characters at the beginning and ending of text.

    What type of strings match this pattern?
    
    
15. Issue the following Linux command to match strings that consist of only 3 digits:
    grep "^[0-9][0-9][0-9]$" textfile1.txt
    
    What did you notice?
    
    
16. Issue the following Linux command to match strings that consist of only 3 alphanumeric digits:
    grep "^[a-zA-Z0-9][a-zA-Z0-9][a-zA-Z0-9]$" textfile1.txt
    
    What did you notice?
    
    The "*" complex regular expression symbol is often confused with the "*" filename expansion symbol.
    In other words, it does NOT represent zero or more of any character, but zero or more occurrences
    of the character that comes before the "*" symbol.
    
    
17. To demonstrate, issue the following Linux command to display zero or more occurrences of the letter "x":
    grep "x*" textfile1.txt
    
    You will most likely notice most lines of the file is displayed.
    
    
18. Let's issue a Linux command to display strings that contain more than one occurrence of the letter "x":
    grep "xx*" textfile1.txt
    
    Why did this work? because the pattern indicates one occurrence of the letter "x",
    followed by zero or MORE occurrences of the next letter "x".
    
    If you combine the complex regular expression symbols ".*" it will act like
    zero or more occurrences of any character (i.e. like "*" did in filename expansion).
    
    
19. Issue the following Linux command to match strings begin and end with a number with nothing or anything inbetween:
    grep "^[0-9].*[0-9]$" textfile1.txt
    
    Using simultaneous anchors combined with the ".*" symbol(s) can help you to refine your search patterns of strings.
    
    
20. Issue the following Linux command to display strings that begin with a capital letter,
    end with a number, and contains a capital X somewhere inbetween:
    grep "^[A-Z].*X.*[0-9]$" textfile1.txt
    
    Let's look at another series of examples involving searching for strings that only contain valid numbers.
    We will use pipeline commands to both display stdout to the screen and save to files
    for confirmation of running these pipeline commands when run a checking-script later in this investigation.
    
    
21. Issue the following Linux command to create the regexps directory: mkdir ~/regexps
    
    
22. Change to the regexps directory and confirm that you have moved to this directory.
    
    
23. First, issue the following Linux command to download another data file called numbers1.dat:
    wget https://ict.senecacollege.ca/~murray.saul/uli101/numbers1.dat
    
    
24. View the contents of the numbers.dat file using the more command and quickly view the contents of this file.
    You should notice valid and invalid numbers contained in this file. When finished, exit the more command.
    
    
25. Issue the following linux pipeline command to display only whole numbers (i.e. no + or - sign):
    grep "^[0-9]*$" numbers1.dat | tee faulty.txt
    
    You may have noticed that the command does not entirely work. You may notice an empty line
    (which is NOT a whole number). This occurs since the * regular expression symbol represents
    ZERO or MORE occurrences of a number. You can use an additional numeric character class
    with the * regular expression symbol to search for one or more occurrences of a number.
    
    
26. Issue the following Linux pipeline command to display only whole numbers:
    grep "^[0-9][0-9]*$" numbers1.dat | tee whole.txt
    
    You should see that this now works.
    
    
27. Issue the following Linux pipeline command to display only signed integers:
    grep "^[+-][0-9][0-9]*$" numbers1.dat | tee signed.txt
    
    What did you notice? Positive and negative numbers display, not unsigned numbers.
    
    
    

    Simultaneous anchoring of regular expressions using character class and zero or more occurrences to display signed and unsigned integers.
28. Issue the following Linux pipeline command to display signed or unsigned integers:
    grep "^[+-]*[0-9][0-9]*$" numbers1.dat | tee all.txt
    
    Did this command work?
    
    
29. Issue the following command to check that you created those hard links:
    ~uli101/week9-check-1
    
    If you encounter errors, then view the feedback to make corrections, and then re-run the checking script. If you receive a congratulation message that there are no errors, then proceed with this tutorial.
    
    You can also use the grep command using regular expression as a filter in pipeline commands.
    
    
30. Issue the following Linux pipeline command:
    ls | grep "[0-9].*dat$"
    
    What did this pipeline display?
    
    
31. Issue the following Linux pipeline command:
    ls | grep "[a-z].*txt$"
    
    What did this pipeline display?
    
    

Although very useful, complex regular expressions do NOT entirely solve our problem of displaying
valid unsigned and signed numbers (not to mention displaying decimal numbers).

In the next investigation, you will learn how to use extended regular expressions that will completely solve this issue.

You can proceed to INVESTIGATION 2.