1,576
edits
Changes
no edit summary
{{Admon/caution|DO NOT USE THIS VERSION OF THE LAB. LOOK FOR LAB 10: SED & AWK|'''This is an archive version.'''}}
=USING SED & AWK UTILTIES=
<br>
:* Use the '''sed''' command to '''manipulate text''' contained in a file.
:* List and explain several '''addresses''' and '''instructions''' associated with the '''sed''' command.
:* Use the '''sed''' command as a '''filter''' with Linux pipeline commands.
:* Use the '''awk''' command to '''manipulate text''' contained in a file.
:* List and explain several '''comparison operators''', '''variables''' and variables '''actions''' associated with the '''awk''' command.
:* Use the '''awk''' command as a '''filter''' with Linux pipeline commands.
|- valign="top" style="padding-left:15px;"
|colspan="2" |Course Notes'''Slides''':<ul><li>Week 11 Lecture 1 Notes:<br> [[Media:ULI101-Week11.1.pdf | PDF]] | [https://ictmatrix.senecacollege.ca/~murraychris.sauljohnson/ULI101/uli101ULI101-Week11.1.pptx PPTX]</li><li>Week 11 Lecture 2 Notes:<br> [[Media:ULI101-Week11.2.pdf | PDF]] | [https://ictmatrix.senecacollege.ca/~murrayjason.saulcarman/uli101slides/ULI101-Week11.2.pptx PPTX]<br></li></ul>
| style="padding-left:15px;" |'''Text Manipulation:'''
* [https://www.digitalocean.com/community/tutorials/the-basics-of-using-the-sed-stream-editor-to-manipulate-text-in-linux Purpose of using the sed utility]
* [https://www.digitalocean.com/community/tutorials/how-to-use-the-awk-language-to-manipulate-text-in-linux Purpose of using the awk utility]
| style="padding-left:15px;" |Man Pages'''Commands:'''
* [https://man7.org/linux/man-pages/man1/sed.1p.html sed]
* [https://man7.org/linux/man-pages/man1/awk.1p.html awk]
|colspan="1" style="padding-left:15px;" width="30%"|'''Brauer Instructional Videos:'''<ul><li>[https://www.youtube.com/watch?v=npU6S61AIko&list=PLU1b1f-2Oe90TuYfifnWulINjMv_Wr16N&index=14 Using the sed Utility]</li><li>[https://www.youtube.com/watch?v=OV3XzjDYgJo&list=PLU1b1f-2Oe90TuYfifnWulINjMv_Wr16N&index=13 Using the awk Utility]</ul>
|}
'''NotesHow it Works:'''
*The sed command reads '''all reads all lines in the input file''' and file and will be exposed to the expression <br>(i.e. area contained in within quotes) one line at a time.* The expression can be within single quotes or double quotes.* The expression contains an address (match condition) and an instruction (operation).* If the line matches the address, then it will perform the instruction.* Lines will display be default unless the '''address–n''', then it will perform the option is used to suppress default display<br>'''instructionAddress:'''.
* Can use a line number, to select a specific line (for example: '''5''Address')* Can specify a range of line numbers (for example:'''5,7''')* Regular expressions are contained within forward slashes (e.g. /regular-expression/)* Can specify a regular expression to select all lines that match a pattern (e.g '''/^[0-9].*[0-9]$/''') * If NO address is present, the instruction will apply to ALL lines
[[Image:sed.png|thumb|right|500px|'''Common instructions''' to take action if text matches an address.]]
'''Instruction:'''
*'''Action''' to take for matched line(s)
*Refer to table on right-side for list of some <br>'''common instructions'''<br>and their purpose '''Examples:''' <span style="font-family:courier;">'''sed -n '3 p' text.txt''' (print 3rd line)<br>'''sed -n '1-5 p' text.txt''' (print lines 1 to 5)<br>'''sed -n '4,7 p' text.txt''' (print only lines 4 and 7)<br>'''sed -n'/^Therefore/ p' text.txt''' (print lines that begin with the patterh "Therefore")<br><br>'''sed '/^#/ d' myscript.bash''' (remove comments at beginning of line)<br>'''sed '/exit/ q' text.txt''' (print all lines until line containing pattern: "exit")<br>'''sed 's/this/that/' text.txt''' (replace first occurrence of pattern "this" with "that")<br>'''sed 's/this/that/' text.txt''' (replace ALL occurrences of pattern "this" with "that")<br><br></span>
===Using the awk Utility===
'''Usage:'''
<span style="color:blue;font-weight:bold;font-family:courier;">awk options [-F] 'selection-criteria {action}’ file-name</span>
'''NotesHow It Works:'''
*The '''awk command reads '''command reads all lines in the input file''' and will be exposed to the expression (contained within quotes) for processing.*Expression The '''expression''' (contained in quotes) represents '''selection criteria''', and '''action ''' to execute (contained within braces) '''{}'''* if selection criteria is matched *If no pattern is specified, awk selects all lines in the input*If no then action (between braces) is specified, awk copies the selected lines to standard outputexecuted.*You The '''–F''' option can use parameters like be used to specify the default '''$1field delimiter''', (separator) character<br>eg. '''$2awk –F”;”''' to represent first field, second field, etc (would indicate a semi-colon delimited input file).*You can use the <br>'''-FSelection Criteria''' option with the awk command to specify the field delimiter.
* You can use a regular expression, enclosed within slashes, as a pattern. For example: '''/pattern/'''
* The ~ operator tests whether a field or variable matches a regular expression. For example: '''$1 ~ /^[0-9]/'''
* The '''!~''' operator tests for no match. For example: '''$2 !~ /line/'''
* You can perform both numeric and string comparisons using relational operators ( '''>''' , '''>=''' , '''<''' , '''<=''' , '''==''' , '''!=''' ).
* You can combine any of the patterns using the Boolean operators '''||''' (OR) and '''&&''' (AND).
* You can use built-in variables (like NR or "record number" representing line number) with comparison operators.<br>For example: '''NR >=1 && NR <= 5'''
<br>
'''Action (execution):'''
* Action to be executed is contained within braces '''Patterns: Regular Expressions{}'''* The '''print''' command can be used to display text (fields).*You can use a parameters which represent fields within records (lines) within the expression of the awk utility.* The parameter '''regular expression$0''', enclosed within slashes, as a patternrepresents all of the fields contained in the record (line). *The parameters '''$1''', '''$2''', '''~$3''' operator tests whether a field or variable matches a regular expression *The … '''!~$9''' operator tests for no matchrepresent the first, second and third to the 9th fields contained within the record.*You can perform both numeric and string comparisons using relational operators *You can combine any Parameters greater than nine requires the value of the patterns using the Boolean operators parameter to be placed within braces (for example: '''${10}''','''||${11}''' (OR) and ,'''&&${12}''' (AND, etc.)*You can use use built-in '''built-in variables''' (like such as '''NR''' or or "record number" representing line number) with comparison operators [[Image:awk<br>eg.png|thumb|right|300px|'''Comparison operators''' used with the '''awk''' command.]]''{print NR,$0}'Patterns: Relational Operators''' *The following operators (in the table on the right-sidewill print record number, then entire record) can be used with the awk utility to pattern searching. *Since those symbols are used within the expression, they are NOT confused with redirection symbols.
=INVESTIGATION 1: USING THE SED UTILITY=
<span style="color:red;">'''ATTENTION'''Examples:Effective '''May 9, 2022''' - this online tutorial will be required to be completed by '''Friday in week 11 by midnight'''<br>to obtain a grade of '''2%'''towards this course</span><br><br>
'''Perform the Following Steps:'''
# '''Login''' to your matrix account.<br><br># Issue a Linux command to <u>and confirm</u> you are located in your '''home''' directory.<br><br>
# Issue a Linux command to create a directory called '''sed'''<br><br>
# Issue a Linux command to <u>change</u> to the '''sed''' directory.<br><br># Issue a Linux command to <u>and confirm</u> that you are located in the '''sed''' directory.<br><br># Issue the following linux Linux command (to download the data.txt file<br>('''copy and paste''' to save time):<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/data.txt</nowiki></span><br><br># Issue the '''more''' command to quickly view the contents of the '''data.txt''' file.<br>When finished, exit the more command by pressing the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span><br><br># The [[Image:sed-1.png|thumb|right|300px|Issuing the '''p''' command in sed is used instruction without using the '''-n''' option (to print or suppress original output) will display the contents of a text filelines twice.]]<br>Issue the following linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;">The '''p''' instruction with the '''sed 'p' data.txt</span><br>' command is used to<br>You should notice that each line appears '''twiceprint'''(i. The reason why standard output appears twice is that the sed command (without the '''-n option'e. ''display'') displays all lines regardless if they had been specified as the contents of a patterntext file.<br><br># Issue the following linux pipeline Linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed -n 'p' data.txt | tee sed-1.txt</span><br><br>What do you '''NOTE: You should notice?<that each line appears twice'''.<br><br>You can specify an address (line #, line #s or range of line #s) when using the The reason why standard output appears twice is that the sed utility.command<br><br># Issue (without the following linux pipeline command:'''-n option''') displays all lines regardless of an address used.<br><span style="color:blue;font-weight:bold;font-family:courier;">sed -n br>We will use ''1 p' data.txt | tee sed-2.txtpipeline commands''' to both display stdout to the screen and save to files</spanbr>for <bru>confirmation<br/u>You should see the first line of the text file displayedrunning these pipeline commands when run a '''checking-script''' later in this investigation.<br><br># Issue the following linux Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed -n '2,5 p' data.txt | tee sed-31.txt</span><br><br>What is displayed? How do you change command to display lines 2 to 5notice?You should see only one line.<br><br>The You can specify an '''saddress''' command is used to substitute patterns (similar to method demonstratedin vi editor).display lines using the sed utility<br><br>(eg. ''line #'', '''line # Issue the following linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed s''' or range of '''2,5 line #s/TUTORIAL/LESSON/g' data'').txt | tee sed-4.txt</span><br><br>What do you notice? View # Issue the original contents of lines 2 to 5 in the 'following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed -n '1 p'data.txt''' file in another shell to confirm that the substitution occurred| tee sed-2.txt</span><br><br>The '''q''' command terminates or '''quits''' You should see the execution first line of the sed utility as soon as it read text file displayed.<br>What other command is used to only display the first line in a particular line or matching pattern.file?<br><br># Issue [[Image:sed-2.png|thumb|right|500px|Using the following linux pipeline sed command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed to display a '''range''11 q' dataof lines.txt | tee sed-5.txt</span>]]# Issue the following Linux pipeline command:<br><br>What did you notice?<brspan style="color:blue;font-weight:bold;font-family:courier;">sed -n '2,5 p' data.txt | tee sed-3.txt</span><br><br>You can use regular expressions What is displayed? How would you modify the sed command to select lines that match a pattern. The rules remain display the same for using regular expressions as demonstrated in lab8 except the regular expression must be contained within delimiters such as the forward slash "/" when using the sed utility.line range 10 to 50?<br><br>The '''s''' instruction is used to '''substitute''' text<br>(a similar to method was demonstrated in the vi editor in tutorial 9).<br><br># Issue the following linux Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed -n '2,5 s/^TheTUTORIAL/ pLESSON/g' data.txt | tee sed-64.txt| more</span><br><br>What do you notice?<br><br># Issue View the following linux pipeline command:original contents of lines 2 to 5 in the '''data.txt''' file<br>in another shell to confirm that the substitution occurred.<span style="colorbr><br>[[Image:blue;font-weight:bold;font-family:courier;">sed -n '/d$/ p' data3.txt png|thumb|right|500px| tee Using the sedcommand with the '''-7.txt</span><br><br>What do you notice?<br><br>The q''' option to display up to a line number, then quit.]]The '''q''' instruction terminates or '''sedquits''' the execution of the sed utility can also be used as a filter to manipulate text that was generated from linux commandssoon as it is read in a particular line or matching pattern.<br><br># Issue the following linux Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | sed -n '/11 q' data.txt$/ p' | tee sed-85.txt</span><br><br>What did you notice?How many lines were displayed<br>before the sed command exited?<br># Issue the following linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;"You can use '''regular expressions''' to select lines that match a pattern. In fact,<br>who | the sed -n '/^[a-m]/ p' | tee sed-9.txt | morecommand was one of the <u>first</span><bru>Linux commands that used regular expression.<br>What did you notice?<br><br># Issue The rules remain the following to run a checking script:same for using regular expressions as demonstrated in '''tutorial 9'''<br><span style="colorexcept the regular expression must be contained within '''forward slashes'''<br>(eg. <span style="font-family:bluecourier;font-weight:bold;font-family:courier;">bash /home">/murray.saulregexp/scripts/week11-check-1</span>).<br>br><br>[[Image:sed-4.png|thumb|right|400px|Using the sed command using regular expressions with '''anchors'''.]]# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed -n '/^The/ p' data.txt | tee sed-6.txt</span><br><br>What do you notice?<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed -n '/d$/ p' data.txt | tee sed-7.txt</span><br><br>What do you notice?<br><br>The '''sed''' utility can also be used as a '''filter''' to manipulate text that<br>was generated from Linux commands.<br><br>[[Image:sed-5.png|thumb|right|400px|Using the sed command with '''pipeline''' commands.]]# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">who | sed -n '/^[a-m]/ p' | tee sed-8.txt | more</span><br><br>What did you notice?<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | sed -n '/txt$/ p' | tee sed-9.txt</span><br><br>What did you notice?<br><br># Issue the following to run a checking script:<br><span style="color:blue;font-weight:bold;font-family:courier;">~uli101/week11-check-1</span><br><br>If you encounter errors, make corrections and '''re-run''' the checking script<br># If you encounter errors, make corrections and '''re-run''' the checking script until you<br>receive a congratulations message, then you can proceed.<br><br>
:In the next investigation, you will learn how to manipulate text using the '''awk''' utility.<br><br>
=INVESTIGATION 2: USING THE AWK UTILITY =
In this sectioninvestigation, you will learn how to use the awk utility to manipulate text and generate reports.
'''Perform the Following Steps:'''
# Change to your '''home''' directory and issue a command to '''confirm''' <br>you are located<br>in your ''home'' directory.<br><br>
# Issue a Linux command to create a directory called '''awk'''<br><br>
# Issue a Linux command to <u>change</u> to the '''awk''' directory.<br><br># Issue a Linux command to <u>and confirm</u> you are located in the '''awk''' directory.<br><br>Let's download a database file that contains information regarding classic cars.<br><br>
# Issue the following linux command ('''copy and paste''' to save time):<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/cars.txt</nowiki></span><br><br>
# Issue the '''morecat''' command to quickly view the contents of the '''cars.txt''' file.<br>When finished, exit the more <br>The "'''print'''" action (command by pressing ) is the letter <span style="color:blue;font-weight:bold;font-family:courier;">qu>default</spanu>action of awk to print<br><br>The "all selected lines that match a '''printpattern'''" action (command) is the .<ubr>default</ubr> This '''action of awk to print all selected lines that match a pattern.<br>This action ''' (contained in braces) can provide more options <br>such as printing '''specific fields ''' of selected lines (or records) from a database.<br><br>[[Image:awk-1.png|thumb|right|400px|Using the awk command to display matches of the pattern '''ford'''.]]# Issue the following linux command all to display all lines (i.e. records ) in the "'''cars.txt" ''' database that contain matches the pattern (or "make ") called '''ford"''':<br><span style="color:blue;font-weight:bold;font-family:courier;">awk '/ford/ {print}' cars.txt</span><br><br># Issue the following linux We will use '''pipeline command all commands''' to both display records in stdout to the "cars.txt" database that contain the make "ford":screen and save to files for <u>confirmation</u> of running these pipeline commands when run a '''checking-script''' later in this investigation.<br><span style="color:blue;font-weight:bold;font-family:courier;">awk br># Issue the following linux pipeline command all to display records<br>in the ''/ford/' cars.txt | tee awk-1''' database that contain the pattern (i.e.txt</span>make) '''ford''':<br><br>What do you notice?<brspan style="color:blue;font-weight:bold;font-family:courier;">awk '/ford/' cars.txt | tee awk-1.txt<br/span>You can use variables with the "print" action for further processing. We will discuss the following variables in this tutorial:<br><br>What do you notice? You should notice ALL lines displayed <u>without</u> using '''$0'search criteria''' - Current record (entire line).<br><br>You can use '''$1builtin''' - First field in record<br>''variables'$n''with the ' - nth field in record<br>''print'NR''' - Record Number (order command for further processing.<br>We will discuss the following variables in database)this tutorial:<br> <br>[[Image:awk-2.png|thumb|right|400px|Using the awk command to print search results by '''field number''''NF.]]''' - Number of fields in current $0''' - Current record(entire line)<br>'''$1''' - First field in record<br>For a listing of more variables, please consult your course notes.<br>'''$n''' - nth field in record<br>The '''tilde characterNR''' - Record Number (order in database)<br> '''~NF''' is used to search for a pattern or display standard output for - Number of fields in current record<br><br>For a particular fieldlisting of more variables, please consult your course notes.<br><br># Issue the following linux pipeline command to display the '''model''', '''year''', '''quantity ''' and price <br>in the "'''cars.txt" ''' database for makes of "'''chevy"''':<br><span style="color:blue;font-weight:bold;font-family:courier;">awk '/chevy/ {print $2,$3,$4,$5}' cars.txt | tee awk-2.txt</span><br><br>Notice that a '''space " " ''' is the delimiter for the fields that appear as standard output.<br><br># Issue the following linux pipeline command to The '''tilde character''' '''~''' is used to search for a pattern or display all plymouths (plyms) by model name, price and quantity:<standard output for a particular field.<br><span style="color:blue;font-weight:bold;font-family:courier;"br>awk # Issue the following linux pipeline command to display all ''/chevy/ {print $2,$3,$4,$5}' cars.txt | tee awk-3.txt</span>plymouths''' ('''plym''')<br><by '''model name''', '''price''' and '''quantity''':<br>You can also use comparison operators to specify conditions for processing with matched patterns when using the <span style="color:blue;font-weight:bold;font-family:courier;">awk command. Since they are used WITHIN the awk expression'$1 ~ /plym/ {print $2, they are not confused with redirection symbols$3,$4,$5}' cars.txt | tee awk-3.txt<br/span><br> Comparison Operators:<br><br>You can also use '''<comparison operators''' Less thanto specify conditions for processing with matched patterns<br>when using the awk command. Since they are used WITHIN the awk expression,<br>'''they are not confused with redirection symbols<=''' Less than or equalbr><br>[[Image:awk-3.png|thumb|right|400px|Using the awk command to display results based on '''>''comparison operators'''.]]'''<''' Greater Less than<br>'''><=''' Greater Less than or equal<br>'''==>''' Equal Greater than<br>'''!>=''' Not Greater than or equal<br><br># Issue the following linux pipeline command to display display the car make, model number, quantity and price of all vehicles that are prices less than $5,000:<br><span style'''==''' Equal<br>'''!="color:blue''' font-weight:bold font-family:courier "Not equal<br>awk '$5 < 5000 {print $1br># Issue the following linux pipeline command to display display the '''car make''',$2'''model''',$4,$5}' cars.txt | tee awk-4.txt''quantity''' and '''price''' of all vehicles whose '''prices are less than $5,000''':</spanbr><brspan style="color:blue;font-weight:bold;font-family:courier;">awk '$5 <br5000 {print $1,$2,$4,$5}' cars.txt | tee awk-4.txt</span>What <br><br>What do you notice?<br><br># Issue the following linux pipeline command to display display the car make, model number'''price''', <br>'''quantity and price ''', '''model''' and '''car make''' of all vehicles that whose '''prices are prices less than $5,000''':<br><span style="color:blue;font-weight:bold;font-family:courier;">awk '$5 < 5000 {print $15,$24,$42,$51}' cars.txt | tee awk-5.txt</span><br><br>The symbol tilde # Issue the following linux pipeline command to display the '''~car make''' is used to match a pattern for a particular field number.<br,<br><br># Issue the following linux pipeline command to display the car make, year and '''year''' and '''quantity ''' of all car makes cars that '''begin ''' with the '''letter 'f'''':<br><span style="color:blue;font-weight:bold;font-family:courier;">awk '$1 ~ /^f/ {print $1,$2,$4}' cars.txt | tee awk-6.txt</span><br><br>Compound criteria symbols can be used [[Image:awk-4.png|thumb|right|400px|Using the awk command to join display combined search statements together<br><results based on '''compound operators'''.]]Combined pattern searches can be made<br>Compound Operatorsby using '''compound operator''' symbols:<br><br>'''&&''' (and)<br>'''||''' (or)<br><br># Issue the following linux pipeline command to list all "'''fords" that are greater '''<br>whose '''price is greater than $10,000 in price''':<br><span style="color:blue;font-weight:bold;font-family:courier;">awk '$1 ~ /ford/ && $5 > 10000 {print $0}' cars.txt | tee awk-7.txt</span><br><br># After you complete Issue the Review Questions sections to get additional practice, then work on your following linux command ('''online assignment 3,copy and paste'''<to save time):<br>'''sections 4 to 6''' labelled<span style="color:blue;font-weight: bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/cars2.txt</nowiki></span><br><br># Issue the '''More Scripting (add)''', '''Yet More Scripting (oldfiles)''', and 'cat''sed And awk'' command to quickly view the contents of the '''cars2.txt''' file.<br><br># Issue the following linux pipeline command to display the '''year'''<br>and '''quantity''' of cars that '''begin''' with the '''letter 'f'''' for the '''cars2.txt''' database:<br><span style="color:blue;font-weight:bold;font-family:courier;">awk '$1 ~ /^f/ {print $2,$4}' cars2.txt | tee awk-8.txt</span><br><br>What did you notice?<br><br>The problem is that the '''cars2.txt''' database separates each field by a semi-colon (''';''') <u>instead</u> of '''TAB'''.<br>Therefore, it does not recognize the second and fourth fields.<br><br>You need to issue awk with the -F option to indicate that this file's fields are separated (delimited) by a semi-colorn.<br><br># Issue the following linux pipeline command to display the '''year'''<br>and '''quantity''' of cars that '''begin''' with the '''letter 'f'''' for the '''cars2.txt''' database:<br><span style="color:blue;font-weight:bold;font-family:courier;">awk -F";" '$1 ~ /^f/ {print $2,$4}' cars2.txt | tee awk-9.txt</span><br><br>What did you notice this time?<br><br># Issue the following to run a checking script:<br><span style="color:blue;font-weight:bold;font-family:courier;">~uli101/week11-check-2</span><br><br>If you encounter errors, make corrections and '''re-run''' the checking script until you<br>receive a congratulations message, then you can proceed.<br><br>
= LINUX PRACTICE QUESTIONS =
