13,420
edits
Changes
no edit summary
=INTRODUCTION TO SHELL SCRIPTINGUSING SED & AWK UTILTIES=
<br>
===Main Objectives of this Practice Tutorial===
:* Understand Use the process for planning prior '''sed''' command to writing '''manipulate text''' contained in a shell scriptfile.
:* Understand List and explain several '''addresses''' and '''instructions''' associated with the purpose of a she-bang line contained at the top of a shell script'''sed''' command.
:* Setting permissions for Use the '''sed''' command as a shell script and properly execute a shell script'''filter''' with Linux pipeline commands.
:* Understand and use environment and user-defined variables within Use the '''awk''' command to '''manipulate text''' contained in a shell scriptfile.
:* Understand List and explain '''comparison operators''', '''variables''' and '''actions''' associated with the purpose of control flow statements used with shell scripts'''awk''' command.
:* Use the test '''awk''' command to test various conditionsas a '''filter''' with Linux pipeline commands. :* Use the if logic statement and the for loop statement within shell scripts.<br><br>
===Tutorial Reference Material===
|- valign="top" style="padding-left:15px;"
|colspan="2" |Course Notes'''Slides''':<ul><li>Week 11 Lecture 1 Notes:<br> [[Media:ULI101-Week11.1.pdf | PDF]] | [https://ictmatrix.senecacollege.ca/~murraychris.sauljohnson/uli101ULI101/ULI101-Week11.1.pptx PPTX]</li><li>Week 11 Lecture 2 Notes:<br> [[Media:ULI101-Week10Week11.2.pdf | PDF]] | [https://ictmatrix.senecacollege.ca/~murrayjason.saulcarman/uli101slides/ULI101-Week10Week11.2.pptx PPTX]<br></li></ul>
| style="padding-left:15px;" |Shell Scripting'''Text Manipulation:'''* [https://www.digitalocean.com/community/tutorials/the-basics-of-using-the-sed-stream-editor-to-manipulate-text-in-linux Purposeof using the sed utility]* Planning a Shell Script* Executing a Shell Script<br>Variables* Environment * User Defined[https://www.digitalocean.com/community/tutorials/how-to-use-the-awk-language-to-manipulate-text-in-linux Purpose of using the awk utility]
| style="padding-left:15px;"|Control Flow Statements'''Commands:'''* Purpose[https://man7.org/linux/man-pages/man1/sed.1p.html sed]* test command* if statement* for loop[https://man7.org/linux/man-pages/man1/awk.1p.html awk]
|colspan="1" style="padding-left:15px;" width="30%"|'''Brauer Instructional Videos:'''<ul><li>[https://www.youtube.com/watch?v=kxEP-KUhOSgnpU6S61AIko&list=PLU1b1f-2Oe90TuYfifnWulINjMv_Wr16N&index=5 Introduction to Shell Scripting14 Using the sed Utility]</li><li>[https://www.youtube.com/watch?v=XVTwbINXnk4OV3XzjDYgJo&list=PLU1b1f-2Oe90TuYfifnWulINjMv_Wr16N&index=6 13 Using Variables and Control Flow Statements in Shell Scriptingthe awk Utility]</ul>
|}
= KEY CONCEPTS =
[[Image:shebangsed.png|thumb|right|200px500px|The ]]'''shebang lineInstruction:''' <u>must</u> appear on the *'''first line''' and at the '''beginningAction''' to take for matched line(s)*Refer to table on right-side for list of the shell script.]]If you are learning Bash scripting by reading other people’s code you might have noticedsome<br>that the first line in the scripts starts with the #! characters and the path to the Bash interpreter.This sequence of characters (#!) is called '''shebangcommon instructions''' and is used to tell the operating systemand their purpose<br><br>which interpreter to use to parse the rest of the file. Reference: https://linuxize.com/post/bash-shebang/
'''Setting Permissions & Running a Shell ScriptUsage:'''
* The '''Definitionawk''' command reads all lines in the input file and will be exposed to the expression (contained within quotes) for processing.*The '''expression''' (contained in quotes) represents '''selection criteria''', and '''action''' to execute contained within braces '''{}'''* if selection criteria is matched, then action (between braces) is executed.* The '''–F''' option can be used to specify the default '''field delimiter''' (separator) character<br>eg. '''awk –F”;”''' (would indicate a semi-colon delimited input file).<br>'''Selection Criteria'''
* You can use a regular expression, enclosed within slashes, as a pattern. For example: '''/pattern/'''* The ~ operator tests whether a field or variable matches a regular expression. For example: '''$1 ~ /^[0-9]/'''* The '''!~''' operator tests for no match. For example: '''$2 !~ /line/'''* You can perform both numeric and string comparisons using relational operators ( '''>''' , '''>=''' , '''<''' , '''<=''' , '''==''' , '''Variables!=''' are used to ).* You can combine any of the patterns using the Boolean operators '''store information||''' to be referenced (OR) and manipulated '''&&''' (AND).* You can use built-in a computer programvariables (like NR or "record number" representing line number) with comparison operators.<br>They also provide a way of labeling data with a descriptive name, so our programs can be understoodFor example: '''NR >=1 && NR <br>more clearly by the reader and ourselves.= 5''' <br>'''ReferenceAction (execution):''' https://launchschool.com/books/ruby/read/variables
* Action to be executed is contained within braces '''{}'''
* The '''print''' command can be used to display text (fields).
* You can use parameters which represent fields within records (lines) within the expression of the awk utility.
* The parameter '''$0''' represents all of the fields contained in the record (line).
* The parameters '''$1''', '''$2''', '''$3''' … '''$9''' represent the first, second and third to the 9th fields contained within the record.
* Parameters greater than nine requires the value of the parameter to be placed within braces (for example: '''${10}''','''${11}''','''${12}''', etc.)
* You can use built-in '''variables''' (such as '''NR''' or "record number" representing line number)<br>eg. '''{print NR,$0}''' (will print record number, then entire record).
'''User Defined VariablesPerform the Following Steps:'''
# '''UserLogin''' to your matrix account and confirm you are located in your '''home''' directory.<br><br># Issue a Linux command to create a directory called '''sed'''<br><br># Issue a Linux command to <u>change</u> to the '''sed''' directory and confirm that you are located in the '''sed''' directory.<br><br># Issue the following Linux command to download the data.txt file<br>('''copy and paste''' to save time):<br><span style="color:blue;font-weight:bold;font-family:courier;">wget <nowiki>https://ict.senecacollege.ca/~murray.saul/uli101/data.txt</nowiki></span><br><br># Issue the '''more''' command to quickly view the contents of the '''data.txt''' file.<br>When finished, exit the more command by pressing the letter <span style="color:blue;font-weight:bold;font-family:courier;">q</span>[[Image:sed-defined variables1.png|thumb|right|300px|Issuing the '''p''' instruction without using the '''-n''' option (to suppress original output) will display lines twice.]]<br><br>The '''p''' instruction with the '''sed''' command is used to<br>'''print''' (i.e. ''display'' are variables which can be ) the contents of a text file.<br><br># Issue the following Linux command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed 'p' data.txt</span><br><br>'''NOTE: You should notice that each line appears twice'''.<br><br>The reason why standard output appears twice is that the sed command<br>(without the '''-n option''') displays all lines regardless of an address used.<br><br>We will use '''pipeline commands''' to both display stdout to the screen and save to files<br>for <u>confirmation</u> of running these pipeline commands when run a '''created by the userchecking-script''' and exist later in this investigation.<br><br># Issue the sessionfollowing Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed -n 'p' data.txt | tee sed-1.txt</span><br><br>This means that no What do you notice? You should see only one line.<br><br>You can access userspecify an '''address''' to display lines using the sed utility<br>(eg. ''line #'', '''line #s''' or range of '''line #s''').<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed -defined variables that have been set by another usern '1 p' data.txt | tee sed-2.txt</span><br><br>You should see the first line of the text file displayed.<br>What other command is used to only display the first line in a file?<br><br>[[Image:sed-2.png|thumb|right|500px|Using the sed command to display a '''range''' of lines.]]# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed -n '2,5 p' data.txt | tee sed-3.txt</span><br><br>What is displayed? How would you modify the sed command to display the line range 10 to 50?<br><br>The '''s''' instruction is used to '''substitute''' text<br>(a similar to method was demonstrated in the vi editor in tutorial 9).<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed '2,5 s/TUTORIAL/LESSON/g' data.txt | tee sed-4.txt | more</span><br><br>and when What do you notice? View the original contents of lines 2 to 5 in the '''data.txt''' file<br>in another shell to confirm that the substitution occurred.<br><br>[[Image:sed-3.png|thumb|right|500px|Using the sed command with the '''-q''' option to display up to a line number, then quit.]]The '''q''' instruction terminates or '''quits''' the execution of the session sed utility as soon as it is closed these variables expireread in a particular line or matching pattern.<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed '11 q' data.txt | tee sed-5.txt</span><br><br>What did you notice? How many lines were displayed<br>before the sed command exited?<br><br>You can use '''regular expressions''' to select lines that match a pattern. In fact,<br>the sed command was one of the <u>first</u> Linux commands that used regular expression.<br><br>The rules remain the same for using regular expressions as demonstrated in '''tutorial 9'''<br>except the regular expression must be contained within '''forward slashes'''<br>(eg. <span style="font-family:courier;font-weight:bold;">/regexp/</span> ).<br><br>[[Image:sed-4.png|thumb|right|400px|Using the sed command using regular expressions with '''anchors'''.]]# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed -n '/^The/ p' data.txt | tee sed-6.txt</span><br><br>What do you notice?<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">sed -n '/d$/ p' data.txt | tee sed-7.txt</span><br><br>What do you notice?<br><br>The '''sed''' utility can also be used as a '''filter''Reference' to manipulate text that<br>was generated from Linux commands.<br><br>[[Image:sed-5.png|thumb|right|400px|Using the sed command with '''pipeline' https'' commands.]]# Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">who | sed -n '/^[a-m]/mariadbp' | tee sed-8.comtxt | more</kbspan><br><br>What did you notice?<br><br># Issue the following Linux pipeline command:<br><span style="color:blue;font-weight:bold;font-family:courier;">ls | sed -n '/entxt$/userp' | tee sed-defined9.txt</span><br><br>What did you notice?<br><br># Issue the following to run a checking script:<br><span style="color:blue;font-weight:bold;font-variablesfamily:courier;">~uli101/week11-check-1</span><br><br>If you encounter errors, make corrections and '''re-run''' the checking script<br>until you receive a congratulations message, then you can proceed.<br><br>
Note the contents from the following tab-delimited file called '''~murray.saul/uli101/stuff.txt''':
(this file pathname exists for checking your work)
<pre>
Line one.
This is the second line.
This is the third.
This is line four.
Five.
Line six follows
Followed by 7
Now line 8
and line nine
Finally, line 10
</pre>
Write a single Linux command to perform the specified tasks for each of the following questions.
# Write a Linux sed command to display only lines 5 to 9 for the file: '''~murray.saul/uli101/stuff.txt'''<br><br># Write a Linux sed command to display only lines the begin the pattern “and” for the file: '''~murray.saul/uli101/stuff.txt'''<br><br># Write a Linux sed command to display only lines that end with a digit for the file: ''Perform '~murray.saul/uli101/stuff.txt'''<br><br># Write a Linux sed command to save lines that match the pattern “line” (upper or lowercase) for the Following Stepsfile:'''~murray.saul/uli101/stuff.txt''' and save results (overwriting previous contents) to: '''~/results.txt'''<br><br>
'''Write the results of each of the following Linux commands for the above-mentioned file:'''
# x<span style="font-family:courier;font-weight:bold">awk ‘NR == 3 {print}’ ~murray.saul/uli101/stuff.txt</span><br><br># <span style="font-family:courier;font-weight:bold">awk ‘NR >= 2 && NR <= 5 {print}’ ~murray.saul/uli101/stuff.txt</span><br><br># <span style="font-family:courier;font-weight:bold">awk ‘$1 ~ /This/ {print $2}’ ~murray.saul/uli101/stuff.txt</span><br><br># <span style="font-family:courier;font-weight:bold">awk ‘$1 ~ /This/ {print $3,$2}’ ~murray.saul/uli101/stuff.txt</span><br><br>
# Write a Linux awk command to display all records for the file: '''~/cars''' whose fifth field is greater than 10000.<br><br>
# Write a Linux awk command to display the first and fourth fields for the file: '''~/cars''' whose fifth field begins with a number.<br><br>
# Write a Linux awk command to display the second and third fields for the file: '''~/cars''' for records that match the pattern “chevy”.<br><br>
# Write a Linux awk command to display the first and second fields for all the records contained in the file: '''~/cars'''<br><br>
[[Category:ULI101]]