Revision as of 11:37, 5 September 2017

1 LAB OBJECTIVES
- 1.1 PYTHON REFERENCE
2 INVESTIGATION 1: Working with Files
- 2.1 PART 1 - Reading Data From Files
  - 2.1.1 Create a Python Script Demonstrating Reading Files
- 2.2 PART 2 - Writing To Files
  - 2.2.1 Create a Python Script Demonstrating Writing to Files
3 INVESTIGATION 2: Exceptions and Error Handling
- 3.1 PART 1 - Handling Errors
  - 3.1.1 Create a Python Script Which Handles Errors
4 LAB 5 SIGN-OFF (SHOW INSTRUCTOR)
5 Practice For Quizzes, Tests, Midterm & Final Exam

LAB OBJECTIVES

So far, you have created Python scripts to prompt a user to input data from the keyboard. When creating Python scripts, you may also need to be able to process large volumes of information, or store processed data for further processing. The first investigation in this lab will focus on file management, opening files, saving data to files, and reading files.

NOTE: Since many tasks that system administrators deal with files, this is a crucial skill to understand.

During the first investigation, you will be explosed to some object oriented programming elements. Object oriented programming languages (such as Python) have great advantages or other programming languages. Some advantages include greater modularity for more effective troubleshooting, the ability to re-use objects, and to provide greater flexibility to allow objects to handle unique situations from various data (for example a date in different formats).introduce the student to basic object oriented programming. To keep things simple, this investigation will focus more on the concept that objects are primarily used to store data and code

It is very important to provide logic in your Python script in case it encounters an error. An example would be an invalid path-name or trying to close a file that is already closed. The second investigation in this lab will teach you how to handle exception errors (commonly referred to as "exception handling") so that your Python script will run gracefully problems occur during execution.

PYTHON REFERENCE

In previous labs, you have been advised to make notes and use online references. This also relates to working with files and learning about objected oriented programming to help becoming "overwhelmed" with the volume of information in this lab.

Below is a table with links to useful online Python reference sites (by category). You may find these references useful when performing assignments, etc.

Category	Resource Link
Handling Errors & Exceptions	Errors & Exceptions
Built-in Exceptions	Built-in Exceptions
Data Manipulation via Files	Reading & Writing Files
Opening Files	Open File Usage
Using Classes	Classes

INVESTIGATION 1: Working with Files

You will now learn how to write Python scripts in order to open text files, to read the contents within a text file, to process the contents, and finally to write the processed contents back into a file. These operations are very common, and are used extensively in programming. Examples of file operations would include situations such as logging output, logging errors, reading and creating configuration/temporary files, etc.

Files are accessed through the use of file objects. An object is a storage location which stores data in the form of attributes (variables) and methods (functions). Creating our own objects will be covered later in investigation 3.

PART 1 - Reading Data From Files

Perform the Following Steps:

Launch your Centos VM, open a shell terminal (as a regular user) and start a new ipython3 session:
```
ipython3
```

Create a new text file in the lab5 directory:

%cd ~/ops435/lab5
%vim ~/ops435/lab5/data.txt

Place the following content inside the new file and save it:
```
Hello World
This is the second liner
Third line
Last line
```
In order to read data from a text file, we need to create a special storage area (or a "buffer") that will be used to access and storage the data in a file. This special storage has different names for various programming languages (like a "file pointer" in the C programming language or as an "object" in object oriented programming languages).
In Python, we define an object to act as a buffer to help access the data contained within the file. To simplify things for now, you can think of an object as a special variable. You will learn more about object oriented programming later in this lab.
Now lets write some python code from the ipython3 prompt to open this created file for reading. We will define and object called "f" in order to act as a buffer to help retrieve content from our text file. Issue the following:
```
f = open('data.txt', 'r')
```
The open() function takes two string arguments: a path to a file, and a mode option for reading, writing, appending, etc. The open() function will return a special object to us, this object will allow us to read the lines inside the file.
You may recall that we used the dir() function to display library information for our Python library (eg. dir(sys) ). This function can also display other elements such as attributes and methods for an object. Issue the following:
```
dir(f)
```

Although the dir(f) function displays a lot of information, we will focus on only a few elements. Separately issue the following commands in order to inspect some of the functions. attributes and methods that we can use with this file object:

help(f.read)            # help for reading all lines and stores in a string
help(f.readlines)       # help for reading all lines and stores in a list
help(f.readline)        # help for reading first line, if run a second time it will read the second line, then third
help(f.writable)        # help for determining if a file is writable
help(f.close)           # help for closing the opened file
f.writable()            # Object method (confirm if file is writable)
f.name                  # Object attribute (contains name of opened file)
f.closed                # Object attribute (confirm if file is closed)

Next, issue the following commands to read data from the buffer of the opened file and store the contents into a variable called "read_data", and then confirm the contents of the variable "read_data:
```
read_data = f.read()
read_data
```
After you have completed accessing data within a file, you should close the file in order to free up the computer resources. It is useful to first confirm that the file is still open prior to closing it.
Issue the following object attribute will provide a boolean value to confirm that the file is still open (true indicates closed and false indicates open):
```
f.closed
```
Finally, issue the following object method and object attribute to close the file, and then verify that the file has been successfully closed:
```
f.close()                               # This method will close the file
f.closed                                # Confirm that the file is closed
```
Let's take a moment to revisit the file read sequence. The following code sequence will open a file, store the contents of a file into a variable, close the file and provide confirmation that the file has been closed:

f = open('data.txt', 'r') # Open file
read_data = f.read() # Read from file
f.close() # Close file
f.closed # Confirm file is closed

Another way to read data from a file is using the with looping statement. The advantage by using the with loop is that the file will automatically close when the data within the file has been completely read

To demonstrate, issue the following code block:

with open('data.txt', 'r') as f:        # Open file
    read_data = f.read()                # Read from file
f.closed                                # Confirm file is closed

Let us take a few moments to revisit the data that has been read into the variable called "read_data".

Let's re-issue the following command:
```
read_data
```
This command displays the data from the file in a single long string. The end of each line in the file will show the special character '\n' which represents the newline character in a file used to separate lines (or records in a traditional "flat database file"). It would be convenient to split the line on the new-line characters, so each line can be stored into a separate array elements (or in our case, a list!).
First issue the following commands to get help on the split method:
```
dir(read_data)
help(read_data.split)
```
Next, issue the following commands to store the contents of our file into a list called list_of_lines:
```
read_data.split('\n')                       # Returns a list
list_of_lines = read_data.split('\n')       # Saves returned list in variable
list_of_lines
```
Although the above sequence works, there are functions and methods the we can use with our object (called "f") to place lines from our file into a list. This would help to reduce code and is considered a more common method to store multiple lines or records within a list.
Issue both method1 and method2 sequence of commands to see how you can store data into a list more efficiently:
```
# METHOD 1:

f = open('data.txt', 'r')
method1 = list(f)
f.close()
method1

# METHOD 2:

f = open('data.txt', 'r')
method2 = f.readlines()
f.close()
method2
```
Sometimes, you may need to only display retrieved data on the screen as opposed to processing the data. In these simple cases, there is no need to create extra-large variables to store the data, but rather print one line at a time onto the screen. Using this method will save computer resources while reading files.
To demonstrate, issue the following code sequence:
```
f = open('data.txt', 'r')
for line in f:
    print(line, end='')
f.close()
```
We can improve upon the previous coding sequence by using the strip() function. In the previous coding example, the python print() function by default adds the new-line character to the end of the line. Using the end=' argument used in print replaces the '\n' at the end with nothing . This allows the print() to use the new-line characters found on each line of the file that was read. Though (if desired) you can always strip the new-line characters from any lines. The strip() function will remove all leading and trailing white-space, which may help in processing some lines or data.

To demonstrate, issue the following code sequence:

f = open('data.txt', 'r')
for line in f:
    print(line.strip())
f.close()

Create a Python Script Demonstrating Reading Files

Perform the Following Instructions

Create the ~/ops435/lab5/lab5a.py script.

Use the following as a template:

#!/usr/bin/env python3

def read_file_string(file_name):
    # Takes a filename string, returns a string of all lines in the file

def read_file_list(file_name):
    # Takes a filename string, returns a list of lines without new-line characters

if __name__ == '__main__':
    file_name = 'data.txt'
    print(read_file_string(file_name))
    print(read_file_list(file_name))

This Python script will read the same file (data.txt) that you previously created
The read_file_string() function should return a string
The read_file_list() function should return a list
The read_file_list() function must remove the new-line characters from each line in the list
Both functions must accept one argument which is a string
The script should show the exact output as the samples
The script should contain no errors

Sample Run 1:

python3 lab5a.py 
Hello World
This is the second line
Third line
Last line
['Hello World', 'This is the second line', 'Third line', 'Last line']

Sample Run 2 (with import):

import lab5a        
file_name = 'data.txt'                             
lab5a.read_file_string(file_name)
'Hello World\nThis is the second line\nThird line\nLast line\n'
lab5a.read_file_list(file_name)
['Hello World', 'This is the second line', 'Third line', 'Last line']

3. Exit the ipython3 shell, download the checking script and check your work. Enter the following commands from the bash shell.

cd ~/ops435/lab5/
pwd #confirm that you are in the right directory
ls CheckLab5.py || wget matrix.senecac.on.ca/~acoatley-willis/CheckLab5.py
python3 ./CheckLab5.py -f -v lab5a

4. Before proceeding, make certain that you identify any and all errors in lab5a.py. When the checking script tells you everything is OK before proceeding to the next step.

PART 2 - Writing To Files

Up to this point, you have learned how to access text from a file. In this section, you will learn how to write text to a file. Writing data to a file is useful for creating new content in a file or updating (modifying) existing data contained within a file.

Perform the Following Steps:

Risk of Losing File Contents
A common problem that new Python programmers may encounter is to accidentally erase existing contents of a file when writing new data to a file. When opening files for writing (using the 'w' open function option), Python assumes existing content in the file is no longer wanted and it's immediately deleted; therefore, if you wish to write data to a file but keep existing content, it is recommended to use the open file option 'a' in order to append new data to a file.

To start, open the ipython3 shell:
```
ipython3
```
When opening a file for writing, the 'w' option is specified with the open() function. When the 'w' option is specified, previous contents inside the file are deleted. This deletion takes place the moment the open() function is executed as opposed to the actual writing process. If the file that is being written to doesn't exist, the file will be created upon the file opening process.
Let's open a non-existent file (called file1.txt) for writing. Issue the following command:
```
f = open('file1.txt', 'w')
```
To confirm that the new file exists and is empty, issue the following command:
```
ls -l file1.txt
```
The ls -l command should work, but if you experience any problems, place a % sign in front of the command. This represents the alias for the ls -l command (i.e. %ls -l).

To add lines of text to the file, you would use the write() method for the file object. For safe file management, always end every line with the special character '\n' to represent a "new line". Multiple lines may also be placed inside a single write operation: simply put the special character '\n' wherever a line should end.
To demonstrate adding multiple lines, issue the following command:
```
f.write('Line 1\nLine 2 is a little longer\nLine 3 is too\n')
```
Once the write() operation has been run, the final step would be to close() the file. The file MUST be closed properly or else data will not consistently be written to the file. NOTE: Not closing a file can lead to corruption or not changes being made.
Issue the following command to close our file:
```
f.close()
```
Issue the following command to view the contents of the file to make sure the write data was saved.
```
cat file1.txt  # or %cat file2.txt to display the same results as the cat command
```
You will now create a new file called file2.txt, but run multiple write() methods in a series of operations. The ability to write() multiple lines like this allows for writes to take place inside loops and more complex programs to continuously write to a file.

Issue the following commands:

f = open('file2.txt', 'w')
f.write('Line 1\nLine 2 is a little longer\nLine 3 is as well\n')
f.write('This is the 4th line\n')
f.write('Last line in file\n')
f.close()

Issue the following command to confirm that the contents were written to file2.txt:
```
cat file2.txt
```

Make Backup Copies of Your Data Files
Since you might make a mistake, and accidentally destroy file contents when writing to your file, it is highly recommended to make backup copies of your files prior to running your Python scripts. This can be particularly useful when performing any future assignment involving writing to files.

Issue the following command to backup both of your newly-created files and confirm backup:
```
cp file1.txt file1.txt.bk
cp file2.txt file2.txt.bk
ls -l file*
```
Let's demonstrate what can happen if you perform an incorrect write() operation. Issue the following commands:
```
f = open('file2.txt', 'w')
cat file2.txt
```
You should notice that the previous content in your file2.txt file was destroyed. Why do you you think the previous data was destroyed?
Issue the following commands to restore your file from the backup and verify the backup restoration:
```
cp file2.txt.bk file2.txt
cat file2.txt
```
In the event that the data in the file is important and should not be overwritten, we can append data to the end of the file instead. Use the option 'a' instead of 'w' to perform appending.
To demonstrate, issue the following commands:
```
f = open('file1.txt', 'a')
f.write('This is the 4th line\n')
f.write('Last line in file\n')
f.close()
cat file1.txt
```
The final thing to consider when writing to files is to make certain that the values being written are strings. This means that before trying to place integers, floats, lists, or dictionaries into a file, first either convert the value using str() function or extract the specific strings from items in the list.

To see how to convert numbers into strings to be stored into a file, issue the following commands:

my_number = 1000
my_list = [1,2,3,4,5]
f = open('file3.txt', 'w')
f.write(str(my_number) + '\n')
for num in my_list:
    f.write(str(num) + '\n')
f.close()

Issue the following command to confirm that the write() operation was successful.
```
cat file3.txt  # or %cat file3.txt
```

Create a Python Script Demonstrating Writing to Files

Perform the Following Instructions

Copy ~/ops435/lab5/lab5a.py script to ~/ops435/lab5/lab5b.py script (We need the previous read functions that you created).

Add the following functions below the two functions that you already created:

def append_file_string(file_name, string_of_lines):
    # Takes two strings, appends the string to the end of the file

def write_file_list(file_name, list_of_lines):
    # Takes a string and list, writes all items from list to file where each item is one line

def copy_file_add_line_numbers(file_name_read, file_name_write):
    # Takes two strings, reads data from first file, writes data to new file, adds line number to new file

Replace the main section of your Python script (i.e. the boiler-plate) near the bottom with the following:

if __name__ == '__main__':
    file1 = 'seneca1.txt'
    file2 = 'seneca2.txt'
    file3 = 'seneca3.txt'
    string1 = 'First Line\nSecond Line\nThird Line\n'
    list1 = ['Line 1', 'Line 2', 'Line 3']
    append_file_string(file1, string1)
    print(read_file_string(file1))
    write_file_list(file2, list1)
    print(read_file_string(file2))
    copy_file_add_line_numbers(file2, file3)
    print(read_file_string(file3))

append_file_string():

Takes two string arguments
Appends to the file(Argument 1) all data from the string(Argument 2)

write_file_list():

Takes two arguments: a string and a list
Writes to file(Argument 1) all lines of data found in the list(Argument 2)

copy_file_add_line_numbers():

Takes two arguments: Both are files path-names (which happen to be strings)
Reads all data from first file(Argument 1), and writes all lines into second file(Argument 2) adding line numbers
Line numbers should be added to the beginning of each line with a colon next to them(see sample output below for reference)
Hint: Review the len() and range() functions from lab 3 and lab 4

Sample Run 1:

python3 lab5b.py 
First Line
Second Line
Third Line

Line 1
Line 2
Line 3

1:Line 1
2:Line 2
3:Line 3

Sample Run 2 (with import):

import lab5b
file1 = 'seneca1.txt'
file2 = 'seneca2.txt'
file3 = 'seneca3.txt'
string1 = 'First Line\nSecond Line\nThird Line\n'
list1 = ['Line 1', 'Line 2', 'Line 3']

lab5b.append_file_string(file1, string1)

lab5b.read_file_string(file1)
'First Line\nSecond Line\nThird Line\nFirst Line\nSecond Line\nThird Line\n'

lab5b.write_file_list(file2, list1)

lab5b.read_file_string(file2)
'Line 1\nLine 2\nLine 3\n'

lab5b.copy_file_add_line_numbers(file2, file3)

lab5b.read_file_string(file3)
'1:Line 1\n2:Line 2\n3:Line 3\n'

3. Exit the ipython3 shell, download the checking script and check your work. Enter the following commands from the bash shell.

cd ~/ops435/lab5/
pwd #confirm that you are in the right directory
ls CheckLab5.py || wget matrix.senecac.on.ca/~acoatley-willis/CheckLab5.py
python3 ./CheckLab5.py -f -v lab5b

4. Before proceeding, make certain that you identify any and all errors in lab5b.py. When the checking script tells you everything is OK before proceeding to the next step.

INVESTIGATION 2: Exceptions and Error Handling

Running into errors in programming will be a common occurrence. In python when a error occurs, python raises a python object called an exception, which represents the error that occured. These exceptions are raised when python is no long able to handle what the code is trying to do. This section will give the programmer the ability to catch these exceptions when they happen and allow the program to continue running, however in many cases it might be a good idea to stop the program when an exception happens anyway.

PART 1 - Handling Errors

According to Wikipedia, Exception Handling is "the process of responding to the occurrence, during computation, of exceptions – anomalous or exceptional conditions requiring special processing – often changing the normal flow of program execution."
( https://en.wikipedia.org/wiki/Exception_handling)

There are a massive amount of exceptions (way too many to cover in this course). On the other hand, online references can be useful. If you are searching for a specific exception check out the Python Exception Documentation.

In this section, we will provide examples of how to handle a few exceptions when creating Python scripts.

Perform the Following Steps:

To start, open the ipython3 shell:
```
ipython3
```
Before attempting to handle exception errors, let's create an error, and then see how to can "handle" the exception error.

Issue the following to create an exception error:

print('5' + 10)

You should get an exception error similar to the following:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-1b929b80ca50> in <module>()
----> 1 print('5' + 10)

TypeError: Can't convert 'int' object to str implicitly

Question: According to the exception error message, what do you think caused the error?

Click on the link 'https://docs.python.org/3/library/exceptions.html#concrete-exceptions.' and scroll or search for TypeError. Take a few moments to determine what a TypeError exception error means.

You should have learned that the TypeError exception error indicates a mismatch of a type (i.e. string, int, float, list, etc). If Python doesn't know how to handle it, perhaps we could change the number into a string or change the string into a number or at least provide a more user-friendly error message.

If we want to write this program safely, we can catch/trap/handle this error while it's happening. This is done with a specific block of code called a try clause where you place code in-between the try: and the except: coding blocks. In a general sense, it works like a modified if-else statement, where the try statement acts as a test, and the except statement will or will not handle the exception depending if it occurs or does NOT occur. That is to say, If no error occurs in the code contained in the except section, the script will continue as usual but if an error occurs in the except section, then it can be handled with additional coding (like an user-friendly error message).
Let's demonstrate to handle our TypeError error by issuing code that first does not contain an error and then re-issue similar code that DOES generate an error.
Issue the following code that does NOT generate an error:
```
try:
    print(5 + 10)
except TypeError:
    print('At least one of the values is NOT an integer')

15
```
You should notice that since there was NOT an error, the Python script performed the required task.

Now, issue the following code that handles an exception error to provide user-friendly feedback that at least one of the values is not an integer:

try:
    print(5 + 'ten')
except TypeError:
    print('At least one of the values is NOT an integer')

At least one of the values is NOT an integer

Let's generate another type of error where we try to open a file that doesn't exist:
```
f = open('filethatdoesnotexist', 'r')
```
Now, to catch and handle this exception error, issue the following lines of code:
```
try:
    f = open('filethatdoesnotexist', 'r')
    f.write('hello world\n')
    f.close()
except FileNotFoundError:
    print('no file found')
```
Multiple exceptions can also be caught at the same time, such as does not exist, is a directory, or we don't have permission.

To test out the error handling code (previously issued), try removing permissions from the file, or specify a directory instead of a regular file, and then try to open it:

try:
    f = open('filethatdoesnotexist', 'r')
    f.write('hello world\n')
    f.close()
except (FileNotFoundError, PermissionError, IsADirectory): 
    print('failed to open file')

By taking the time to view the Python Exception Hierarchy, you can see how errors get caught in python. The options FileNotFoundError, PermissionError, and IsADirectory are all inherited from OSError. This means that while using more specific errors might be useful for better error messages and handling, it's not always possible to catch every error all the time.
Another way to catch multiple exceptions can be performed by issuing the following code:
```
try:
    f = open(abc, 'r')
    f.write('hello world\n')
    f.close()
except (FileNotFoundError, PermissionError): 
    print('file does not exist or wrong permissions')
except IsADirectoryError:
    print('file is a directory')
except OSError:
    print('unable to open file')
except:
    print('unknown error occured')
    raise
```
When catching multiple exceptions, make certain to catch the lowest ones contained in the exception-hierarchy first. For example, if you put 'Exception' first, both 'OSError' and 'FileNotFoundError', would never get caught.

TIP: In python it's usually best to 'try:' and 'except:' code rather than to try and determine everything that could go wrong with logic and if statements. For example, instead of checking to see if a file exists and we have read permissions, it can be better to just try and read the file and fail and catch any errors with 'OSError'.

Create a Python Script Which Handles Errors

Perform the Following Instructions

Create the ~/ops435/lab5/lab5c.py script.

Use the following as a template:

#!/usr/bin/env python3

def add(number1, number2):
    # Add two numbers together, return the result, if error return string 'error: could not add numbers'

def read_file(filename):
    # Read a file, return a list of all lines, if error return string 'error: could not read file'

if __name__ == '__main__':
    print(add(10,5))                        # works
    print(add('10',5))                      # works
    print(add('abc',5))                     # exception
    print(read_file('seneca2.txt'))         # works
    print(read_file('file10000.txt'))       # exception

Sample Run 1:

python3 lab5c.py 
15
15
error: could not add numbers
['Line 1\n', 'Line 2\n', 'Line 3\n']
error: could not read file

Sample Run 2 (with import):

import lab5c                

lab5c.add(10,5)             
15                          

lab5c.add('10',5)           
15                          

lab5c.add('10','5')         
15                          

lab5c.add('abc','5')                                                                                                                     
'error: could not add numbers'                                  

lab5c.add('hello','world')                                                                                                               
'error: could not add numbers'                                  

lab5c.read_file('seneca2.txt')
['Line 1\n', 'Line 2\n', 'Line 3\n']           

lab5c.read_file('file10000.txt')   
error: could not read file'

3. Exit the ipython3 shell, download the checking script and check your work. Enter the following commands from the bash shell.

cd ~/ops435/lab5/
pwd #confirm that you are in the right directory
ls CheckLab5.py || wget matrix.senecac.on.ca/~acoatley-willis/CheckLab5.py
python3 ./CheckLab5.py -f -v lab5c

4. Before proceeding, make certain that you identify any and all errors in lab5c.py. When the checking script tells you everything is OK before proceeding to the next step.

LAB 5 SIGN-OFF (SHOW INSTRUCTOR)

Have Ready to Show Your Instructor:

✓ Output of: ./CheckLab5.py -f -v

✓ Output of: cat lab5a.py lab5b.py lab5c.py

Practice For Quizzes, Tests, Midterm & Final Exam

x
x
x

@@ Line 432: / Line 432: @@
      raise
 </source>When catching multiple exceptions, make certain to catch the <u>lowest</u> ones contained in the exception-hierarchy first. For example, if you put 'Exception' first, both 'OSError' and 'FileNotFoundError', would never get caught.<br><br>'''TIP:''' In python it's usually best to 'try:' and 'except:' code rather than to try and determine everything that could go wrong with logic and if statements. For example, instead of checking to see if a file exists and we have read permissions, it can be better to just try and read the file and fail and catch any errors with 'OSError'.
 === Create a Python Script Which Handles Errors ===

Difference between revisions of "OPS435 Python Lab 5"

Revision as of 11:37, 5 September 2017

Contents

LAB OBJECTIVES

PYTHON REFERENCE

INVESTIGATION 1: Working with Files

PART 1 - Reading Data From Files

Create a Python Script Demonstrating Reading Files

PART 2 - Writing To Files

Create a Python Script Demonstrating Writing to Files

INVESTIGATION 2: Exceptions and Error Handling

PART 1 - Handling Errors

Create a Python Script Which Handles Errors

LAB 5 SIGN-OFF (SHOW INSTRUCTOR)

Practice For Quizzes, Tests, Midterm & Final Exam

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

get involved with CDOT

courses

course projects

links

Tools