OPS435 Python Lab 4
OBJECTIVES
- This lab will provide you will additional scripting tools to help us write even more effective Python scripts to be applied to practical application involving VM management and deployment in future labs.
- The first investigation in this lab will focus on Data Structures. In Wikipedia (http://searchsqlserver.techtarget.com/definition/data-structure)
"A data structure is defined as a specialized format for organizing and storing data. Any data structure is designed to organize data to suit a specific purpose so that it can be accessed and worked with in appropriate ways."
- Each data structure has its own advantages and limitations. This lab will emphasize the most important differences as they relate to Python scripting.
- The second investigation will focus closely on strings. You have been using and storing strings since our first class, however in this lab we will dive into the more complex nature of string manipulation. Finally, this lab will cover how to use a variety of different regular expression functions, for searching and input validation.
PYTHON REFERENCE
- As you develop your Python scripting skills, you may start to be "overwhelmed" with the volume of information that you have absorbed over these labs. One way to help, is to write what you have learned in your labs into your lab logbook. Also, in programming, it is important to use online references in order to obtain information regarding Python scripting techniques and tools.
- Below is a table with links to useful online Python reference sites (by category). You may find these references useful when performing assignments, etc.
Data Structures | Lists & List Comprehension | Strings | Regular Expressions | Miscellaneous |
INVESTIGATION 1: DATA STRUCTURES
- In this investigation, you will learn several tools when using data structures in Python scripting.
- These tools include tuples, sets, dictionaries, and more advanced list functions.
PART 1 - Tuples
- Many often confuse a tuple with a list (which you learned about in a previous lab). A tuple is a type of list whose values cannot be changes. In fact, the structure of a tuple cannot be changed (like adding, removing list elements).
- There are many advantages to using tuples when creating Python scripts:
- Data protection (eg. values are are NOT allowed to change like income tax rate, social insurance number, etc)
- The data structure in a tuple cannot be changed (eg. structure cannot be corrupted)
- Tuples can be used as keys in data dictionaries (which are NOT allowed to change)
- Tuples allow for faster access than lists
- Term to indicate that a data structure cannot be changed is called immutable (as opposed to "mutable" which means the data structure can be changed).
- Perform the Following Steps:
- Launch your ipython3 shell:Let's create two tuples, so we can learn how to use them and learn how they differ from lists.
ipython3
Note: tuples are defined by using parenthesis ( ) as opposed to lists are defined by using square brackets [ ] - Issue the following:
t1 = ('Prime', 'Ix', 'Secundus', 'Caladan') t2 = (1, 2, 3, 4, 5, 6)
- Values from a tuple can be retrieved in the same way as a list. For example, issue the following:
t1[0] t2[2:4]
- You can also check to see whether a value exists inside a tuple or not. To demonstrate, issue the following:Let's now see how a tuple differs from a list. We will now create a list and note the difference between them.
'Ix' in t1 'Geidi' in t1
- Issue the following to create a list:
list2 = [ 'uli101', 'ops235', 'ops335', 'ops435', 'ops535', 'ops635' ]
- See if you can change the value of your list by issuing the following:.You should have been successful in changing the value of your list.
list2[0]= 'ica100' list2[0] print(list2)
- Now, try changing the value of your previously-created tuple by issuing:Did it work? Once created the tuple values will not be able to change.
t2[1] = 10
If you would like a tuple with different values than the tuple you currently have, then you must create a new one. - To create a new tuple, issue the following:
t3 = t2[2:3]
- You can use most of the basic operations with tuples as you did with lists.
- To demonstrate, issue the following:
len(t1) # list the length of the tuple t1 * 3 # repetition t1 + t2 # concatenation, remember this is creating a new tuple, not modifying
- Also, as with lists, you can use loops with tuples. Issue the following to demonstrate:
for item in t1: print('item: ' + item)
- Launch your ipython3 shell:
PART 2 - Sets
- So far, you have been exposed to two structures that are used to contain data: lists and tuples. You can modify the values within a list as well as modify the structure of a list (i.e. add and remove elements), whereby you cannot with a tuple.
- In this section, you will learn about sets. A set has similar characteristics as a list, but there are two major differing characteristics:
- Sets are un-ordered
- Sets cannot contain duplicate values
- Since new duplicate entries will be automatically removed when using sets, they are very useful for performing tasks such as comparisons: finding similarities or differences in multiple sets. Also, sets are considered to be fast!
- Perform the Following Steps:
- Within your ipython3 shell, create a few sets to work with by issuing the following:Note: Sets are defined by using braces { } as opposed to tuples that use parenthesis ( ), or lists that use square brackets [ ]
s1 = {'Prime', 'Ix', 'Secundus', 'Caladan'} s2 = {1, 2, 3, 4, 5} s3 = {4, 5, 6, 7, 8}
- Try to issue the following to access a set through the index.This should have created an error, this is not how to access data inside a set because they are un-ordered. Instead, you should use the method (used in the previous section) to check to see if a value is contained within the set.
s1[0]
- To demonstrate, issue the following:
'Ix' in s1 'Geidi' in s1
Sets can be combined, but it is important to note that any duplicate values (shared among sets) will be deleted. - Issue the following, and note the items (and values) that are common to the following sets:
s2 s3
- Now, issue the following to return a set containing only UNIQUE values (no duplicates) from both sets:Notice that both methods above provides the same result, but the first method requires less keystrokes.
s2 | s3 # returns a set containing all values from both sets s2.union(s3) # same as s2 | s3
Instead of combining sets, we can display values that are common to both sets. This is known in mathematical terms as an intersection between the lists. - To demonstrate intersection between sets s2 and s3, issue the following:
s2 & s3 # returns a set containing all values that s2 and s3 share s2.intersection(s3) # same as s2 & s3
- Sets can also have their values compared against other sets. First find out what items are in s2 but not in s3. This is also called a difference. But notice that it only shows values that s2 contains, specifically values that s3 doesn't have. So this isn't really the true difference between the sets.
s2 s3 s2 - s3 # returns a set containing all values in s2 that are not found s3 s2.difference(s3) # same as s2 - s3
- In order to see every difference between both sets, you need to find the symmetric difference. This will return a set that shows all numbers that both sets do not share together.
- To demonstrate, issue the following:These powerful features can be useful and efficient. Unfortunately, lists cannot perform these operations, unless we have to convert the lists into sets. In order to that, you should first perform a comparison, then convert the list to a set.
s2 ^ s3 # returns a set containing all values that both sets DO NOT share s2.symmetric_difference(s3) # same as s2 ^ s3
There are two problems with performing the above-mentioned technique:
- Sets are un-ordered so if the list order is important this will cause problems and remove order
- Sets cannot contain duplicate values, if the list contains any duplicate values they will be deleted.
- Within your ipython3 shell, create a few sets to work with by issuing the following:
- However, if the list does not have any of the above requirements this is a great solution to some problems.
- 10. To demonstrate, issue the following:
l2 = [1, 2, 3, 4, 5] l3 = [4, 5, 6, 7, 8] new_list = list(set(l2).intersection(set(l3))) # set() can make lists into sets. list() can make sets into lists new_list
- 10. To demonstrate, issue the following:
Create a Python Script Demonstrating Comparing Sets
- Perform the Following Instructions
- Create the ~/ops435/lab4/lab4a.py script. The purpose of this script will be to demonstrate the different way of comparing sets. There will be three functions, each returning a different set comparison.
- Use the following template to get started:
#!/usr/bin/env python3 def join_sets(set1, set2): # join_sets will return a set that has every value from both set1 and set2 inside it def match_sets(set1, set2): # match_sets will return a set that contains all values found in both set1 and set2 def diff_sets(set1, set2): # diff_sets will return a set that contains all different values which are not shared between the sets if __name__ == '__main__': set1 = set(range(1,10)) set2 = set(range(5,15)) print('set1: ', set1) print('set2: ', set2) print('join: ', join_sets(set1, set2)) print('match: ', match_sets(set1, set2)) print('diff: ', diff_sets(set1, set2))
- The match_sets() function should return a set that contains all values found in both sets
- The diff_sets() function should return a set that contains all values which are not shared between both sets
- The join_sets() function should return a set that contains all values from both sets
- All three functions should accept two arguments both are sets
- The script should show the exact output as the samples
- The script should contain no errors
- Sample Run 1:
run lab4a.py set1: {1, 2, 3, 4, 5, 6, 7, 8, 9} set2: {5, 6, 7, 8, 9, 10, 11, 12, 13, 14} join: {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14} match: {8, 9, 5, 6, 7} diff: {1, 2, 3, 4, 10, 11, 12, 13, 14}
- Sample Run 2 (with import):
import lab4a set1 = {1,2,3,4,5} set2 = {2,1,0,-1,-2} lab4a.join_sets(set1,set2) {-2, -1, 0, 1, 2, 3, 4, 5} lab4a.match_sets(set1,set2) {1, 2} lab4a.diff_sets(set1,set2) {-2, -1, 0, 3, 4, 5}
- Sample Run 1:
- 3. Exit the ipython3 shell, download the checking script and check your work. Enter the following commands from the bash shell.
cd ~/ops435/lab4/ pwd #confirm that you are in the right directory ls CheckLab4.py || wget matrix.senecac.on.ca/~acoatley-willis/CheckLab4.py python3 ./CheckLab4.py -f -v lab4a
- 4. Before proceeding, make certain that you identify any and all errors in lab4a.py. When the checking script tells you everything is OK before proceeding to the next step.
Create a Python Script Demonstrating Comparing Lists
- Perform the Following Instructions
- Create the ~/ops435/lab4/lab4b.py script. The purpose of this script will be to improve the previous script to perform the same joins, matches, and diffs, but this time on lists.
- Use the following as a template:
#!/usr/bin/env python3 def join_lists(list1, list2): # join_lists will return a list that contains every value from both list1 and list2 inside it def match_lists(list1, list2): # match_lists will return a list that contains all values found in both list1 and list2 def diff_lists(list1, list2): # diff_lists will return a list that contains all different values, which are not shared between the lists if __name__ == '__main__': list1 = list(range(1,10)) list2 = list(range(5,15)) print('list1: ', list1) print('list2: ', list2) print('join: ', join_lists(list1, list2)) print('match: ', match_lists(list1, list2)) print('diff: ', diff_lists(list1, list2))
- The match_lists() function should return a list that contains all values found in both lists
- The diff_lists() function should return a list that contains all values which are not shared between both lists
- The join_lists() function should return a list that contains all values from both sets
- All three functions should accept two arguments both are lists
- The script should show the exact output as the samples
- The script should contain no errors
- Sample Run 1:
run lab4b.py list1: [1, 2, 3, 4, 5, 6, 7, 8, 9] list2: [5, 6, 7, 8, 9, 10, 11, 12, 13, 14] join: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14] match: [8, 9, 5, 6, 7] diff: [1, 2, 3, 4, 10, 11, 12, 13, 14]
- Sample Run 2 (with import):
import lab4b list1 = [1,2,3,4,5] list2 = [2,1,0,-1,-2] join_lists(list1,list2) [0, 1, 2, 3, 4, 5, -2, -1] match_lists(list1,list2) [8, 9, 5, 6, 7] diff_lists(list1,list2) [1, 2, 3, 4, 10, 11, 12, 13, 14]
- Sample Run 2 (with import):
- 3. Exit the ipython3 shell, download the checking script and check your work. Enter the following commands from the bash shell.
cd ~/ops435/lab4/ pwd #confirm that you are in the right directory ls CheckLab4.py || wget matrix.senecac.on.ca/~acoatley-willis/CheckLab4.py python3 ./CheckLab4.py -f -v lab4b
- 4. Before proceeding, make certain that you identify any and all errors in lab4b.py. When the checking script tells you everything is OK before proceeding to the next step.
PART 3 - Dictionaries
- By now, you have probably been exposed to database terminology. For example, a database is a collection of related records. In turn, records are a collection of related fields. In order to access a record in a database, you would need to access it by key field(s). In order words, those key field(s) are a key that unlocks the access to a record within a database.
- In Python, a dictionary is a set of key-value pairs. Dictionaries are unordered, like sets, however any value can be retrieved from a dictionary if you know the key. This section will go over how to create, access, and change dictionaries, providing a new powerful tool to store and manipulate data.
- Perform the Following Steps:
- Launch the ipython3 shell:
ipython3
- Let's begin by creating a new dictionary (for practice):You should note that the syntax to define a dictionary is similar to defining sets (i.e. using {}).
dict_york = {'Address': '70 The Pond Rd', 'City': 'Toronto', 'Postal Code': 'M3J3M6'}
Unlike sets, dictionaries usekey:value
pairs within the dictionary, each key:value pair in turn, are separated by commas.
You can get help associated with your dictionary by using functions such as dir() and help(). - Issue the following and note all the available functions available and how to obtain assistance with dictionary objects:All values can be viewed by using the dictionary.values() function. This particular function provides a list containing all values.
dir(dict_york) help(dict_york)
- To demonstrate, issue the following:All keys to access the key:pair values within a dictionary can be viewed by using the dictionary.keys() function. This function provides a list containing all keys
help(dict_york.values) dict_york.values()
- To demonstrate this, issue the following:Armed with this information, We can retrieve individual values from a dictionary by provide the key associated with the key:pair value
help(dict_york.keys) dict_york.keys()
- For example, issue the following:
dict_york['Address'] dict_york['Postal Code']
- Dictionary keys can be any immutable values (i.e. not permitted for value to be changed). Types of values include: strings, numbers, and tuples. Trying adding a couple new keys and values to the dictionary by issuing:
dict_york['Country'] = 'Canada' dict_york dict_york.values() dict_york.keys()
- Let's add another key:value pair to our dictionary to change the province key:pair value to BC:WARNING: Dictionary keys must be unique. Attempting to add a key that already exists in the dictionary will overwrite the existing value for that key!
dict_york['Province'] = 'BC' dict_york dict_york.values() dict_york.keys()
- To demonstrate, issue the following:You should notice that key value for 'Province' has been changed back to 'ON'.
dict_york['Province'] = 'ON' dict_york dict_york.values() dict_york.keys()
These lists that contain the values and keys of the dictionary are not real python lists - they are "views of the dictionary" and therefore are immutable. You could change these views into usable lists by using the list() function (where the index can be used to access individual values). - For example, issue the following:
list_of_keys = list(dict_york.keys()) list_of_keys[0]
- In addition, lists can be changed into sets if we would like to perform comparisons with another set. To demonstrate, issue the following:
set_of_keys = set(dict_york.keys()) set_of_values = set(dict_york.values()) set_of_keys | set_of_values
- Lists can be used with for loops. To Demonstrate, issue the following:Additional Information regarding Dictionaries:
list_of_keys = list(dict_york.keys()) for key in list_of_keys: print(key) for value in dict_york.values() print(value)
- The values and keys can be looped over using the index as well
- The range() function provides a list of numbers in a range.
- The len() function provides a the number of items in a list.
- Used together len() and range() can be used to create a list of usable indexes for a specific list
Let's construct a table using lists created to store our dictionary data. First, we need to pair the keys and values of two separate lists. - Issue the following:Now, let's use these newly-created lists, len() & range() functions with a for loop to construct our table:
list_of_keys = list(dict_york.keys()) list_of_values = list(dict_york.values()) list_of_indexes = range(0, len(dict_york.keys())) list_of_indexes list_of_keys[0] list_of_values[0]
- Issue the following:
list_of_keys = list(dict_york.keys()) list_of_values = list(dict_york.values()) for index in range(0, len(list_of_keys)): print(list_of_keys[index] + '--->' + list_of_values[index])
- Looping using indexes is not the best way to loop through a dictionary. A new dictionary could be created using this method, but this is not good:
list_of_keys = list(dict_york.keys()) list_of_values = list(dict_york.values()) new_dictionary = {} for index in range(0, len(list_of_keys)): new_dictionary[list_of_keys[index]] = list_of_values[index]
- The above method uses a lot of memory and loops. The best method to create a dictionary from two lists is to use the zip() function:
list_of_keys = list(dict_york.keys()) list_of_values = list(dict_york.values()) new_dictionary = dict(zip(list_of_keys, list_of_values))
- Looping through the keys in a dictionary also provides a easy way to get the value for each key at the same time:
for key in dict_york.keys(): print(key + '--->' + dict_york[key])
- An alternative (possibly more efficient) method would be to cause both the key and its value to be extracted into a single (using a for loop, and using a special object):
for key, value in dict_york.items(): print(key + ' | ' + value)
- Launch the ipython3 shell:
- Perform the Following Steps:
Create a Python Script for Managing Dictionaries
- Perform the Following Instructions
- Create the ~/ops435/lab4/lab4c.py script. The purpose of this script will be to create dictionaries, extract data from dictionaries, and to make comparisons between dictionaries.
- Use the following as a template:
#!/usr/bin/env python3 # Dictionaries dict_york = {'Address': '70 The Pond Rd', 'City': 'Toronto', 'Country': 'Canada', 'Postal Code': 'M3J3M6', 'Province': 'ON'} dict_newnham = {'Address': '1750 Finch Ave E', 'City': 'Toronto', 'Country': 'Canada', 'Postal Code': 'M2J2X5', 'Province': 'ON'} # Lists list_keys = ['Address', 'City', 'Country', 'Postal Code', 'Province'] list_values = ['70 The Pond Rd', 'Toronto', 'Canada', 'M3J3M6', 'ON'] def create_dictionary(keys, values): # Place code here def split_dictionary(dictionary): # Place code here def shared_values(dict1, dict2): # Place code here if __name__ == '__main__': york = create_dictionary(list_keys, list_values) print('York: ', york) keys, values = split_dictionary(dict_newnham) print('Newnham Keys: ', keys) print('Newnham Values: ', values) keys, values = split_dictionary(york) print('York Keys: ', keys) print('York Values: ', values) common = shared_values(dict_york, dict_newnham) print('Shared Values', common)
- The script should contain three functions:
- create_dictionary()
- accepts two lists as arguments keys and values, combines these lists together to create a dictionary
- returns a dictionary that has the keys and associated values from the lists
- create_dictionary()
- split_dictionary() accepts a single dictionary as a argument and splits the dictionary into two lists, keys and values
- split_dictionary() returns two lists: return keys, values
- shared_values() accepts two dictionaries as arguments finds all values that are shared between the two dictionaries
- shared_values() returns a set containing ONLY values found in BOTH dictionaries
- make sure the functions have the correct number of arguments required
- The script should show the exact output as the samples
- The script should contain no errors
- Sample Run 1:
run lab4c.py York: {'Country': 'Canada', 'Postal Code': 'M3J3M6', 'Address': '70 The Pond Rd', 'Province': 'ON', 'City': 'Toronto'} Newnham Keys: ['Country', 'Postal Code', 'Address', 'Province', 'City'] Newnham Values: ['Canada', 'M2J2X5', '1750 Finch Ave E', 'ON', 'Toronto'] York Keys: ['Country', 'Postal Code', 'Address', 'Province', 'City'] York Values: ['Canada', 'M3J3M6', '70 The Pond Rd', 'ON', 'Toronto'] Shared Values {'Canada', 'ON', 'Toronto'}
- Sample Run 2(with import):
import lab4c dict_york = {'Address': '70 The Pond Rd', 'City': 'Toronto', 'Country': 'Canada', 'Postal Code': 'M3J3M6', 'Province': 'ON'} dict_newnham = {'Address': '1750 Finch Ave E', 'City': 'Toronto', 'Country': 'Canada', 'Postal Code': 'M2J2X5', 'Province': 'ON'} list_keys = ['Address', 'City', 'Country', 'Postal Code', 'Province'] list_values = ['70 The Pond Rd', 'Toronto', 'Canada', 'M3J3M6', 'ON'] york = create_dictionary(list_keys, list_values) york {'Address': '70 The Pond Rd', 'City': 'Toronto', 'Country': 'Canada', 'Postal Code': 'M3J3M6', 'Province': 'ON'} keys, values = split_dictionary(dict_newnham) keys ['Country', 'Postal Code', 'Address', 'Province', 'City'] values ['Canada', 'M2J2X5', '1750 Finch Ave E', 'ON', 'Toronto'] keys, values = split_dictionary(york) keys ['Country', 'Postal Code', 'Address', 'Province', 'City'] values ['Canada', 'M3J3M6', '70 The Pond Rd', 'ON', 'Toronto'] common = shared_values(dict_york, dict_newnham) common {'Canada', 'ON', 'Toronto'}
- 3. Exit the ipython3 shell, download the checking script and check your work. Enter the following commands from the bash shell.
cd ~/ops435/lab4/ pwd #confirm that you are in the right directory ls CheckLab4.py || wget matrix.senecac.on.ca/~acoatley-willis/CheckLab4.py python3 ./CheckLab4.py -f -v lab4c
- 4. Before proceeding, make certain that you identify any and all errors in lab4c.py. When the checking script tells you everything is OK before proceeding to the next step.
PART 4 - List Comprehension
- We've already covered lists to a degree. Lets move into more advanced functions to use and generate lists. This is a very common practice in Python, understanding how to generate, manipulate, and apply functions to items inside a list can be incredibly useful. List comprehension is a way to build new lists from existing list and to do it faster than simply looping over lists.
- Perform the Following Steps
- Lets start with creating a list and applying some function to each item in the list. The below will print out the square of each item.
l1 = [1, 2, 3, 4, 5] for item in l1: print(item ** 2)
- To store these squares for later use, create a new list and append the squares to it. This will generate a new list that contains squared values in the same positions of the first list. This is using an existing list to create a new list.
l1 = [1, 2, 3, 4, 5] l2 = [] for item in l1: l2.append(item ** 2) l1 l2
- Move the squaring of numbers out into it's own separate function. While the squaring example is a simple function, this example could include a more complex function that does more processing on each item in the list.
def square(number): return number ** 2 l1 = [1, 2, 3, 4, 5] l2 = [] for item in l1: l2.append(square(item)) l1 l2
- The map function can be used to apply a function on each item in a list. This is exactly what happened above, however it gives much better syntax, removes the loop, including the variable that was created inside the loop. This will make the script a little more efficient while performing the same task.
def square(number): return number ** 2 l1 = [1,2,3,4,5] l2 = list(map(square, l1)) l1 l2
- The above map function requires a function, and a list. This meant that before map() could be used a function needed to be defined earlier in the script. This entire process can be avoided through the use of anonymous functions. This is the ability to create a simple function without defining it, and pass it off for use. Below we will use lambda, which will return a function, and we can use that function immediately. The function takes 1 argument x, and it will perform a single operation on x, square it.
square = lambda x: x ** 2 l1 = [1,2,3,4,5] l2 = list(map(square, l1)) l1 l2
- The above code is actually not particularly good, the whole purpose of using lambda here is we were avoiding the function definition and just quickly returning a function. However this does break down exactly what lambda does, it returns a function for use. Fix this by removing the square function and just use the return function from lambda. Now remember what map requires? map's first argument is a function, and map's second argument is a list. Here lambda will return a function and provide it as the first argument.
l1 = [1,2,3,4,5] l2 = list(map(lambda x: x ** 2, l1)) l1 l2
- Using the list comprehensions above our code will be faster and more efficient than using multiple variables and loops.
- Lets start with creating a list and applying some function to each item in the list. The below will print out the square of each item.
INVESTIGATION 2: STRINGS
- Strings are in their most basic form a list of characters, or a bit of text. Strings store text so that we can use them later. In this section we will cover more than just displaying that text to the screen. Here, we will go over cutting strings into sub-strings, joining strings together, searching through strings, and matching strings against patterns.
PART 1 - String Basics
PART 2 - String Manipulation
PART 3 - Regular Expressions
LAB 4 SIGN-OFF (SHOW INSTRUCTOR)
Practice For Quizzes, Tests, Midterm & Final Exam
- x
- x
- x