04 Lists_Sets.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"04 Lists_Strings.ipynb","provenance":[],"collapsed_sections":[],"authorship_tag":"ABX9TyMd6HKQ+Ua000jNUsuP4Gei"},"kernelspec":{"name":"python3","display_name":"Python 3"}},"cells":[{"cell_type":"markdown","metadata":{"id":"W8B9DFMOFy28","colab_type":"text"},"source":["# Complex Data Structures\n","\n","## List\n","\n","Reading:\n","* Chapter 10: http://greenteapress.com/thinkpython2/html/thinkpython2011.html#sec114\n","* Chapter 10: https://runestone.academy/runestone/books/published/thinkcspy/Lists/toctree.html\n","\n","## Data types\n","* Primitive data types: int, float, Boolean, character;\n","* Data structures: fundamental component of programming, a collection of elements of data that adhere to certain properties, depending on the type;\n","* Python data structures are very rich. CST 1101 deals only with lists. Python strings are special types of lists (more on it later)\n","\n","## Basics of Lists\n","### Creating Lists\n","A list, sometimes called and array, or a vector, is an ordered collection of values.\n","\n","In Python, lists are specified by square brackets,` [ ]`, containing zero or more values, separated by commas.\n","\n","For example:"]},{"cell_type":"code","metadata":{"id":"FghcJhGGGREk","colab_type":"code","colab":{}},"source":["number_list = [1, 2, 3, 0, 5, 10, 11, 5, 10, 11, 1, 2, 5]\n","print(number_list)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"qDlXZZXIFxmL","colab_type":"code","colab":{}},"source":["name_list = [\"Elena\", \"John\", \"Jane\", \"Mike\"]\n","print (name_list)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"R2iallgwGXR7","colab_type":"code","colab":{}},"source":["# The items in a list usually have the same type, but that is not required.\n","mixed_list = [\"Elena\", 42, \"Jake\", 22.5]\n","print(mixed_list)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"fNsrPum6GcZw","colab_type":"code","colab":{}},"source":["empty_list = []\n","print (empty_list)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"Kl2x5aK9GgNt","colab_type":"code","colab":{}},"source":["emptu_list2 = list()\n","print (empty_list)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"VtK1LJ9QGjH9","colab_type":"text"},"source":["### Terminology\n","`x = [2, 5, -1, 3.14]`\n","\n","How do we talk about what is in a list:\n","* is an item in the list `x`\n","* is an entry in the list `x`\n","* is an element in the list `x`\n","* is an value in the list `x`\n","\n","At any point in time a list has a specific length, but this length is not either pre-defined or fixed.\n","\n","The entries in a list are accessed using elements' indecies. \n","\n","We can retrieve the value of a particular element of a list by writing in square brackets the location of the element. In Python, like most programming languages, list indices start at **0**, that is, to get the first element in a list, request the element at index **0**."]},{"cell_type":"code","metadata":{"id":"VyAiPsipGu5C","colab_type":"code","colab":{}},"source":["x = [2, 5, -1, 3.14]\n","a = x[2]\n","print(a)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"-EExmuBkGxRD","colab_type":"code","colab":{}},"source":["another_list = [\"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"a\", \"b\"]\n","print(another_list[1])"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"zntadESGGz10","colab_type":"text"},"source":["Negative indices can be used to traverse the list from the right.\n"]},{"cell_type":"code","metadata":{"id":"t5VpcUMNG16y","colab_type":"code","colab":{}},"source":["print(another_list[-2])\n","print(another_list[-3])"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"RhT-IZ1ggfii","colab_type":"text"},"source":["### Lists\n","* Why using lists? to collect the associated data into one convenient “place.”\n","* In Python a list is created when comma-separated expressions are placed between\n","square brackets.\n","* There is the list as a whole and then there are the individual items in the list which we\n","typically refer to as the elements of the list.\n","* Lists have length.\n","\n","The function `len(list)` returns the number of elements in a list.\n","\n","Pay attention: the last available index = length-1"]},{"cell_type":"code","metadata":{"id":"zNkSVFXzhAab","colab_type":"code","colab":{}},"source":["list_length = len(another_list)\n","print (list_length)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"yurP5MfKhIMF","colab_type":"code","colab":{}},"source":["another_list[8]"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ySvOrDb0hLxU","colab_type":"text"},"source":["### Lists Can be Sliced\n","\n","While indexing is used to obtain individual characters, slicing allows you to obtain parts of lists and create new lists out of an existing list.\n","\n","The full slice syntax is: **start:stop:step**. \n","* **start** refers to the index of the element which is used as a start of our slice.\n","* **stop** refers to the index of the element we should stop just before to finish our slice. \n","* **step** allows you to take each nth-element within a start:stop range."]},{"cell_type":"code","metadata":{"id":"Oxog3D-ZiCbt","colab_type":"code","colab":{}},"source":["x = [10, 20, 40, 70, 8, 6, 2]\n","slice1 = x[1:4]\n","print (slice1)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"uBaKa-1HiR14","colab_type":"code","colab":{}},"source":["# if no start is specified, then start with the very first element (element with the index 0)\n","slice2 = x[:4]\n","print (slice2)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"L0g2zfNtibZu","colab_type":"code","colab":{}},"source":["# if no stop is specified, then include into the slice all the elements following the start element\n","slice3 = x[4:]\n","print (slice3)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"IR4em2EPiqb4","colab_type":"code","colab":{}},"source":["# example with the step parameter specified (output all elements with the EVEN index, including 0)\n","slice4 = x[::2]\n","print (slice4)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"cQnXGe-ijAQ5","colab_type":"code","colab":{}},"source":["# example with the step parameter specified (output all elements with the ODD index)\n","slice4 = x[1::2]\n","print (slice4)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"suTl2CKVJKaw","colab_type":"code","colab":{}},"source":["another_list = [\"a\", \"b\", \"c\", \"d\", \"e\", \"f\"]\n","print(another_list[1])"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Wd_Ob9H_JXDY","colab_type":"text"},"source":["Negative indices can be used to traverse the list from the right.\n"]},{"cell_type":"code","metadata":{"id":"EUd0bw4cJKeJ","colab_type":"code","colab":{}},"source":["print(another_list[-2])\n"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"iFWwnRxaJKja","colab_type":"code","colab":{}},"source":["# Retrieves the elements at positions 2, 3, and 4.\n","# Remember these are the 3rd, 4th, and 5th elements of the list\n","print(another_list[2:5])"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"RuPpZ1ouJccs","colab_type":"code","colab":{}},"source":["# Retrieved the last 3 elements of the list\n","print(another_list[-3:])"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"zmLvHDOAjLTJ","colab_type":"text"},"source":["Lists are **MUTABLE**: You can change the values in a list.\n","\n","Strings are special lists that are not mutable (more on this later)."]},{"cell_type":"code","metadata":{"id":"aI8auwokjSxp","colab_type":"code","colab":{}},"source":["x = [3, 5, 7, 8, 9]\n","x[0] = 4\n","print (x)"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"30weAwrsjx-T","colab_type":"text"},"source":["## Functions that apply to lists\n","\n","* `len`: The function` len(list)` returns the number of elements in a list.\n","* `sorted`: The function `sorted(list)` Returns the list sorted.\n","* `max`: Returns the maximum element of a list\n","* `min`: Returns the minimum element of a list\n","* `sum`: The function `sum(list`) sums up all the (numeric) elements of a list"]},{"cell_type":"code","metadata":{"id":"jcPZV0umkiFW","colab_type":"code","colab":{}},"source":["names = [\"Jake\", \"John\", \"Chris\", \"Josh\", \"Mary\", \"Anna\"]"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"STm1chUjkmp-","colab_type":"code","colab":{}},"source":["print(\"Length of names list:\", len(names))\n"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"e2uBhiAekpma","colab_type":"code","colab":{}},"source":["numbers = [3, 41, 12, 9, 74, 15]\n"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"lbSkWDxjkv4m","colab_type":"code","colab":{}},"source":["print(\"Length of numbers list:\", len(numbers))\n"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"VXCDN3vBk2cn","colab_type":"code","colab":{}},"source":["print(\"Sorted List:\", sorted(names))\n","print(\"Original List:\", names)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"4oBf5lsRk38n","colab_type":"code","colab":{}},"source":["print(\"Sorted List:\", sorted(numbers))\n","print(\"Original List:\", numbers)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"Ifd-rorik8sF","colab_type":"code","colab":{}},"source":["print(\"We have \", len(numbers), \"numbers\")\n","print(\"Max number:\", max(numbers))\n","print(\"Min number:\", min(numbers))\n","print(\"Sum:\", sum(numbers))"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"b8FQvKK-k_g3","colab_type":"code","colab":{}},"source":["# Min and max also operate on strings\n","print(\"Min name:\", min(names))\n","print(\"Max name:\", max(names))"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"tK3jR5i5MwK_","colab_type":"code","colab":{}},"source":[""],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"5pUL_iEal4c7","colab_type":"text"},"source":["### Adding / Removing Elements to a List\n","\n","Appending items at the end of a list\n","* `list.append(x)`: add an element ot the end of a list\n","*  `list_1.extend(list_2)`: add all elements in the second list to the end of the first list. Alternatively, it is possible to use the + and concatenate two lists."]},{"cell_type":"code","metadata":{"id":"j6MN-bxIl2zd","colab_type":"code","colab":{}},"source":["names = [\"Jake\", \"John\", \"Chris\", \"Josh\", \"Mary\", \"Anna\"]\n","names.append(\"Blake\")\n","names.append(\"Sofia\")\n","print(names)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"16MWvnrSMwN9","colab_type":"code","colab":{}},"source":[""],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"M6upmXRvmmLh","colab_type":"text"},"source":["### Inserting and removing items in the list (any position)\n","* `list.insert(index, x)`: insert element x into the list at the specified index. Elements to the right of this index are shifted over;\n","* `list.pop(index)`: remove the element at the specified position."]},{"cell_type":"code","metadata":{"id":"o7xypQDfmx0R","colab_type":"code","colab":{}},"source":["names = [\"Jake\", \"John\", \"Chris\", \"Josh\", \"Mary\", \"Anna\"]\n","names.insert(4, \"Elena\")\n","print(names)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"vIRnw5BYm4I8","colab_type":"code","colab":{}},"source":["names.pop(3)\n","print(names)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"jyBPUb7gNAW1","colab_type":"code","colab":{}},"source":[""],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"nY3Ri1YXnByd","colab_type":"text"},"source":["### Finding items in lists\n","Common functions\n","* `x in list`: checks if x appears in the list\n","* `list.index(x)`: looks through the list to find the specified element, returning it's position if it's found, else returns an error\n","* `list.count(x)`: counts the number of occurrences of the input element"]},{"cell_type":"code","metadata":{"id":"r901ey-0nLHC","colab_type":"code","colab":{}},"source":["names = [\"Jake\", \"John\", \"Chris\", \"Josh\", \"Mary\", \"Anna\"]\n"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"VeOB0_uRnOc1","colab_type":"code","colab":{}},"source":["\"Chris\" in names\n"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"Uk8ZZx2nnPxO","colab_type":"code","colab":{}},"source":["\"Peter\" in names\n"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"9hf4We_AnUfs","colab_type":"code","colab":{}},"source":["#index\n","name = 'Chris'\n","print(\"Location of\", name, \"in the list:\", names.index(name))"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"iue2v59unXn0","colab_type":"code","colab":{}},"source":["# Count\n","names = [\"Jake\", \"John\", \"Chris\", \"Josh\", \"Mary\", \"Anna\", \"Jake\"]\n","print(\"# of John  in the list\", names.count(\"John\"))\n","print(\"# of Jake in the list\", names.count(\"Jake\"))\n","print(\"# of Peter  in the list\", names.count(\"Peter\"))"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"xMMy0RxFnrR5","colab_type":"text"},"source":["### Another way of sorting lists\n","\n","* `list.sort()`: sorts in the ascending order\n","* `list.sort(reverse = True)`: sorts in the descending order\n","\n","The value of the list variable **changes**.\n"]},{"cell_type":"code","metadata":{"id":"I49w3tBunmm7","colab_type":"code","colab":{}},"source":["numbers = [1, 4, 6, 2, 7, 234, 56]\n","numbers.sort()\n","print(numbers)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"vdHCI0EFoXjn","colab_type":"code","colab":{}},"source":["numbers = [1, 4, 6, 2, 7, 234, 56]\n","numbers.sort(reverse = True)\n","print(numbers)"],"execution_count":null,"outputs":[]},{"cell_type":"code","metadata":{"id":"VwjtxbM1NAZZ","colab_type":"code","colab":{}},"source":[""],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"FcsRqBMgqL7G","colab_type":"text"},"source":["### Multidimensional Lists\n"," A list element can be a list"]},{"cell_type":"code","metadata":{"id":"LCp9Z-saqORC","colab_type":"code","colab":{}},"source":["matrix = [[1, 2], [3, 4]]\n","print(matrix[0])\n","print(matrix[1])\n","print(matrix[0][1])"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"oqcb9--fNVE0","colab_type":"text"},"source":["## Let us see how we can use LISTs in NLP\n","\n","https://www.nltk.org/book/ch01.html"]},{"cell_type":"code","metadata":{"id":"dElVkFGzHQiU","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1600700844071,"user_tz":240,"elapsed":2709,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}}},"source":["# Setup\n","!pip install -q wordcloud\n","import wordcloud\n","\n","import nltk\n","\n"],"execution_count":3,"outputs":[]},{"cell_type":"code","metadata":{"id":"UdqCrNTNKbUK","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":1000},"executionInfo":{"status":"ok","timestamp":1600700849463,"user_tz":240,"elapsed":767,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"f93475dc-4d36-487a-9486-54731058d2ce"},"source":["nltk.download('book') \n"],"execution_count":4,"outputs":[{"output_type":"stream","text":["[nltk_data] Downloading collection 'book'\n","[nltk_data]    | \n","[nltk_data]    | Downloading package abc to /root/nltk_data...\n","[nltk_data]    |   Package abc is already up-to-date!\n","[nltk_data]    | Downloading package brown to /root/nltk_data...\n","[nltk_data]    |   Package brown is already up-to-date!\n","[nltk_data]    | Downloading package chat80 to /root/nltk_data...\n","[nltk_data]    |   Package chat80 is already up-to-date!\n","[nltk_data]    | Downloading package cmudict to /root/nltk_data...\n","[nltk_data]    |   Package cmudict is already up-to-date!\n","[nltk_data]    | Downloading package conll2000 to /root/nltk_data...\n","[nltk_data]    |   Package conll2000 is already up-to-date!\n","[nltk_data]    | Downloading package conll2002 to /root/nltk_data...\n","[nltk_data]    |   Package conll2002 is already up-to-date!\n","[nltk_data]    | Downloading package dependency_treebank to\n","[nltk_data]    |     /root/nltk_data...\n","[nltk_data]    |   Package dependency_treebank is already up-to-date!\n","[nltk_data]    | Downloading package genesis to /root/nltk_data...\n","[nltk_data]    |   Package genesis is already up-to-date!\n","[nltk_data]    | Downloading package gutenberg to /root/nltk_data...\n","[nltk_data]    |   Package gutenberg is already up-to-date!\n","[nltk_data]    | Downloading package ieer to /root/nltk_data...\n","[nltk_data]    |   Package ieer is already up-to-date!\n","[nltk_data]    | Downloading package inaugural to /root/nltk_data...\n","[nltk_data]    |   Package inaugural is already up-to-date!\n","[nltk_data]    | Downloading package movie_reviews to\n","[nltk_data]    |     /root/nltk_data...\n","[nltk_data]    |   Package movie_reviews is already up-to-date!\n","[nltk_data]    | Downloading package nps_chat to /root/nltk_data...\n","[nltk_data]    |   Package nps_chat is already up-to-date!\n","[nltk_data]    | Downloading package names to /root/nltk_data...\n","[nltk_data]    |   Package names is already up-to-date!\n","[nltk_data]    | Downloading package ppattach to /root/nltk_data...\n","[nltk_data]    |   Package ppattach is already up-to-date!\n","[nltk_data]    | Downloading package reuters to /root/nltk_data...\n","[nltk_data]    |   Package reuters is already up-to-date!\n","[nltk_data]    | Downloading package senseval to /root/nltk_data...\n","[nltk_data]    |   Package senseval is already up-to-date!\n","[nltk_data]    | Downloading package state_union to /root/nltk_data...\n","[nltk_data]    |   Package state_union is already up-to-date!\n","[nltk_data]    | Downloading package stopwords to /root/nltk_data...\n","[nltk_data]    |   Package stopwords is already up-to-date!\n","[nltk_data]    | Downloading package swadesh to /root/nltk_data...\n","[nltk_data]    |   Package swadesh is already up-to-date!\n","[nltk_data]    | Downloading package timit to /root/nltk_data...\n","[nltk_data]    |   Package timit is already up-to-date!\n","[nltk_data]    | Downloading package treebank to /root/nltk_data...\n","[nltk_data]    |   Package treebank is already up-to-date!\n","[nltk_data]    | Downloading package toolbox to /root/nltk_data...\n","[nltk_data]    |   Package toolbox is already up-to-date!\n","[nltk_data]    | Downloading package udhr to /root/nltk_data...\n","[nltk_data]    |   Package udhr is already up-to-date!\n","[nltk_data]    | Downloading package udhr2 to /root/nltk_data...\n","[nltk_data]    |   Package udhr2 is already up-to-date!\n","[nltk_data]    | Downloading package unicode_samples to\n","[nltk_data]    |     /root/nltk_data...\n","[nltk_data]    |   Package unicode_samples is already up-to-date!\n","[nltk_data]    | Downloading package webtext to /root/nltk_data...\n","[nltk_data]    |   Package webtext is already up-to-date!\n","[nltk_data]    | Downloading package wordnet to /root/nltk_data...\n","[nltk_data]    |   Package wordnet is already up-to-date!\n","[nltk_data]    | Downloading package wordnet_ic to /root/nltk_data...\n","[nltk_data]    |   Package wordnet_ic is already up-to-date!\n","[nltk_data]    | Downloading package words to /root/nltk_data...\n","[nltk_data]    |   Package words is already up-to-date!\n","[nltk_data]    | Downloading package maxent_treebank_pos_tagger to\n","[nltk_data]    |     /root/nltk_data...\n","[nltk_data]    |   Package maxent_treebank_pos_tagger is already up-\n","[nltk_data]    |       to-date!\n","[nltk_data]    | Downloading package maxent_ne_chunker to\n","[nltk_data]    |     /root/nltk_data...\n","[nltk_data]    |   Package maxent_ne_chunker is already up-to-date!\n","[nltk_data]    | Downloading package universal_tagset to\n","[nltk_data]    |     /root/nltk_data...\n","[nltk_data]    |   Package universal_tagset is already up-to-date!\n","[nltk_data]    | Downloading package punkt to /root/nltk_data...\n","[nltk_data]    |   Package punkt is already up-to-date!\n","[nltk_data]    | Downloading package book_grammars to\n","[nltk_data]    |     /root/nltk_data...\n","[nltk_data]    |   Package book_grammars is already up-to-date!\n","[nltk_data]    | Downloading package city_database to\n","[nltk_data]    |     /root/nltk_data...\n","[nltk_data]    |   Package city_database is already up-to-date!\n","[nltk_data]    | Downloading package tagsets to /root/nltk_data...\n","[nltk_data]    |   Package tagsets is already up-to-date!\n","[nltk_data]    | Downloading package panlex_swadesh to\n","[nltk_data]    |     /root/nltk_data...\n","[nltk_data]    |   Package panlex_swadesh is already up-to-date!\n","[nltk_data]    | Downloading package averaged_perceptron_tagger to\n","[nltk_data]    |     /root/nltk_data...\n","[nltk_data]    |   Package averaged_perceptron_tagger is already up-\n","[nltk_data]    |       to-date!\n","[nltk_data]    | \n","[nltk_data]  Done downloading collection book\n"],"name":"stdout"},{"output_type":"execute_result","data":{"text/plain":["True"]},"metadata":{"tags":[]},"execution_count":4}]},{"cell_type":"code","metadata":{"id":"_qKxEu4KOQzt","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":242},"executionInfo":{"status":"ok","timestamp":1600700910167,"user_tz":240,"elapsed":5479,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"7554977b-d033-49f0-f616-7cb7fd10e80d"},"source":["from nltk.book import *"],"execution_count":5,"outputs":[{"output_type":"stream","text":["*** Introductory Examples for the NLTK Book ***\n","Loading text1, ..., text9 and sent1, ..., sent9\n","Type the name of the text or sentence to view it.\n","Type: 'texts()' or 'sents()' to list the materials.\n","text1: Moby Dick by Herman Melville 1851\n","text2: Sense and Sensibility by Jane Austen 1811\n","text3: The Book of Genesis\n","text4: Inaugural Address Corpus\n","text5: Chat Corpus\n","text6: Monty Python and the Holy Grail\n","text7: Wall Street Journal\n","text8: Personals Corpus\n","text9: The Man Who Was Thursday by G . K . Chesterton 1908\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"PMS1oJzJKbIf","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1600700912472,"user_tz":240,"elapsed":338,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}}},"source":["# Get stopwords, stemmer and lemmatizer\n","stopwords = nltk.corpus.stopwords.words('english')\n","stemmer = nltk.stem.PorterStemmer()\n","lemmatizer = nltk.stem.WordNetLemmatizer()\n"],"execution_count":6,"outputs":[]},{"cell_type":"code","metadata":{"id":"xkUsB2KDKbOa","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700915734,"user_tz":240,"elapsed":275,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"8c715449-f4e9-4b2b-d470-4ac74ee051af"},"source":["type(stopwords)"],"execution_count":7,"outputs":[{"output_type":"execute_result","data":{"text/plain":["list"]},"metadata":{"tags":[]},"execution_count":7}]},{"cell_type":"code","metadata":{"id":"vMOnL3KuKsRu","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":35},"executionInfo":{"status":"ok","timestamp":1600700917410,"user_tz":240,"elapsed":343,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"43cac5f7-9447-4a3c-d847-feca5a1e515e"},"source":["stopwords[0]"],"execution_count":8,"outputs":[{"output_type":"execute_result","data":{"application/vnd.google.colaboratory.intrinsic+json":{"type":"string"},"text/plain":["'i'"]},"metadata":{"tags":[]},"execution_count":8}]},{"cell_type":"code","metadata":{"id":"cDxgXFxLKbRM","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":54},"executionInfo":{"status":"ok","timestamp":1600700918960,"user_tz":240,"elapsed":308,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"5e253f3f-162c-4c9f-8d8c-05741efa7c85"},"source":["print(stopwords)"],"execution_count":9,"outputs":[{"output_type":"stream","text":["['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', \"you're\", \"you've\", \"you'll\", \"you'd\", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', \"she's\", 'her', 'hers', 'herself', 'it', \"it's\", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', \"that'll\", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', \"don't\", 'should', \"should've\", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', \"aren't\", 'couldn', \"couldn't\", 'didn', \"didn't\", 'doesn', \"doesn't\", 'hadn', \"hadn't\", 'hasn', \"hasn't\", 'haven', \"haven't\", 'isn', \"isn't\", 'ma', 'mightn', \"mightn't\", 'mustn', \"mustn't\", 'needn', \"needn't\", 'shan', \"shan't\", 'shouldn', \"shouldn't\", 'wasn', \"wasn't\", 'weren', \"weren't\", 'won', \"won't\", 'wouldn', \"wouldn't\"]\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"iCgbcs7pKbWP","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700921843,"user_tz":240,"elapsed":307,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"05f3d364-19d8-48e2-df8b-d3b31a031f20"},"source":["sent1 = ['Call', 'me', 'Ishmael', '.']\n","sent1"],"execution_count":10,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['Call', 'me', 'Ishmael', '.']"]},"metadata":{"tags":[]},"execution_count":10}]},{"cell_type":"code","metadata":{"id":"8-gJFqwZKbcx","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700924566,"user_tz":240,"elapsed":342,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"c9b540b1-cc97-4df1-8218-0fe618bda94f"},"source":["len(sent1)"],"execution_count":11,"outputs":[{"output_type":"execute_result","data":{"text/plain":["4"]},"metadata":{"tags":[]},"execution_count":11}]},{"cell_type":"code","metadata":{"id":"T4IxtzszOQmu","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1600700941904,"user_tz":240,"elapsed":307,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}}},"source":["sent2 = ['The', 'family', 'of', 'Dashwood', 'had', 'long', 'been', 'settled', 'in', 'Sussex', '.']\n","sent3 = ['In', 'the', 'beginning', 'God', 'created', 'the', 'heaven', 'and', 'the', 'earth', '.']"],"execution_count":12,"outputs":[]},{"cell_type":"code","metadata":{"id":"aRpOEOLyOQrE","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":399},"executionInfo":{"status":"ok","timestamp":1600700948022,"user_tz":240,"elapsed":280,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"ca75736f-ce22-482a-829d-495a4902a224"},"source":["sent2+sent3"],"execution_count":13,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['The',\n"," 'family',\n"," 'of',\n"," 'Dashwood',\n"," 'had',\n"," 'long',\n"," 'been',\n"," 'settled',\n"," 'in',\n"," 'Sussex',\n"," '.',\n"," 'In',\n"," 'the',\n"," 'beginning',\n"," 'God',\n"," 'created',\n"," 'the',\n"," 'heaven',\n"," 'and',\n"," 'the',\n"," 'earth',\n"," '.']"]},"metadata":{"tags":[]},"execution_count":13}]},{"cell_type":"code","metadata":{"id":"3ZB_FSxhOQt6","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700952091,"user_tz":240,"elapsed":268,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"a52521f7-3dea-40d3-d67a-dfedee81cb75"},"source":["sent1.append(\"Some\")\n","sent1"],"execution_count":14,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['Call', 'me', 'Ishmael', '.', 'Some']"]},"metadata":{"tags":[]},"execution_count":14}]},{"cell_type":"code","metadata":{"id":"i_Bj67QHOQx3","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700953639,"user_tz":240,"elapsed":336,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"9e57146e-0f38-4984-d767-d1317bb5f265"},"source":["len(text1)"],"execution_count":15,"outputs":[{"output_type":"execute_result","data":{"text/plain":["260819"]},"metadata":{"tags":[]},"execution_count":15}]},{"cell_type":"code","metadata":{"id":"-e_FK-bjPYkW","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700956523,"user_tz":240,"elapsed":333,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"addb0f27-7eab-41d0-def2-60fca909b4e4"},"source":["text1[100:110]"],"execution_count":16,"outputs":[{"output_type":"execute_result","data":{"text/plain":["['and', 'to', 'teach', 'them', 'by', 'what', 'name', 'a', 'whale', '-']"]},"metadata":{"tags":[]},"execution_count":16}]},{"cell_type":"code","metadata":{"id":"lOgSt9w9Ps3-","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":225},"executionInfo":{"status":"ok","timestamp":1600700959688,"user_tz":240,"elapsed":367,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"cd579781-15d4-4c74-9277-c338b2145c90"},"source":["text1.concordance(\"monstrous\")"],"execution_count":17,"outputs":[{"output_type":"stream","text":["Displaying 11 of 11 matches:\n","ong the former , one was of a most monstrous size . ... This came towards us , \n","ON OF THE PSALMS . \" Touching that monstrous bulk of the whale or ork we have r\n","ll over with a heathenish array of monstrous clubs and spears . Some were thick\n","d as you gazed , and wondered what monstrous cannibal and savage could ever hav\n","that has survived the flood ; most monstrous and most mountainous ! That Himmal\n","they might scout at Moby Dick as a monstrous fable , or still worse and more de\n","th of Radney .'\" CHAPTER 55 Of the Monstrous Pictures of Whales . I shall ere l\n","ing Scenes . In connexion with the monstrous pictures of whales , I am strongly\n","ere to enter upon those still more monstrous stories of them which are to be fo\n","ght have been rummaged out of this monstrous cabinet there is no telling . But \n","of Whale - Bones ; for Whales of a monstrous size are oftentimes cast up dead u\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"_PeOwJRqp5tz","colab_type":"text"},"source":["# Complex Data Structures: Sets"]},{"cell_type":"markdown","metadata":{"id":"REcbnaVdp8Ar","colab_type":"text"},"source":["A set is a data structure where all elements are unique. While they are similar \n","to lists, in sets there is **no order** among the items in a set, and sets do **not** support indexing. Sets are ideal for membership queries, i.e., checking if an items appears in the set or not."]},{"cell_type":"markdown","metadata":{"id":"5GfzItlBuP0a","colab_type":"text"},"source":["## Creating Sets"]},{"cell_type":"markdown","metadata":{"id":"1R3cNucEuRgw","colab_type":"text"},"source":["Sets are specified by curly braces, `{ }`, containing one or more comma separated values. To specify an empty list, you can use the alternative construct, `set()`."]},{"cell_type":"code","metadata":{"id":"bBVvkjpsuO7Z","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700975817,"user_tz":240,"elapsed":262,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"d3e4de82-b323-4900-f1b3-1b6e21f4d669"},"source":["# creating sets\n","one_set = {4, 2, 1, 3}\n","print(one_set)"],"execution_count":18,"outputs":[{"output_type":"stream","text":["{1, 2, 3, 4}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"hOjTaSeep7ew","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700977101,"user_tz":240,"elapsed":271,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"f93b7114-7098-4e38-d05f-8f3919c1df35"},"source":["two_set = {6, 5, 4, -2}\n","print(two_set)"],"execution_count":19,"outputs":[{"output_type":"stream","text":["{4, 5, 6, -2}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"P-ku1DVOuar4","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700978252,"user_tz":240,"elapsed":379,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"8334f0b5-2a7b-441a-b331-95a2ada6a7bb"},"source":["three_set = {5, 0, -3, 4, 4, 4, 4}\n","print(three_set)"],"execution_count":20,"outputs":[{"output_type":"stream","text":["{0, -3, 4, 5}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"46bOXpp1udj4","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700980331,"user_tz":240,"elapsed":507,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"88cb29a9-fd15-4dbe-e439-b115e7ee8a92"},"source":["# creating an empty set; notice that we do *not* use the \"empty set = {}\" command\n","# as someone would expect based on the way that we create an empty list\n","empty_set = set()\n","print(empty_set)"],"execution_count":21,"outputs":[{"output_type":"stream","text":["set()\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"dQHsGAyiukzx","colab_type":"text"},"source":["We can also create a set from a list:"]},{"cell_type":"code","metadata":{"id":"_ZjEvwNbumF4","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":51},"executionInfo":{"status":"ok","timestamp":1600700982614,"user_tz":240,"elapsed":231,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"82938939-115d-48bb-f9d7-5572ddfd0df5"},"source":["my_list = [1, 2, 3, 0, 5, 10, 11, 1, 5]\n","my_set = set(my_list)\n","print(my_list)\n","print(my_set)"],"execution_count":22,"outputs":[{"output_type":"stream","text":["[1, 2, 3, 0, 5, 10, 11, 1, 5]\n","{0, 1, 2, 3, 5, 10, 11}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"Y6QWytY0un9I","colab_type":"text"},"source":["**As expected, the set created from a list does not contain any duplicate elements.**"]},{"cell_type":"code","metadata":{"id":"zeWd_iSWurph","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":51},"executionInfo":{"status":"ok","timestamp":1600700984728,"user_tz":240,"elapsed":275,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"2af4ca0d-0f0a-48b9-805e-b5c946e58404"},"source":["print(\"Length of list:\", len(my_list))\n","print(\"Length of set:\", len(my_set))"],"execution_count":23,"outputs":[{"output_type":"stream","text":["Length of list: 9\n","Length of set: 7\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"w3JZ1aZuuvnn","colab_type":"text"},"source":["As seen above, we can use len() to get the number of elements in a list. Similarly, we can use the other functions min(), max(), sum(), sorted() etc., that we also used for lists."]},{"cell_type":"markdown","metadata":{"id":"PVefhSPOU23F","colab_type":"text"},"source":["#### Checking for membership in a set\n","\n","For sets, we can only check if an item appears within the set or not. We achieve this using the `in` operator:"]},{"cell_type":"code","metadata":{"id":"l1vd5mgZU9EV","colab_type":"code","colab":{},"executionInfo":{"status":"ok","timestamp":1600700989328,"user_tz":240,"elapsed":400,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}}},"source":["my_set = {1, 2, 3, 4}"],"execution_count":24,"outputs":[]},{"cell_type":"code","metadata":{"id":"QRlDHRqxU_JO","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700991939,"user_tz":240,"elapsed":369,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"02f08ad6-443b-4f92-eb9b-0e7c987ea758"},"source":["val = 1\n","if val in my_set:\n","    print(\"The value\", val ,\"appears in the set\", my_set)\n","else: \n","    print(\"The value\", val ,\"does not appear in the set\", my_set)"],"execution_count":25,"outputs":[{"output_type":"stream","text":["The value 1 appears in the set {1, 2, 3, 4}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"15lUfaf3VAPN","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700994554,"user_tz":240,"elapsed":263,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"6a8ab1e4-4d0d-44a8-858e-ec614c3224ad"},"source":["val = 0\n","if val in my_set:\n","    print(\"The value\", val ,\"appears in the set\", my_set)\n","else: \n","    print(\"The value\", val ,\"does not appear in the set\", my_set)"],"execution_count":26,"outputs":[{"output_type":"stream","text":["The value 0 does not appear in the set {1, 2, 3, 4}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"LlqFnNvpVEHw","colab_type":"text"},"source":["We also have the `not in` operator, which behaves as expected:"]},{"cell_type":"code","metadata":{"id":"GROCT4DJVEqc","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600700998292,"user_tz":240,"elapsed":314,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"0e903b10-39c5-47d5-9d1c-336cd8620178"},"source":["if val not in my_set:\n","    print(\"The value\", val ,\"does not appear in the set\", my_set)\n","else: \n","    print(\"The value\", val ,\"appears in the set\", my_set)"],"execution_count":27,"outputs":[{"output_type":"stream","text":["The value 0 does not appear in the set {1, 2, 3, 4}\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"dIiJ-dQ6VKfE","colab_type":"code","colab":{}},"source":[""],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"V_WFxbffNwBG","colab_type":"text"},"source":["### NLTK Example\n"]},{"cell_type":"code","metadata":{"colab_type":"code","id":"Z-GP0PnAQjJ_","colab":{},"executionInfo":{"status":"ok","timestamp":1600701018370,"user_tz":240,"elapsed":269,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}}},"source":["def lexical_diversity(text):\n","  return len(set(text)) / len(text) "],"execution_count":29,"outputs":[]},{"cell_type":"code","metadata":{"colab_type":"code","id":"WSwyZF6XQjKF","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600701019453,"user_tz":240,"elapsed":312,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"097fd68b-1528-4092-c236-d202c3e6adbb"},"source":["lexical_diversity(sent1)"],"execution_count":30,"outputs":[{"output_type":"execute_result","data":{"text/plain":["1.0"]},"metadata":{"tags":[]},"execution_count":30}]},{"cell_type":"code","metadata":{"id":"ESiOvnu-QupO","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600701021556,"user_tz":240,"elapsed":242,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"2ddf25f7-4285-4107-fa5d-4bd9b66a85e4"},"source":["'Moby' in text1"],"execution_count":31,"outputs":[{"output_type":"execute_result","data":{"text/plain":["True"]},"metadata":{"tags":[]},"execution_count":31}]},{"cell_type":"code","metadata":{"id":"RcPHl0TNUPZG","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600701023189,"user_tz":240,"elapsed":334,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"4191cc95-afbb-4083-e94a-72ca8e8d47e6"},"source":["'Moby' in text2"],"execution_count":32,"outputs":[{"output_type":"execute_result","data":{"text/plain":["False"]},"metadata":{"tags":[]},"execution_count":32}]},{"cell_type":"markdown","metadata":{"id":"BelGHkOPVWUk","colab_type":"text"},"source":["### Set operators: Union, intersection, difference, subset. Plus, Jaccard Similarity"]},{"cell_type":"markdown","metadata":{"id":"VhPXBd-9VbR_","colab_type":"text"},"source":["Sets also support operations that allow us to quickly compute the difference, intersection, and union of two sets. For example:\n","\n","+ `set_a - set_b`: elements in a but not in b. Equivalent to `set_a.difference(set_b)`\n","+ `set_a | set_b`: elements in a or b. Equivalent to `set_a.union(set_b)`\n","+ `set_a & set_b`: elements in both a and b. Equivalent to `set_a.intersection(set_b)`\n","+ `set_a ^ set_b`: elements in a or b but not both. Equivalent to `set_a.symmetric_difference(set_b)` \n","+ `set_a <= set_b`:\ttests whether every element in set_a is in set_b. Equivalent to `set_a.issubset(set_b)`"]},{"cell_type":"markdown","metadata":{"id":"q5Ahy-JYVg7_","colab_type":"text"},"source":["**Exercise**\n","\n","Try the above yourself using the `set_A` and `set_B` variables, and compute the difference, union, intersection, and symmetric difference, between the two sets."]},{"cell_type":"code","metadata":{"id":"we1PHF8XVl0n","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":121},"executionInfo":{"status":"ok","timestamp":1600701041835,"user_tz":240,"elapsed":259,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"2b1e1358-cd9c-4930-b7ef-21657fe17d43"},"source":["# Your code here\n","set_A = {1, 2, 3, 4, 5}\n","set_B = {4, 5, 6, 7}\n","\n","print(\"Set A\", set_A)\n","print(\"Set B\", set_B)\n","print(\"Difference A-B\", {1, 2, 3} )\n","print(\"Union\", {1,2, 3, 4, 5, 6, 7})\n","print(\"Intersection\", {4, 5})\n","print(\"Symmetric Difference\", {1, 2, 3, 6, 7})\n"],"execution_count":34,"outputs":[{"output_type":"stream","text":["Set A {1, 2, 3, 4, 5}\n","Set B {4, 5, 6, 7}\n","Difference A-B {1, 2, 3}\n","Union {1, 2, 3, 4, 5, 6, 7}\n","Intersection {4, 5}\n","Symmetric Difference {1, 2, 3, 6, 7}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"edtD5Vs-VppF","colab_type":"text"},"source":["#### Solution"]},{"cell_type":"code","metadata":{"id":"IeXLgHmdVrNe","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":121},"executionInfo":{"status":"ok","timestamp":1600701044322,"user_tz":240,"elapsed":233,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"2852c023-71a5-4a32-80c9-81d123979fb1"},"source":["print(\"Set A\", set_A)\n","print(\"Set B\", set_B)\n","print(\"Difference A-B\", set_A - set_B )\n","print(\"Union\", set_A | set_B )\n","print(\"Intersection\", set_A & set_B )\n","print(\"Symmetric Difference\", set_A ^ set_B )"],"execution_count":35,"outputs":[{"output_type":"stream","text":["Set A {1, 2, 3, 4, 5}\n","Set B {4, 5, 6, 7}\n","Difference A-B {1, 2, 3}\n","Union {1, 2, 3, 4, 5, 6, 7}\n","Intersection {4, 5}\n","Symmetric Difference {1, 2, 3, 6, 7}\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"SoumVadyWjuj","colab_type":"text"},"source":["#### Exercise cont.\n","\n","Now, lets try to use the [Jaccard index similarity](https://en.wikipedia.org/wiki/Jaccard_index) to compute the similarity of the two sets. The Jaccard coefficient is defined as the ratio of the size of the intersection of the two sets, divided by the size of the union of the two sets."]},{"cell_type":"code","metadata":{"id":"Y4By2C9YWrnV","colab_type":"code","colab":{}},"source":["# Your code here"],"execution_count":null,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"wTo-g_m1Wtcb","colab_type":"text"},"source":["#### Solution"]},{"cell_type":"code","metadata":{"id":"4Fl-6s08WyOe","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600701051590,"user_tz":240,"elapsed":272,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"0c0493ad-c050-438f-ff5c-b69a4e8cfcc0"},"source":["intersection = set_A & set_B\n","print(\"The intersection has\", len(intersection), \"items\")"],"execution_count":36,"outputs":[{"output_type":"stream","text":["The intersection has 2 items\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"Slg-Z62mWzdt","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600701054076,"user_tz":240,"elapsed":242,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"4bc07b9a-5456-4140-c208-0ff319d673ff"},"source":["union = set_A | set_B\n","print(\"The union has\", len(union), \"items\")"],"execution_count":37,"outputs":[{"output_type":"stream","text":["The union has 7 items\n"],"name":"stdout"}]},{"cell_type":"code","metadata":{"id":"nb98aE_2W0b_","colab_type":"code","colab":{"base_uri":"https://localhost:8080/","height":34},"executionInfo":{"status":"ok","timestamp":1600701055741,"user_tz":240,"elapsed":290,"user":{"displayName":"Elena Filatova","photoUrl":"","userId":"16590435872187398682"}},"outputId":"06601484-0c8a-419f-8aa3-a80b90f89605"},"source":["jaccard = len(intersection) / len(union)\n","print(\"The similarity of set_A with set_B is\", jaccard)"],"execution_count":38,"outputs":[{"output_type":"stream","text":["The similarity of set_A with set_B is 0.2857142857142857\n"],"name":"stdout"}]},{"cell_type":"markdown","metadata":{"id":"PMKfpZ8rN_Hq","colab_type":"text"},"source":["## Next time we will talk about strings"]}]}