Summary Sample solutions Association rule mining

Sample solutions Association rule miningSample text input Let's carry out this analysis on a "dummy" text fragment, which graphic designers refer to as the lorem ipsum ( In [1]: latin_text = """ Sed ut perspiciatis, unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam eaque ipsa, quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt, explicabo. Nemo enim ipsam voluptatem, quia voluptas sit, aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos, qui ratione voluptatem sequi nesciunt, neque porro quisquam est, qui dolorem ipsum, quia dolor sit amet consectetur adipisci[ng] velit, sed quia non numquam [do] eius modi tempora inci[di]dunt, ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit, qui in ea voluptate velit esse, quam nihil molestiae consequatur, vel illum, qui dolorem eum fugiat, quo voluptas nulla pariatur? At vero eos et accusamus et iusto odio dignissimos ducimus, qui blanditiis praesentium voluptatum deleniti atque corrupti, quos dolores et quas molestias excepturi sint, obcaecati cupiditate non provident, similique sunt in culpa, qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio, cumque nihil impedit, quo minus id, quod maxime placeat facere possimus omnis voluptas a ⟹ b a b conf(a ⟹ b) b a Pr[b | a]    Downloaded by noureddine FATTOUH () lOMoARcPSD| Onlystudents quod maxime placeat, facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet, ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat. """ print("First 100 characters:n {} ...".format(latin_text[:100])) Exercise 0 (ungraded). Look up and read the translation of lorem ipsum! Data cleaning. Like most data in the real world, this dataset is noisy. It has both uppercase and lowercase letters, words have repeated letters, and there are all sorts of non-alphabetic characters. For our analysis, we should keep all the letters and spaces (so we can identify distinct words), but we should ignore case and ignore repetition within a word. For example, the eighth word of this text is "error." As an itemset, it consists of the three unique letters, . That is, treat the word as a set, meaning you only keep the unique letters. This itemset has three possible itempairs: , , and . Start by writing some code to help "clean up" the input. Exercise 1 (normalize_string_test: 2 points). Complete the following function, normalize_string(s). The input s is a string (str object). The function should return a new string with (a) all characters converted to lowercase and (b) all non-alphabetic, non-whitespace characters removed. Clarification. Scanning the sample text, latin_text, you may see things that look like special cases. For instance, inci[di]dunt and [do]. For these, simply remove the non-alphabetic characters and only separate the words if there is explicit whitespace. For instance, inci[di]dunt would become incididunt (as a single word) and [do] would become do as a standalone word because the original string has whitespace on either side. A period or comma without whitespace would, similarly, just be treated as a non-alphabetic character inside a word unless there is explicit whitespace. So e basium would become e pluribusunum basium even though your common-sense understanding might separate pluribus and unum. Hint. Regard as a whitespace character anything "whitespace-like." That is, consider not just regular spaces, but also tabs, newlines, and perhaps others. To detect whitespaces easily, look for a "high-level" function that can help you do so rather than checking for literal space characters. In [2]: def normalize_string(s): assert type (s) is str ### BEGIN SOLUTION essential_chars = [c for c in () if ha() or ce()] return ''.join(essential_chars) ### END SOLUTION # Demo: print(latin_text[:100], "...n=", normalize_string(latin_text[:100]), "...") In [3]: # `normalize_string_test`: Test cell norm_latin_text = normalize_string(latin_text) assert type(norm_latin_text) is str assert len(norm_latin_text) == 1694 assert all([ha() or ce() for c in norm_latin_text]) assert norm_latin_text == norm_latin_() print("n(Passed!)") {e, o,r} {e, o} {e,r} {o,r} First 100 characters: Sed ut perspiciatis, unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, ... Sed ut perspiciatis, unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, ... = sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium ... (Passed!) Downloaded by noureddine FATTOUH () lOMoARcPSD| Onlystudents Exercise 2 (get_normalized_words_test: 1 point). Implement the following function, get_normalized_words(s). It takes as input a string s (i.e., a str object). It should return a list of the words in s, after normalization per the definition of normalize_string(). (That is, the input s may not be normalized yet.) In [4]: def get_normalized_words (s): assert type (s) is str ### BEGIN SOLUTION return normalize_string (s).split () ### END SOLUTION # Demo: print ("First five words:n{}".format (get_normalized_words (latin_text)[:5])) In [5]: # `get_normalized_words_test`: Test cell norm_latin_words = get_normalized_words(norm_latin_text) assert len(norm_latin_words) == 250 for i, w in [(20, 'illo'), (73, 'eius'), (144, 'deleniti'), (248, 'asperiores')]: assert norm_latin_words[i] == w print ("n(Passed.)") Exercise 3 (make_itemsets_test: 2 points). Implement a function, make_itemsets(words). The input, words, is a list of strings. Your function should convert the characters of each string into an itemset and then return the list of all itemsets. These output itemsets should appear in the same order as their corresponding words in the input. In [6]: def make_itemsets(words): ### BEGIN SOLUTION return [set(w) for w in words] ### END SOLUTION In [7]: # `make_itemsets_test`: Test cell norm_latin_itemsets = make_itemsets(norm_latin_words) # Lists should have the same size assert len(norm_latin_itemsets) == len(norm_latin_words) # Test a random sample from random import sample for i in sample(range(len(norm_latin_words)), 5): print('[{}]'.format(i), norm_latin_words[i], "--", norm_latin_itemsets[i]) assert set(norm_latin_words[i]) == norm_latin_itemsets[i] print("n(Passed!)") Implementing the basic algorithm Recall the pseudocode for the algorithm that Rachel and Rich derived together: First five words: ['sed', 'ut', 'perspiciatis', 'unde', 'omnis'] (Passed.) [32] enim -- {'n', 'e', 'm', 'i'} [220] eveniet -- {'e', 'i', 'n', 'v', 't'} [23] et -- {'e', 't'} [224] repudiandae -- {