Calculating N-Grams Frequencies

The assignment is to write a function called `calculate_ngram_frequencies` that takes a string of text and an integer n as input. The function should register all sequences of words of length n in the text and return a dictionary where the keys are the n-grams and the values are the number of occurrences. The function will also call another function to split the text into sentences. Then, it will iterate over these fragments and detect n-grams within each of them. Next, the assignment asks to use this function with n=6 on two files (`clinton_speeches_all` and `trump_speeches_all`) and save the results in two variables called `clinton_6gram_dict` and `trump_6gram_dict`, respectively. Finally, it will sort these dictionaries by frequency and print out the 10 most frequent 6-grams for both Clinton and Trump.

Python Programming Background
Background
The American presidential election is right around the corner, and both presidential candidates have
been busy giving speeches. It is intuitively clear that Trump and Clinton have quite different styles
of speaking. In this exam, we will see if we can quantify such differences by analyzing the speeches
using the programming tools that we have learned through the course.
The field of research of analyzing natural language is called Natural Language Processing (NLP).
The techniques that we will use in this exam are quite basic, and we will omit some of the details
that one would normally use in NLP. However, despite this simplicity, we will see that we can get
some interesting results.
Our analyses will be based on the following two data files, containing a selection of speeches given
by the candidates since their nomination some months ago at the national conventions for each of
the parties. The files can be found here:
• http://people.binf.ku.dk/wb/data/clinton_speeches.txt
• http://people.binf.ku.dk/wb/data/trump_speeches.txt
When opening the files, you will see that each speech consists of a title line starting with "#",
followed by a number of speech lines:
# Greensboro, North Carolina on Oct. 14, 2016:
Thank you. Wow. Thank you, everybody. Nice place. Plenty of room.
In 25 days, we're going win the state of North Carolina, which I love, and we're going to win
the White House.
...
Formal requirements:
Format: You should hand in:
1. A PDF file called exam.pdf containing the output from the different exams
below
2. A Python file called exam.py, containing the function definitions.
3. A Python file called exam_test.py, containing test code for the individual
questions.
These should be handed in as separate files (not zipped).
Content: For the Python part, remember to comment your code, use meaningful variable names,
and include docstrings for each function. Also, please limit yourself to the curriculum covered in
the course, that is, refrain from using list and dict comprehension, map, zip, reduce, filter and
lambda for the Python part, and tools like awk for the Unix part. Also, please use only those
external modules that are explicitly mentioned in the exam.

Question 1: Reading in the data
The rest of the exam will be done in Python.
1. In the exam.py file, write a function called read_speeches that takes a speech filename as
argument, and returns a dictionary, where the keys are the title of the speeches in the file (as
strings), and the values are the corresponding speeches (as strings). You should remove the "#"
character from the titles. Replace \n characters in the speech lines with a single space. Also, all
speech lines starting with "[" should be omitted. Remember to close the files after reading from
them.In the exam_test.py file, call the read_speeches on both the clinton_speeches.txt and the
trump_speeches.txt files, and save the results in two variables called clinton_speeches_dict and
trump_speeches_dict, respectively. Print out the size (number of keys) in each of these two
dictionaries.
If you have problems completing this exam, and therefore do not have the requested dictionaries,
please follow the following instructions in order to be able to complete the remainder of the exams:
Download the following files:
 trump_speeches.json
 clinton_speeches.json
Insert the following lines of code in your program:
import json
json_file_trump = open("trump_speeches.json")
json_file_clinton = open("clinton_speeches.json")
trump_speeches_dict = json.load(json_file_trump)
clinton_speeches_dict = json.load(json_file_clinton)
json_file_trump.close()
json_file_clinton.close()
Note that these dictionaries are not exactly identical to the ones you get from solving the exam
yourself (so you cannot use them for verification purposes).
2. In the exam.py file, write a function called merge_speeches that takes a list of speeches excluding
titles (i.e. list of strings) as argument, and returns a single string containing all speeches. As a
separator between the speech strings, use a single space.
In the exam_test.py file, call the merge_speeches to merge all of Clinton's speeches and all of
Trump's speeches, and save the result in variables called clinton_speeches_all and the
trump_speeches_all, respectively (Hint: there is an easy way to get all of the values in a dictionary
as a list). Print the length of the two lists to screen.
If you have problems completing this exam, and therefore do not have the requested strings
containing all speeches, please follow the following instructions in order to be able to complete the
remainder of the exams:
Download the following files:

• trump_speeches_all.txt
• clinton_speeches_all.txt
Insert the following lines of code in your program:
trump_file = open("trump_speeches_all.txt")
clinton_file = open("clinton_speeches_all.txt")
trump_speeches_all = trump_file.read()
clinton_speeches_all = clinton_file.read()
trump_file.close()
clinton_file.close()
Note that these strings are not exactly identical to the ones you get from solving the exam yourself
(so you cannot use them for verification purposes).
Question 2: Counting
In the exam.py file, write a function called count_words that takes a text (string) as input, and
returns the number of words in this text. You can assume that words are defined as anything that is
separated by whitespace.In the exam_test.py file, test the count_words function on the
trump_speeches_all variable, and print the result.
If you have problems completing this exam, and therefore do not have the requested function,
please follow the following instructions in order to be able to complete the remainder of the exams:
Insert the following lines of code in your program:
def count_words(text):
"""Dummy function that always returns 50000"""
return 50000
You can use this function as a replacement for the correct function in the questions below.
2.In the exam.py file, write a function called count_sentences that takes a text (string) as input, and
returns the number of sentences in this text. You can assume that sentences are defined as being
separated by any of the following characters: ".", "!" or "?". Hint: an easy way to solve this is using
a regular expression, combined with re.split.
In the exam_test.py file, test the count_sentences function on the trump_speeches_all variable, and
print the result.
If you have problems completing this exam, and therefore do not have the requested function,
please follow the following instructions in order to be able to complete the remainder of the exams:
Insert the following lines of code in your program:
def count_sentences(text):
"""Dummy function that always returns 5000"""
return 5000
You can use this function as a replacement for the correct function in the questions below.

Question 3: The Flesch reading-ease score
The Flesch reading-ease score is a measure of how easy a text is to read. There is a direct translation
from this score to the primary-school grade in which children are generally able to read a text of a
given difficulty. For more details, see the Wikipedia page
(Links to an external site.)
In this Question, we will use this score to evaluate the complexity of the language used by the two
presidential candidates.
The definition of the Flesch reading-ease is: 206.835 - 1.015 *
number_of_words/number_of_sentences - 84.6 * number_of_syllables/number_of_words.
Calculating the number of syllables in a text is not trivial. There is a discussion of this topic here
(Links to an external site.)
The top posts suggests to count the number of matches of a regular expression that describes a
syllable. We will use the same idea here, but have changed the regular expression slightly: "[aiouy]
+e*|e(?!d\b|ly)[aiouye]?|[td]ed|le\b" - in order to make it work on an entire text, rather than just
single word.
1.In the exam.py file, write a function called count_syllables that takes a text (string) as argument
and returns the number of syllables in the text. The function should use the regular expression
defined above. You should convert the text to all lower-case letters in order for the regular
expression to work propertly. Hint: the findall method of a regular expression object returns all
matches as a list.
In the exam_test.py file, test your function on the string "I eat apples", and print the result.
If you have problems completing this exam, and therefore do not have the requested function,
please follow the following instructions in order to be able to complete the remainder of the exams:
Insert the following lines of code in your program:
def count_syllables(text):
"""Dummy function that always returns 100000"""
return 100000
You can use this function as a replacement for the correct function in the questions below.
2.In the exam.py file, write a function called calculate_flesch_score that takes a text (string) as
input, and returns the flesch score (a number), calculated using the definition above.
In the exam_test.py file, write a loop that iterates over the trump_speeches_dict dictionary,
calculates the flesch score for each speech using the calculate_flesch_score function, and saves the
result in a list called trump_scores. Do the same with the Clinton speeches (in
clinton_speeches_dict) and save that result to variable called clinton_scores. Print both lists to
screen.
If you have problems completing this exam, and therefore do not have the requested lists of scores,
please follow the following instructions in order to be able to complete the remainder of the exams:
Download the following files:

• trump_scores.json
• clinton_scores.json
•
Insert the following lines of code in your program:
import json
json_file_trump = open("trump_scores.json")
json_file_clinton = open("clinton_scores.json")
trump_scores = json.load(json_file_trump)
clinton_scores = json.load(json_file_clinton)
json_file_trump.close()
json_file_clinton.close()
Note that these lists are not exactly identical to the ones you get from solving the exam yourself (so
you cannot use them for verification purposes).
3.To plot the distribution of scores for each candidate, we can use a box or violin plot. Here is a
template for a function that does this:
def plot_scores(scores,
output_filename):
"""
Plots score distributions
Parameters
----------
scores : dict object
The values of this dictionary are lists of scores to be plotted. The keys of the dictionary
will be used as labels for the plot.
output_filename : str
The name of the file used where the plot will be saved.
"""
import matplotlib.pyplot as plt
x_values = range(1, len(scores)+1)
y_values = scores.values()
# Create a box/violin plot
# plt. # <- Fill in this line
# Add labels on x-axis
plt.setp(plt.gca(), xticks=x_values, xticklabels=scores.keys())

# Save figure
plt.savefig(output_filename)
Copy this function to your exam.py file, and fill in the missing line to create either a box or a violin
plot.In the exam_test.py file, call the plotting function on a dictionary which has the two
variables trump_scores and clinton_scores as values (you should find some appropriate keys).
Include the resulting plot in your exam.pdf file.
Question 4: n-grams
Finally, we would like to investigate how often the candidates repeat the same phrases. An n-gram
is a subsequence of a sentence consisting of n words. For instance the sentence "I love Python"
would contain three 1-grams ("I", "love", "Python"), two 2-grams ("I love") and ("love Python"),
and one 3-gram ("I love Python").
1.In the exam.py file, write a function called calculate_ngram_frequencies that takes two
arguments: a string of text (containing one or more sentences) and an integer n. The function should
register all sequences of words of length n in the text. For the particular value of n, the function
should save all such occurrences in a dictionary, where the keys are the n-grams (string consisting
of n words) and the values are the number of occurrences. This dictionary should be returned from
the function. The function should only search for n-grams within sentences. You can use the
following lines of code to turn the text (in the variable text) into a list of sentences:
import re
sentences = re.findall(r'[^\.\?!"]+', text)
You should iterate over these fragments, and detect n-grams within each of them.
In the exam_test.py file, call the calculate_ngram_frequencies function with n=6 on the
clinton_speeches_all and the trump_speeches_all files. Save the results in two variables called
clinton_6gram_dict and trump_6gram_dict, respectively.
If you have problems completing this exam, and therefore do not have the requested 6-gram
dictionaries, please follow the following instructions in order to be able to complete the remainder
of the exams:
Download the following files:
• trump_6grams.json
• clinton_6grams.json
Insert the following lines of code in your program:
import json
json_file_trump = open("trump_6grams.json")
json_file_clinton = open("clinton_6grams.json")

trump_6gram_dict = json.load(json_file_trump)
clinton_6gram_dict = json.load(json_file_clinton)
json_file_trump.close()
json_file_clinton.close()
Note that these lists are not exactly identical to the ones you get from solving the exam yourself (so
you cannot use them for verification purposes).
2.Finally, we want to see which of the 6-grams is most frequent. If you use the items() method on
the returned dictionaries, you get a list of (ngram, frequency) tuples. These lists can be sorted by
using the key option for the sort list method, combined with a function to return the second item
from a sequence (the frequency). It is illustrated on a small example here:
my_dict = {"name_1": 10, "name_2": 7, "name_3": 5} # simple dictionarymy_dict_items
= my_dict.items() # .items() returns a list of (key, value) tuples from the
dictionaryprint my_dict_items# produces output: [('name_2', 7), ('name_3', 5), ('name_1', 10)]
# define a function that returns the part of each item we want to sort bydef get_second(x): """returns
the second item of x""" return x[1]
# pass the above function to the .sort() method with the keyword argument "key"# so that we sort
the items by comparing the values at their second positionsmy_dict_items.sort(key=get_second)
# sort the list of tuples by what is contained in the second position of each tupleprint my_dict_items
# list is now sorted# produces output: [('name_3', 5), ('name_2', 7), ('name_1', 10)]
# Everything in one line using sorted instead of sortprint sorted(my_dict.items(), key=get_second) #
produces output: [('name_3', 5), ('name_2', 7), ('name_1', 10)]
Use a similar technique in your exam_test.py file, to sort the Clinton and Trump 6-grams, and print
out the 10 most frequent 6-grams for both Clinton and Trump.
Final remarks
Please ensure that all your code is well documented (e.g. doc-strings in functions and comments),
and that you use meaningful variable names. Finally, remember not to use features like list and dict
comprehension, map, zip, reduce, filter and lambda.