Homework #2 hints

Roadmap

If you're lost on how to even get started, follow this step-by-step road map.

Miscellaneous tips

• To open the file, use the syntax "f = open('corpus.txt', encoding='utf-8')." You can then use "f.readlines()" to read the entire file in as a list of lines (strings).
• An easy way to slap a list of strings together as one big string is to say "' '.join(listOfStrings)". (This puts a space between each of the strings of the list when it concatenates them all together.)
• An easy way to do the reverse of the previous operation (i.e.., turn a space-separated string into a list of words) is to call ".split(' ')" on it.
• "Normalizing" text by removing extraneous (non-alphabetic) symbols can be achieved in a variety of ways. Two I know of are the ".replace()" method on strings and the ".sub()" method of regular expressions (which uses the re package.)
• ".strip()" can be called on a string to remove the spaces from both sides.
• ".lower()" can be called on a string to return a lower-case version of it.
• There's a Counter class in the collections package which some people like.
• Remember the way Python handles references to objects in lists when you iterate through them. So this accomplishes nothing:

      for word in list_of_words:
          word = word.lower()
  

  since you're just changing the temporary reference your loop variable has. The values in the original list remain unchanged! This, on the other hand, would probably accomplish what you want:

      for i in range(len(list_of_words)):
          list_of_words[i] = list_of_words[i].lower()
  

• Randomly generating one of a set of outcomes according to certain probabilities can be accomplished using Python's random.choices() function (don't forget to import random.) You give it two arguments: a list of things to choose from, and a named list of weights that has the relative probability of each thing.
  
  So if I had a dictionary like this:

  unigrams = { 'one':1, 'two':2, 'thirteen':13 }
  

  I could generate a random unigram — with a ¹⁄₁₆th chance of being "one", a ²⁄₁₆th chance of being "two", and a ¹³⁄₁₆th chance of being "thirteen" — this way:

  a_random_unigram = random.choices(list(unigrams.keys()), weights=list(unigrams.values())