Natural Language Processing

Quite a few people have been asking questions about homework 2, and I’m not sure some of them realized that I had a hints page, and even a walk-through page. So, for these reasons and others, I’m going to further extend homework #2’s due date by 24 hours. It’s now due Saturday, Sept 27th at midnight.

Also, since I don’t want to have the quiz due the same day, I’m extending quiz #1 as well, which is now due Sunday, Sept 28th at midnight.

Finally, everyone who did turn in homework #2 on time will get a +5XP bonus.

(You don’t think I’ll pay for all this in the afterlife, do you?)

I would like your corpus, or some subset thereof, in order to test your homework #2 and future homeworks this semester. This is potentially problematic because of the sizes involved. So please follow these instructions:

1. Find out how big your corpus is. (On Linux, you can type “ls -lh nameOfYourCorpusFile” and look near the middle of the line for the size of the file. It should end in a “K” (for kilobytes), an “M” (for megabytes), or a “G” (for gigabytes). For example, the Bob Dylan lyrics corpus is 930 KB:

   $ ls -lh dylan.txt
   -rw-r--r-- 1 stephen stephen 930K Sep 26 10:38 dylan.txt
   

   On Mac or Windows there’s probably some way to right-click/properties and see the file’s size. Google if you need to.)

2. If your corpus is less than 10 MB, please email it to me as an attachment with subject line “DATA 470 corpus turnin“.
3. If your corpus is between 10 MB and 100 MB, please send me an email with subject line “DATA 470 github repo request“. In the body of the email, include your github username. (If you don’t have a github account, create one. It is free.) I will then add you to a special repo and give you further instructions for uploading your corpus to it.
4. If your corpus is over 100 MB, please create a copy of it with just the first 100 MB. You can do this in Linux with the command:

   $ head -c 100M nameOfYourCorpusFile > stephensShorterCorpus
   

   This will create a new file called stephensShorterCorpus with only the first 100 MB of your corpus. (If you’re on Windows or Mac, google how to do a similar operation.) Then, follow the instructions in step 3.

You’ll get +5XP if you do this before Sunday, Sept 28 at midnight, or 0XP if you do it after that.

As you can tell from the readings page, we’re “behind” where I thought we’d be right now in terms of content. This is not a problem (our course is not the upstream prereq for any other, so there’s no fixed content we have to get through) but I thought I’d let you know that I’m not going to add any additional readings to the readings page until we get caught up.

Cheers!

Inspired by class today…

• For +1XP: find a website that has both the words jedi and hangover (or hungover) not more than 10 words apart. Caveat: the page you send me must not be the same as the page anybody else has already sent me (otherwise I’ll give you another change).
• For +2XP: find me a better example for this. To be clear: what I’m seeking are two words which are less likely to appear together (in the same 10-word window) than their individual probabilities multiplied together. (This would correspond to the “woman who lives with her parents” example. Female-ness, and living with one’s parents, tend not to go together, so P(female|liveWithParents) will be less than P(female) · P(liveWithParents).)

  Note that it’s no use just coming up with two words that are each independently rare, as Garrett tried to do today. Yes, I’m sure that the probability of finding vambrace and phytosphere together is very rare…but is it more rare than expected based on how rare they are individually? I’m far from convinced of that fact.

To claim either or both of these XP, send me an email with subject line “DATA 470 — random XP” by midnight on Sunday.

Because I’m cool.

The hopefully at least somewhat helpful demo_torch.py file we worked through in class on Tuesday has been posted in the team repo. As always, the right way to work with this is to pull from the repo after originally having cloned it directly from github.

I’ve had a lot of smart and ambitious students doing cool things with their N-gram models, and getting stuck pursuing a couple of well-intentioned rabbit trails, so I’m going to give everyone til the end of the week to submit homework #2.

Quiz #1 has been posted to Canvas, and is due Friday the 26th at midnight. This quiz is timed generously at 90 minutes, is open-book, open-notes, open-four-function-calculator, closed-Python, and must be completed entirely on your own, with no help from others.

Students found Emily Dickinson’s to be…

Homework #1 has finally been graded and is on the scoreboard. Great job on these! And congrats to those who have reached new levels:

	Samuel Langhorn Clemens (a.k.a. Mark Twain) was one of the greatest writers in American history, and a potent 19th-century public figure and celebrity. Ernest Hemingway quipped, “All modern American literature comes from Huckleberry Finn.”
	Charles Lutwidge Dodgson (a.k.a. Lewis Carroll) was a brilliant and eccentric polymath who produced, among many other curious things, the Alice in Wonderland stories.