DATA 419 homework #0

Part 1: Find Your Data Set (+20XP)

Throughout this semester, you will be working with one data set that you choose yourself. You will use it repeatedly as we learn different techniques for exploring data and building machine-learning models.

This means that choosing your data is important. Do not rush this assignment. A data set that interests you and contains enough useful information can make the rest of the semester much more enjoyable. A poor choice can make every later assignment harder.

Start with things you care about

Before even looking for data, spend some time brainstorming topics that actually interest you.

These could come from books, technology, food, hobbies, sports, music, movies, games, politics, history, transportation, health, business, the environment, education, astronomy, animals, cities, travel, or almost anything else.

Actually write down several possibilities. As you’ll see later in this assignment, you will eventually need to show that you investigated three different possibilities.

Then start searching. Google and AI are very useful here. Pro tip: be specific in your inquiries. Instead of asking:

Where can I find a data set?

try asking something much more specific, such as:

What publicly available data sets exist about climate change?

or:

I am interested in birds and birdwatching. What publicly available data sets could I use for a semester-long machine-learning project?

or:

What public data sources contain information about Broadway shows, including their running time, theatres, genre, and cast?

The goal at this stage is exploration, not immediately settling on the first thing you find.

Data in the real world can be messy

One important warning: not everything in the world comes prepackaged as a convenient CSV file that you can simply download and import.

Sometimes the universe will cooperate and you will find exactly that. More commonly, though, the data you want might be:

• spread across several CSV or Excel files;
• divided into separate tables that need to be combined;
• stored as JSON;
• available through a public API;
• published as downloadable files that need some cleaning;
• divided by year, region, team, category, etc.;
• accompanied by a separate file explaining what the variables mean.

That is normal.

You don’t need to have all of those problems solved just yet. In the next half of this assignment, you will build the code that actually obtains, combines, cleans, and prepares your data. For now, though, you just need to understand what you are choosing.

Get your hands dirty

Do not choose a data set based only on its description on a web page.

If you are seriously considering a data source, actually obtain some of the data and look at it.

Download the files. Open them. Browse the rows and columns. If there are several files or tables, figure out roughly what each one contains and how they relate to one another.

You should be able to answer questions such as:

• What does one row represent?
• Roughly how many rows are there?
• What kinds of variables are available?
• Which variables are numbers?
• Which variables represent categories?
• Are there multiple files or tables?
• Is there an obvious way to connect those files or tables?
• Are there lots of missing values?
• Are the variable names and meanings understandable?
• Is there documentation explaining the data?
• Can you imagine asking several interesting questions about these data?

You do not need to understand every detail yet. But by the time you select a data set, it should no longer be a mysterious set of files you found on the Internet.

Minimum requirements

Your final data set must be substantial enough that we can use it for many different kinds of analysis throughout the semester.

Use the following as minimum requirements:

• At least 500 observations (rows) after the major pieces of the data have been assembled. 1,000 or more is preferred.
• At least 8 potentially useful variables that describe those observations.
• At least 3 useful numerical variables. (Numerical variables are numbers, like years, ratings, ages, goals scored, etc.)
• At least 2 useful categorical variables. (Categorical variables are not numbers. Examples include genre, color, gender, geographic region, political party, brand, etc.)
• Enough variation in the data that the variables are actually interesting. A column in which almost every row has the same value is not very useful.
• The data must contain enough information that you can imagine investigating several different questions, rather than having only one obvious thing that can be done with it.
• The data must be publicly obtainable in a way that you will be able to reproduce later.

Identifiers and bookkeeping columns do not necessarily count toward these requirements. For example, a row number, database ID, URL, filename, or person’s name may be useful for keeping track of records, but it is not automatically a useful variable for analysis.

Multiple tables or files

A data source consisting of two or more related tables or files is preferred. Learning how information from different sources fits together is an important part of working with real data.

However, multiple tables are not an absolute requirement. A single table can be acceptable if it is sufficiently large and rich.

If you are unsure whether your proposed data set is substantial enough, ask.

Think about what the data could eventually let you predict or explain

Look for a data set containing a mixture of characteristics. Be curious about it. Ideally, you should be able to imagine questions like:

• Why is this number high for some observations and low for others?
• Can other characteristics help us predict this numerical value?
• Why do some observations belong to one category and others to another?
• Can we predict which category an observation belongs to?
• Which observations seem similar to one another?
• Are there natural groups or patterns within the data?

If your data can support several questions like these, that is a good sign.

Consider several possibilities

You are required to investigate at least three candidate data sets before making your final choice.

For the first half of this assignment, you will turn in:

1. A description of your chosen data set, including:
   • What is it?
   • Where does it come from?
   • What does one row represent?
   • Approximately how many rows and useful variables does it contain?
   • What numerical and categorical information does it contain?
   • Why does the topic interest you?
   • Why do you think it will provide enough material for a semester of analysis?
2. A description of Alternative data set #1:
   • Write one paragraph describing the data set.
   • Explain why you considered it.
   • Explain why you ultimately decided against it.
3. A description of Alternative data set #2:
   • Write one paragraph describing the data set.
   • Explain why you considered it.
   • Explain why you ultimately decided against it.

The two rejected alternatives are an important part of the assignment. I do not want you to find the first plausible CSV on Google and immediately commit yourself to it. I want you to compare possibilities and make a deliberate choice.

One final consideration

You are going to live with this data set for a while.

Choose something that you are genuinely curious about. You do not need to know exactly what you will discover, and you do not need to know yet which machine-learning techniques will turn out to work well.

You just need a topic you care about and a sufficiently rich collection of data about it.

Once you think you have found your data, you are ready to begin Part 2 of this assignment. In that assignment, you will create pull_all.py and load.py, which will become the machinery for actually obtaining, combining, cleaning, and preparing your data for use throughout the semester.