DATA 419 homework #1: Linear Regression (+40XP)

Due: Tues, 9/29, midnight

Linear regression: from one predictor to several

In homework #0, you chose a data set and built the machinery needed to obtain, clean, and load it. In this homework, you will begin actually modeling your data.

You will start with the simplest possible regression model: one numerical predictor and one numerical response variable. Then you will gradually make the model richer by adding another numerical predictor and finally a categorical predictor.

Along the way, you will practice:

dividing data into training and test sets;
normalizing or standardizing numerical variables;
fitting linear and polynomial regression models;
measuring predictive accuracy with $R^2$ and RMSE;
encoding categorical variables;
comparing increasingly complicated models; and
checking whether some of the important assumptions behind linear regression appear reasonable.

The important idea is that we will build the model one piece at a time and see what each additional piece buys us.

Preparation

Prep 1. Choose your weapons

Choose four useful variables from your data set:

One numerical dependent variable (“response variable”) $y$ that you would like to be able to predict.
Three independent variables (“predictors”):
1. A numerical i.v. $x_1$ that you suspect may be related to $y$.
2. A second numerical independent variable $x_2$ that you also suspect might help predict $y$.
3. A categorical independent variable $c$ that might also contain useful information about $y$.

For example, if your data concerned WNBA players, you might eventually have a model like:

\[\mathtt{points} = f(\mathtt{offensive\_rebounds}, \mathtt{minutes\_played}, \mathtt{position}) + \epsilon\]

where points is the dependent variable, offensive_rebounds and minutes_played are numerical independent variables, and position (guard, forward, center) is categorical.

Do not choose ID numbers, row numbers, names, URLs, or other bookkeeping variables merely because they happen to be present.

Also avoid categorical variables with hundreds or thousands of different values for this assignment. Choose something for which one-hot or ordinal encoding makes reasonable sense.

If your information lives in several related tables, combine the information you need into one modeling DataFrame. Each row should represent one observation whose dependent and independent variables belong together.

Prep 2. Clean, recode, transform as necessary

In identifying the above, you may well have discovered that you don’t necessarily have perfect columns that contain all the above information. This will thus necessitate some data wrangling before you can actually deploy the algorithms.

Some of you did a great deal of this already in your homework #0. Props! Those who didn’t will likely have a bit more work to do.

Every data set is different, of course, and so the techniques that are called for vary widely. But here are some of the common ones:

Changing/combining units. Perhaps you have two DataFrame columns for a person’s height in feet and inches, and what you need is total inches. You’ll want to write code that converts 5 feet and 10 inches to 70.
Combining categories. Maybe you have a ready-made categorical column, like the genre of a song or book, but there are too many different distinct genres to get much mileage on. Perhaps you want to lump sci-fi and fantasy together, or lump historical fiction, military fiction and romantic fiction all under fiction.
Dealing with missing data. First determine how much data is missing and whether the missingness is concentrated in one of the variables you plan to use. For this assignment, the simplest reasonable choice is often to use complete cases only: discard rows that are missing any of your four selected variables. If that would throw away a substantial fraction of your data, talk to me before proceeding. Filling missing numerical values with a mean or median is sometimes appropriate, but that value must be learned from the training data only and then applied to the test data. Interpolation is generally appropriate only when the observations have a meaningful order, such as a time series; do not interpolate merely because a value is missing.
Fixing malformed or sentinel values. A column that looks numeric may contain entries such as "N/A", "unknown", -999, or percentages stored as strings. Convert these to sensible numerical values or to missing values before modeling. Also check for values that are impossible on their face, such as a negative age or a percentage above 100.
Aggregating when the unit of observation is wrong. Your raw data may have several rows for what you really want to treat as one observation. For example, you might have one row per game but want one row per player-season. If so, aggregate first so that every row of the modeling data represents the thing you actually intend to predict.

Prep 3. Building `X` and `y`

After cleaning and preparation, get your data into the form needed for sklearn. Create an X object containing your three predictors (two numeric and one categorical), and a y NumPy vector containing the numerical values of your target. X may be a Pandas DataFrame at this stage; in fact, keeping the column names around will often make your life easier.

Near the top of your program, set NumPy’s random number seed to a fixed value:

np.random.seed(12345)

Then divide X and y into training and test sets using train_test_split() from sklearn.model_selection, with 80% of the rows in the training set and 20% in the test set. Make this split once, before you fit any scalers, encoders, or regression models, and use the same rows for every model in this assignment.

For example:

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=.2
)

Setting the seed near the top of your program makes the randomized operations in your program reproducible. Calling train_test_split() only once and then reusing those same training and test sets ensures that all of your models are being compared on the same observations.

You’re now ready to perform your actual linear regressions!

1. Begin with one numerical predictor

Start with your first numerical independent variable, $x_1$, and your dependent variable, $y$.

In your report, briefly identify:

what $x_1$ represents;
what $y$ represents; and
why you thought there might be a relationship between them.

One or two sentences is enough.

2. Scale the independent variable

Before fitting the model, transform $x_1$.

You may choose either of the following approaches.

Option A: standardization

Convert the variable to a z-score:

\[ z = \frac{x-\bar{x}}{s} \]

where $\bar{x}$ is the sample mean (the average of the known values) and $s$ is the sample standard deviation (the standard deviation of the known values) so that the transformed variable has approximately mean 0 and standard deviation 1.

You may compute this yourself, or use:

from sklearn.preprocessing import StandardScaler

Option B: normalization

Rescale the values to the interval 0 through 1.

You may compute this yourself, or use:

from sklearn.preprocessing import MinMaxScaler

Very important: fit preprocessing on the training data

The test data must not help determine the transformation.

For example, if you standardize the variable yourself:

Compute the mean and standard deviation of x_train.
Use those values to standardize x_train.
Use those same training mean and standard deviation values to transform x_test.

Do not compute a separate mean and standard deviation from the test data.

Likewise, if you use a sklearn scaler, do something conceptually like:

scaler.fit(x_train)
x_train_scaled = scaler.transform(x_train)
x_test_scaled = scaler.transform(x_test)

rather than fitting the scaler separately on the test data.

This rule will continue to apply to every preprocessing operation in this assignment.

(Btw, depending on how you extracted the column, sklearn may require x_train and x_test to be two-dimensional – for example, one-column DataFrames rather than one-dimensional Series.)

Do not scale the dependent variable at all.

3. Model 1: simple linear regression

Fit a straight-line regression model predicting $y$ from $x_1$. Fit the model using the training data only; the test data are reserved for evaluation.

For this assignment, I want you to explicitly include the intercept in the design matrix instead of asking sklearn to create it automatically.

Your design matrix should therefore contain:

where the first column consists entirely of 1’s.

Then create your model with:

LinearRegression(fit_intercept=False)

The coefficient multiplying the column of 1’s is therefore the intercept that you effectively computed “by hand.”

Report the fitted equation

Report the equation of the fitted line, including the intercept and slope.

For example:

\[y_{predicted} = 13.2 + 4.7 x_1\]

Remember that your $x_1$ in this equation is the scaled version of the original variable.

Measure its performance

Compute both of the following for the training data and for the test data:

$R^2$
root mean squared error (RMSE)

You may use:

from sklearn.metrics import mean_squared_error, r2_score

with:

rmse = np.sqrt(mean_squared_error(y_true, y_pred))

Your test-set scores are the main ones we care about when comparing models. Do not use the test results to decide which variables to add or how to preprocess them; make those choices first, then use the test set to evaluate the result.

Write one sentence interpreting the test $R^2$ in the context of your actual data.

Then write one sentence interpreting the test RMSE.

For example, suppose you were predicting salaries and obtained:

R² = .42
RMSE = $8,300

A reasonable interpretation would be something like:

The model accounts for about 42% of the variation in salary in the test data.

and:

The model’s prediction error has an RMSE of about $8,300.

If your test $R^2$ is negative, do not describe it as a negative percentage of “variance explained.” A negative test $R^2$ means that, on those test cases, your model predicted worse than the simple baseline of always predicting the training-set mean of $y$.

Do not say that $x_1$ causes changes in $y$. Regression by itself does not establish causation.

Fit a quadratic too

Now also fit:

\[ y=b_0+b_1x_1+b_2x_1^2 \]

You are still using LinearRegression, since the model is linear in the coefficients $b_0,b_1,b_2$.

Its design matrix should look like:

1    x1    x1²
1    x1    x1²
1    x1    x1²
...

Again use:

LinearRegression(fit_intercept=False)

Call this Model 1Q (“Q” for “Quadratic.”)

Compute its training and test $R^2$ and RMSE as well.

In one or two sentences, say whether adding the quadratic term appears to have helped.

For the remainder of the assignment, however, Model 1 means the original straight-line model, not the quadratic model.

Make a plot

Create one scatterplot containing:

the training observations;
the Model 1 best-fit straight line in green; and
the Model 1Q best-fit quadratic curve in red.

Both fitted lines must come from models trained on the training data only.

Label the axes and give the graph a useful title.

Your fitted line and curve should appear as smooth lines from left to right. If necessary, sort the $x$ values before plotting the predictions, or create a regularly spaced sequence of $x$ values on which to make predictions.

4. Model 2: add a second numerical predictor

Now bring in your second numerical independent variable, $x_2$.

Briefly explain why you think this variable might help predict $y$.

Normalize or standardize $x_2$, using the same choice you made for $x_1$.

Again, the transformation must be learned from the training data only and then applied unchanged to the test data.

Your new design matrix should contain:

1    x1    x2
1    x1    x2
1    x1    x2
...

Fit:

LinearRegression(fit_intercept=False)

and call this Model 2.

Compute its training and test:

$R^2$
RMSE

Compare Model 2 with Model 1

Compare the test-set scores of Models 1 and 2.

Suppose, for example, that:

                 R²       RMSE
Model 1         .31       9.7
Model 2         .44       8.2

Then Model 2 explains 13 additional percentage points of variation in the test data:

44% - 31% = 13 percentage points

Its RMSE has also decreased by:

9.7 - 8.2 = 1.5

meaning that its RMSE is 1.5 units smaller. Since RMSE is measured in the same units as $y$, this gives you a concrete sense of the improvement in predictive accuracy.

Write two or three sentences explaining how much the second predictor helped.

If it did not help, say that! Adding a predictor does not guarantee better performance on unseen data.

5. Model 3: add a categorical predictor

Now bring in the categorical variable that you selected earlier.

A regression model cannot directly use values such as:

"East"
"West"
"North"
"South"

or:

"freshman"
"sophomore"
"junior"
"senior"

We therefore need to convert the categories into numerical information.

Decide how the variable should be encoded

Choose either one-hot encoding or ordinal encoding, and explain why you made that choice.

Use one-hot encoding when the categories do not have a meaningful numerical order.

Examples include:

red, green, blue
Virginia, Maryland, North Carolina
comedy, drama, horror

You may use:

from sklearn.preprocessing import OneHotEncoder

Because our regression model already contains a column of 1’s for its intercept, drop one of the one-hot categories to serve as the reference category. For example:

OneHotEncoder(drop="first", handle_unknown="ignore", ...)

Otherwise the design matrix contains redundant columns, creating perfect multicollinearity. The handle_unknown="ignore" option also prevents an error if a category happens to appear in the test set but not in the training set.

Use ordinal encoding only when the categories really do have a meaningful order.

For example:

small < medium < large

or:

freshman < sophomore < junior < senior

You may use:

from sklearn.preprocessing import OrdinalEncoder

If you use ordinal encoding, explicitly specify or document what ordering you used. Do not simply let alphabetical order accidentally determine the numerical meanings of the categories. Also remember that linear regression will treat the codes as equally spaced: moving from category 0 to 1 has the same modeled effect as moving from 1 to 2. Therefore, use ordinal encoding only when that assumption is at least reasonably defensible. When in doubt, one-hot encoding is the safer choice.

Avoid data leakage again

Just as with the numerical scalers:

fit the encoder on the training data;
transform the training data;
use that same fitted encoder to transform the test data.

Do not fit a second encoder on the test data.

Fit Model 3

Add the encoded categorical variable to your design matrix and fit another:

LinearRegression(fit_intercept=False)

Call this Model 3.

Remember that one categorical variable may produce several columns when it is one-hot encoded. It is still one conceptual predictor variable even though the design matrix contains several columns representing it.

Compute Model 3’s training and test:

$R^2$
RMSE

Then compare its test-set scores with Model 2.

Explain in two or three sentences how much, if anything, the categorical information improved the model.

6. Summarize the models side-by-side

At the end of your report, create a table summarizing your results.

It should contain at least:

Model	Predictors	Train $R^2$	Test $R^2$	Train RMSE	Test RMSE
Model 1	$x_1$
Model 1Q	$x_1, x_1^2$
Model 2	$x_1, x_2$
Model 3	$x_1, x_2, c$

Then write a short concluding paragraph.

Discuss:

which additions to the model helped;
approximately how much they helped;
whether the training and test results tell noticeably different stories; and
whether the regression diagnostics revealed anything that makes you more cautious about interpreting the model.

You are not trying to find the Greatest Machine-Learning Model Ever Made in this homework.

You are learning how a regression model is assembled, evaluated, extended, and checked.

What to turn in

To turn in this assignment, send me an email with subject line “DATA 419: homework #1 turnin” and with two attachments:

1. Your Python program

Attach the .py file containing the code you used for the analysis.

It should:

load your data using the package you created in homework #0;
create the train/test split;
perform all preprocessing;
fit Models 1, 1Q, 2, and 3;
compute the requested metrics;
generate the required plots.

Do not send me screenshots of your code. Send me the actual source file.

2. Your short report

Attach a single PDF containing your results.

It should contain:

The names and meanings of $y$, $x_1$, $x_2$, and the categorical variable.
Your reason for choosing those variables.
Whether you normalized or standardized the numerical predictors.
The fitted Model 1 equation.
Your scatterplot with the green linear fit and red quadratic fit.
Your one-sentence interpretations of Model 1’s test $R^2$ and RMSE.
Your comparison of Model 1Q with Model 1.
Your comparison of Model 2 with Model 1.
Your categorical encoding choice and why you chose it.
Your comparison of Model 3 with Model 2.
Your final model-comparison table and concluding paragraph.

This does not need to be a long formal paper. Short paragraphs, tables, figures, and clearly labeled answers are preferable to pages of prose.

As always, your work should be committed to your git repository before you turn it in.

Questions?

Send email with subject line DATA 419 homework #1 help!! or come to office hours!

Model	Predictors	Train \(R^2\)	Test \(R^2\)	Train RMSE	Test RMSE
Model 1	\(x_1\)
Model 1Q	\(x_1, x_1^2\)
Model 2	\(x_1, x_2\)
Model 3	\(x_1, x_2, c\)

DATA 419 homework #1: Linear Regression (+40XP)

Linear regression: from one predictor to several

Preparation

Prep 1. Choose your weapons

Prep 2. Clean, recode, transform as necessary

Prep 3. Building X and y

1. Begin with one numerical predictor

2. Scale the independent variable

Option A: standardization

Option B: normalization

Very important: fit preprocessing on the training data

3. Model 1: simple linear regression

Report the fitted equation

Measure its performance

Fit a quadratic too

Make a plot

4. Model 2: add a second numerical predictor

Compare Model 2 with Model 1

5. Model 3: add a categorical predictor

Decide how the variable should be encoded

Avoid data leakage again

Fit Model 3

6. Summarize the models side-by-side

What to turn in

1. Your Python program

2. Your short report

Questions?

Prep 3. Building `X` and `y`