DATA 419 homework #0

Part 2: Lab project setup (+20XP)

This guide will help you create a reproducible local workspace for your semester labs. You may use Linux, macOS, or Windows, and you may edit your code with vim, VS Code, Spyder, or any other text editor. The underlying project structure will be the same.

Your workspace will include a git repository (“repo” in the lingo) to store the whole version history of your evolving project. (If you’re not already in the habit of using git in your personal work, you should begin now.)

The goal is to separate four things:

1. Your source code, which belongs in git.
2. Your generated data, which can be recreated and normally does not belong in git.
3. Your Python environment, which lives outside the project and contains the packages your code needs.
4. Your interactive work, where you load DataFrames, inspect them, make plots, and develop analyses one step at a time.

The examples below use stormlight as a sample topic. Replace stormlight with a short name for your own subject.

1. Install or update Python

You need Python 3.11 or later for these labs. Follow the instructions for your operating system.

python3 --version

which python3

sudo apt update
sudo apt install python3 python3-venv python3-pip

sudo dnf install python3 python3-pip

python3 --version
pip3 --version

python3 --version

command -v python3

python3 --version
pip3 --version

python --version

Get-Command python

python --version
pip --version

2. Choose names and locations

Choose a short topic name containing lowercase letters and, if necessary, underscores. Examples:

• low_income_housing
• baseball
• air_quality

(As mentioned earlier, these instructions will use “stormlight” as the topic name.)

Avoid spaces and hyphens in the Python import name.

You will use three related names:

The installed project name may contain a hyphen. The Python import name must not, so Python code will use:

import stormlight

Create a main folder for all of your lab work this semester. We will call this your lab root folder. In command-line documentation, folders are often called directories; the two terms are exact synonyms.

The examples name the lab root folder:

data419-labs

Also create a separate venvs folder inside your computer’s home folder. Your virtual environment should live inside that folder, not inside your lab root folder. For this example, its full location would look like this:

 /home/your-username/venvs/stormlight-labs

 /Users/your-username/venvs/stormlight-labs

 C:\Users\your-username\venvs\stormlight-labs

On macOS and Linux, the “~” symbol is command-line shorthand for “your home folder”. Thus, ~/venvs/stormlight-labs is an alias for the location shown above.

Keeping the virtual environment outside the project prevents thousands of machine-specific environment files from becoming mixed with your code or accidentally committed to git.

3. Create the lab root folder and virtual environment

Follow the same sequence no matter your platform, but use the commands for your specific operating system.

☞ Linux or macOS

mkdir -p ~/data419-labs
mkdir -p ~/venvs

python3 -m venv ~/venvs/stormlight-labs
source ~/venvs/stormlight-labs/bin/activate

New-Item -ItemType Directory -Force "$HOME\data419-labs"
New-Item -ItemType Directory -Force "$HOME\venvs"

python -m venv "$HOME\venvs\stormlight-labs"
& "$HOME\venvs\stormlight-labs\Scripts\Activate.ps1"

After activation, your prompt will usually show the environment name:

(stormlight-labs)

Confirm that the activated environment owns the Python command.

☞ Linux or macOS

which python

where.exe python

The reported path should be inside your stormlight-labs virtual environment.

4. Install the basic analysis tools

With the virtual environment active (see just above for how to verify that), run:

pip install --upgrade pip
pip install pandas pyarrow ipython

pandas provides DataFrames, pyarrow lets pandas read and write Parquet files, and IPython provides a convenient interactive Python session.

You’ll see Stephen use Polars a lot in class because he likes it better than Pandas. If you want to be like Stephen, you can install it too:

pip install polars

If you’re using VS Code’s notebook editor or Interactive Window, you may also install Jupyter:

pip install jupyter

Install any additional package needed by your particular data source. Examples might include an API client, requests, openpyxl, or a database driver. Whatever documentation accompanied the data source is where you will discover this. (Don’t worry: if it turns out you need a package later on, you can just pip install it at that time.)

5. Create the project structure

Inside your lab root folder, create the following structure:

data419-labs/
|-- .gitignore
|-- README.md
|-- stormlight/
|   |-- pyproject.toml
|   |-- README.md
|   |-- data/
|   |   `-- .gitkeep
|   `-- stormlight/
|       |-- __init__.py
|       |-- load.py
|       `-- pull_all.py
|-- lab0/
|-- lab1/
|-- lab2/
`-- .../

Create additional lab folders later as needed.

Note very carefully: there are two folders named stormlight, and they have different jobs:

• The outer stormlight/ is the small installable project. It contains pyproject.toml, its generated data, and its Python package.
• The inner stormlight/ is the importable Python package. It is what makes import stormlight work.

The lab0/, lab1/, and later folders will contain the scripts, notebooks, figures, notes, and other work specific to each lab.

You may create these folders with the command line (mkdir, cd, ls, and friends), your editor, or a graphical file manager. The final shape matters more than the tool used to create it.

6. Create .gitignore

In the lab root folder, create a .gitignore file containing:

# Python-generated files
__pycache__/
*.py[cod]

# Packaging output
*.egg-info/
*.dist-info/
build/
dist/

# Notebook and editor output
.ipynb_checkpoints/
.DS_Store

# Generated data
stormlight/data/*

(Replace stormlight with your topic name.) You can create this file using the same editor that you edit Python code; simply save it under the name .gitignore rather than something.py. Heads-up: be sure to recognize that this filename must begin with a period! (“.gitignore”, not “gitignore”)

The purpose of this file, by the way, is to keep the generated data files out of git, and your git status uncluttered.

Also, depending on how you get your data set, you may be using some kind of access token, API key, or password. Do not commit any such secret credentials in git! Instead, read them from environment variables or another local configuration mechanism. (Email if you’re unsure whether or how to do this.)

7. Create pyproject.toml

Create stormlight/pyproject.toml:

[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"

[project]
name = "stormlight-labs"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = [
    "pandas",
    "pyarrow",
]

[tool.setuptools]
packages = ["stormlight"]

Also add to the dependencies list whatever other packages were required to acquire your data, and Polars if you want to be cool like Stephen. For example:

dependencies = [
    "pandas",
    "polars",
    "pyarrow",
    "requests",
]

The project name used by packaging tools is stormlight-labs, while the package listed under [tool.setuptools] is the import name stormlight.

8. Create the package interface

Create stormlight/stormlight/__init__.py. The hardest part about this step is actually naming the file correctly. Notice that the name has four underscores in it. (By the way, the cool kids in the biz pronounce this as “dunder init” instead of “underscore underscore init underscore underscore dot py,” but you can call it either one. The point is that the name of the file must match that exactly.)

from stormlight.load import load

__all__ = ["load"]

This makes the intended public interface simple:

import stormlight

tables = stormlight.load()

The other functions in load.py and pull_all.py should normally be internal helpers whose names begin with an underscore.

9. Write pull_all.py

pull_all.py should be the recipe for (re-)creating your analysis-ready data. It should do three basic things:

1. fetch the original data from wherever on the Internet you found it;
2. clean and prepare it into DataFrames;
3. save each DataFrame as a Parquet file, in a folder/directory called “data”.

Step 1 may need to download files, call an API, or screen scrape a website. Step 2 may need to combine source files, rename columns, parse dates, handle missing values, and/or perform other cleaning. Avoid manually editing the generated Parquet files. If something about the data needs to change, change pull_all.py and run it again. The purpose of this file is to give you a safe way to repeatedly re-create your data from its source.

Here’s my own pull_all.py from the WNBA data set, if you’d like to look at it. ChatGPT and I wrote most of it together, under my careful direction. It’s pretty huge and complicated, and the vast majority of it is very WNBA-specific, so I’m certainly not advising a strategy of “copy this and try to evolve it to work with your own dataset.” I offer it to you more just out of transparency.

10. Write load.py

load.py has one simple job: read your analysis-ready Parquet files and return them as Pandas (or Polars) DataFrames.

For a project with three tables, it might be as simple as this:

from pathlib import Path

import pandas as pd

def load():
    table_1 = pd.read_parquet("data/table_1.parquet")
    table_2 = pd.read_parquet("data/table_2.parquet")
    table_3 = pd.read_parquet("data/table_3.parquet")

    return {
        "table_1": table_1,
        "table_2": table_2,
        "table_3": table_3,
    }

If you have several tables, add one read_parquet() call for each one and put each DataFrame in the returned dictionary. The dictionary keys will be the short names you’ll actually type a lot in Python. (Hint: make them pretty short and easy to type, without being totally opaque.) The dictionary keys are the short names you will use in Python. As an example, tables["knightsradiant"] would refer to the DataFrame with information about the Knights Radiant.

If load() says that a Parquet file does not exist, it would mean you probably either have typo, or haven’t run your own pull_all.py first. Keeping these two jobs separate makes the workflow easy to understand: pull_all.py creates the data files; load.py reads them.

Again, out of transparency and general interest, here’s my load.py. Examine at your peril.

11. Install your own package in editable mode

Activate the virtual environment, then run this command from the lab root folder:

☞ Linux or macOS

cd ~/data419-labs
pip install -e ./stormlight

Set-Location "$HOME\data419-labs"
pip install -e .\stormlight

What this does is “install your own package,” and in editable mode. This means that Python records where your package’s source code lives, so you can do:

import stormlight

from anywhere in your lab project directories. Because the installation is editable, changes you make to the Python files take effect immediately; you do not need to reinstall your stormlight package every time you edit your code.

12. Use one common interactive workflow

Your editor and your interactive Python environment are separate choices.

You may edit files with vim, VS Code, Spyder, or another editor. For the least platform-specific workflow, use IPython as the common interactive environment.

Activate the virtual environment and start IPython:

ipython

Then acquire and load the data:

import stormlight

tables = stormlight.load()

Inspect an individual table:

knightsradiant = tables["knightsradiant"]
knightsradiant.head()

Tip: to load every table as a named Python variable, do this:

globals().update(stormlight.load())

After that, each table is available as a variable:

knightsradiant.head()

This globals().update(...) pattern works in IPython, Jupyter notebooks, VS Code notebooks, and Spyder’s IPython console.

13. Editor-specific notes

The project layout and virtual environment remain the same regardless of editor.

If you use an IDE, select the virtual environment created above as your project’s Python interpreter. VS Code and PyCharm can then use that environment when running code, and they can normally activate it automatically in their integrated terminals. You may still run all of the pip, python, and other commands in this guide from the IDE’s terminal exactly as shown.

🖥️ vim (or another terminal editor)

This is what you’ll always see Stephen doing in class. He edits files normally, then activates his virtual environment and runs ipython in a terminal. He thinks this is beautiful.

🖥️ VS Code

If you want to use VS Code, install the Python extension. Use Python: Select Interpreter and select the Python executable inside your stormlight-labs virtual environment. VS Code will then use that environment when running and debugging Python code. It will also normally activate the selected environment automatically when it opens an integrated terminal.

You may then:

• use the VS Code terminal and run ipython or any of the command-line instructions in this guide;
• open a .ipynb notebook using the Jupyter extension; or
• use # %% cells and the Python Interactive Window.

Check the interpreter from Python:

import sys

print(sys.executable)

The printed path should be inside your virtual environment.

🖥️PyCharm

If you want to use PyCharm, configure stormlight-labs as the project’s existing Python interpreter. Select the Python executable at one of these locations:

☞ Linux or macOS

.../venvs/stormlight-labs/bin/python

...\venvs\stormlight-labs\Scripts\python.exe

PyCharm will then use that environment when running and debugging Python code. Its integrated terminal can normally activate the project environment automatically. You may run the pip, python, and other command-line instructions in this guide from that terminal exactly as shown. PyCharm also provides graphical tools for installing packages into the selected interpreter, but using the terminal keeps the procedure consistent with the rest of this guide.

Check the interpreter from Python:

import sys

print(sys.executable)

The printed path should be inside your virtual environment.

🖥️Spyder

If you want to use Spyder, note that it already provides an IPython console. Make sure that console is using the Python interpreter from your project virtual environment. Check with:

import sys

print(sys.executable)

If Spyder requires an external environment to provide its kernel, install the compatible spyder-kernels package in that environment and select the environment in Spyder.

❌ Google Colab

Hosted Colab runs on a remote, temporary machine rather than automatically using the virtual environment and files on your computer.

For that reason, hosted Colab is not the recommended primary environment for this project structure. Keep the authoritative project, package, and reproducible data-acquisition code on your own computer. Consult Stephen before using Colab as your main environment. Colab can connect to a local runtime, but that is an additional setup path rather than the class default.

14. Create convenient shortcuts

These shortcuts are optional, but they reduce typing and help prevent work in the wrong folder or environment.

alias cdlabs='cd "$HOME/data419-labs"'
alias actlabs='source "$HOME/venvs/stormlight-labs/bin/activate"'

source ~/.bashrc

cdlabs
actlabs

alias cdlabs='cd "$HOME/data419-labs"'
alias actlabs='source "$HOME/venvs/stormlight-labs/bin/activate"'

source ~/.zshrc

cdlabs
actlabs

function cdlabs {
    Set-Location "$HOME\data419-labs"
}

function actlabs {
    . "$HOME\venvs\stormlight-labs\Scripts\Activate.ps1"
}

$PROFILE

New-Item -ItemType File -Force $PROFILE

cdlabs
actlabs

15. Use git and document the project

The git commands in this section are the same on Linux, macOS, and Windows. Run them in a terminal after moving into your lab root folder. You can use the cdlabs shortcut from the previous section if you created it.

For a more detailed reference, consult https://stephendavies.org/blue.pdf, especially page 17 and pages 215-228.

Initialize the repository once

Run these commands once, after creating the initial project structure:

git init
git status
git add .
git commit -m "Create semester lab project structure"

Here is what those commands do:

• git init creates a new, empty git repository in the current folder. It does not upload anything to the internet.
• git status reports which files are new, modified, staged, or untracked. It is safe to run at any time, and you should run it often.
• git add . stages the current changes so they will be included in the next commit. Files excluded by .gitignore will not be staged.
• git commit -m "..." records a snapshot of the staged files. The text after -m is a brief description of what changed.

If git says that your identity is unknown, configure your name and email address with these two commands, before trying the commit again:

git config --global user.name "Your Name"
git config --global user.email "your-email@example.com"

Record changes as you work

Whenever you reach a sensible stopping point, use this basic cycle:

git status
git diff
git add .
git status
git commit -m "Briefly describe what changed"

git diff shows changes that have not yet been staged. The second git status lets you confirm exactly what will be committed before you create the commit.

Use a commit message that describes the work in that snapshot, such as:

git commit -m "Add data acquisition code"
git commit -m "Complete lab 2 analysis"
git commit -m "Fix date parsing"

Review the project history

To see the sequence of commits, run:

git log --oneline

This displays one compact line per commit. If the output opens in a scrolling viewer, press q to return to the command prompt.

Document the project

Your root README.md should briefly describe the semester lab collection.

Your topic-specific stormlight/README.md should record:

• the original data source;
• the source URL, API, or acquisition procedure;
• any account or credential requirements, without including secrets;
• what each generated table represents;
• how to install the package;
• how to run pull_all.py;
• how to call load();
• any known limitations or manual steps.

The code in pull_all.py and the instructions in the README together should be enough for another person to recreate the analysis-ready Parquet files.

16. Final installation and test

Complete this test from a fresh terminal.

1. Activate the environment and go to your lab folder

If you created the shortcuts from Section 14, run:

cdlabs
actlabs

Otherwise, type them out again:

cd "$HOME/data419-labs"
source "$HOME/venvs/stormlight-labs/bin/activate"

2. Confirm the interpreter

python -c "import sys; print(sys.executable)"

The path should be inside the virtual environment; i.e., “/venvs/stormlight-labs” should appear somewhere in it.

3. Confirm the editable installation

pip show stormlight-labs

4. Confirm the import from a lab folder

Make the folder lab0 if you haven’t already done so, and move into it:

cd lab0

Then check which copy of the package Python is importing:

python -c "import stormlight; print(stormlight.__file__)"

The reported path should end with something like:

data419-labs/stormlight/stormlight/__init__.py

That confirms that Python is importing the source code in your editable working tree.

5. Materialize the data

python -m stormlight.pull_all

Verify that the expected Parquet files now exist under:

data419-labs/stormlight/data/

6. Load the tables interactively

ipython

Then:

import stormlight

tables = stormlight.load()
print(tables.keys())

globals().update(tables)

Inspect your tables:

knightsradiant.head()

Your setup is complete when all of the following are true:

• the virtual environment is outside the lab root folder;
• the package imports from every lab folder;
• pull_all.py recreates the analysis-ready Parquet files;
• generated data is ignored by git;
• load() returns a dictionary of DataFrames;
• you can inspect those DataFrames interactively;
• your README explains how another person can reproduce the setup.

Turning it in

To turn in this assignment (homework #0 parts 1 and 2), send me an email with subject line “DATA 419: homework #0 turnin”. It should have the following contents:

1. In the body of the email, or in a separate attachment, the descriptions of the three data sets you considered (one of which you actually chose). See Part 1 of the assignment for the components of these descriptions.

2. A screenshot of an IPython session in which you print the head (first few rows) of each table.

3. A screenshot of the output of the command git log. This should include several commits and their messages, which you created on the way to completing this assignment.

4. A screenshot of the output of the command git status, which should show a clean workspace with no outstanding work committed.

5. The contents of your two README.md files (which don’t have to be super long, but do include all of the items I listed).