I’ve decided to first run all the rules-based players by themselves, then do LLMs after that (time-permitting).

Here are the official rules-based-players-only results (I ran 50 simulations for 500 iterations each, and averaged all the results):

Player	Avg winnings	Avg node degree
Aᵀ	626.181	11.61
Smooth Criminal	625.379	8.02
suspiciouslyOdd	618.934	2.28
Pocoyo	552.462	7.50
Sucker	545.710	8.00
CaffieneAddict	511.135	7.63
ColmTheGOAT	504.759	7.96
MyNameIsRetep	485.058	7.57
Goblin	480.656	7.98
Browser	470.466	6.51
Mean	440.548	7.99
Stephen Davies (not the real one)	429.481	5.87
TitForTat	392.847	8.00
GrilledCheeseSandwich	338.137	7.98
Evenstar	313.667	2.65
Peachy Pearl	200.039	3.85
Al-Khwarizmi_	-1840.430	16.13
StrawberryFinch	-2228.890	23.97

That last column is the average degree of the player’s node in the network at simulation’s end. I find that a fascinating column. One might think: “having too many friends is a recipe for failure,” given the large degree averages of the players at the bottom end of the scale. However, the big housefly in that ointment, of course, is that the winner of the competition (Aᵀ) had quite a high average degree.

Here’s a scatterplot of the results (omitting the outliers) if you’re interested. There are apparently multiple very different paths to victory: just look at the top three finishers!

Here’s one sample run (37 of 50, I believe) which is interesting to watch. (MartyMauser does particularly well in this one, but of course it’s the overall performance over many seeds that takes the cake.)

At any rate, I award the following points:

• +40XP to the nine players who beat TitForTat
• +30XP to the five players who didn’t
• +100XP to Aᵀ‘s final exam score (capped at 100XP)
• +50XP to Smooth Criminal‘s final exam score (capped at 100XP)
• +20XP to suspiciouslyOdd‘s final exam score (capped at 100XP)
• +10XP to Pocoyo‘s final exam score (capped at 100XP)

Congratulations to all our winners! And now on to the LLM competition…*(pant, pant)*

Stephen’s trying sooooo hard to complete a fair contest before final grades are due at noon tomorrow…

The good news is that only a couple of players (Garrett and Belen: tsk, tsk) are crashing my simulator with errors.

The bad news is that when I run everyone’s rules-based and LLM players against each other, it takes about one hour to run 4 steps. If I can get it crash-free, and start it tonight before I crash, I can probably get a 50-iteration competition to complete in time. Stay tuned…

I finished grading the reflection papers and will return them at the final exam today. Let me just say that a number of these were truly outstanding analyses. I feel like I learned as much from some of these papers as I did from Nate’s book itself! Everyone got at least 40XP from these, and in a few cases I couldn’t resist awarding even up to 55XP. You guys are a truly brilliant and creative group, and I feel honored to be your teacher!

I know that those who haven’t had CPSC 240 are wrestling a little bit with this whole notion of “writing a class that has methods which some other code (not written by you) will call.” So just to clarify:

Nowhere in your code will you call any of the following methods:

• .decide_against()
• .inform_foaf()
• .request_rewire()

Instead, my simulator will be calling those methods of yours (specifically, if you care, on lines 220, 151, and 254 of model.py, respectively.)

So don’t get wrapped around the axle trying to figure out where to call your own .request_rewire() method. You won’t call it. I will.

In contrast, your code will call:

• self.model.request_foaf_info_from()

when you want to know who your neighbors’ neighbors are (and their play histories).

Send email if you’re confused about any of this. I’m happy to explain.

A diligent and sharp-eyed student studying for the final discovered that there was an error on Quiz #4, problem 6: the “number of nodes desired” absolutely is a parameter of Watts-Strogatz random graphs, but the quiz did not have this marked as a correct answer.

So, I gave the whole class an XP in recompense for what I’m sure were widespread grading errors. You can thank Evenstar for this point!

I was slow, so suspiciouslyOdd enhanced the IPD simulator with the as-per-agreed-upon-rules charges/bonuses now correctly levied. git pull to get the updated simulator.

Several students have noticed and (understandably) complained that the currently simulator doesn’t actually levy the charges for (1) asking a neighbor for foaf information ($1 charged), and (2) lying in your foaf information ($.5 charged), nor does it give a bonus $1 for providing correct foaf information. This will be coming soon, as soon as I catch up on some other tasks, probably tomorrow by midnight.

I’m getting a couple (good) questions about how to compute the node ID (in the IPD graph) for a particular agent. Some of you have discerned that the node ID seems to be equal to the agent’s unique_id minus 1. You can’t count on this, though, since Mesa leaves open the possibility that it will assign unique_ids differently.

This is what the .node property is for (see lines 266-268 of base.py). If you have your hands on an instance of IPDAgent (including any subclass) called, say, “a“, you can simply do:

    ...  a.node  ...

to get its correct node ID.

Well, the calendar isn’t being very friendly. I’m trying to juggle when I make homeworks #7 and #8 due, in order to maximize (1) the amount of time you guys have to work on them and also (2) the probability of me actually being able to run a complete simulation by the time final grades are due.

So I’ve converged on Thursday, April 30th at midnight as the final due date for both assignments. I look forward to competing against your players mere moments after that!

Homework #7 has been posted at last. I decided to split this IPD Tournament into two parts: the rules-based agent phase and the LLM-based agent phase. That latter will be called “homework 8,” and will be posted later in the week, and will be worth up to an additional +40XP (Another bonus opportunity from Stephen!)

Get started and ask questions — this one is involved!

I just ran my woosy LLM player (not a long prompt, not very involved logic) on my CPU/GPU in a 10-round IPD game against 25 opponents. That agent’s node had a degree of 6 throughout. On my system, taking no special steps to do anything fast, this took 3 mins 40 secs.

Assuming linearity (possibly conservative, but then again GPUs are hard to use in parallel if you don’t have additional hardware sitting around to exploit) and assuming our contest has 25 LLM players (1 per student) also with average degree of 6, here’s a rough estimate for how long it would take to run games of varying lengths:

1. 10 iterations — 5500 secs = 91.7 mins = 1.5 hours
2. 100 iterations — 55,000 secs = 917 mins = 15.3 hours
3. 1000 iterations — 550,000 secs = 9167 mins = 153 hours = 6.3 days (Stephen’s original plan)
4. 1,000,000 iterations — 550,000,000 secs = 9,166,666 mins = 152,778 hours = 6365 days = 17.4 years (Garrett’s and Marina’s preference)

Quiz #7, an unexpected opportunity for you to earn more points, has been posted and is due on April 25th.

Quiz #6 has been posted! It is due midnight April 20th, but do not smoke weed while taking it. It is open-book, open-notes, and open-calculator (you’ll need one), and timed at 63 minutes. Good luck!

Apparently I forgot to push the parallelized version of the BVM code from last week; it is now in the github repo, in the file surg_par.py.

Your ideas are solicited for how the following aspects of the IPD rulebook should be specified:

1. What kind of random graph should the agents be put on at the start of the game? And what should the nodes’ average degree be when play starts?
2. How many rounds should the game last at minimum?
3. Once the minimum number of rounds has been played, what should the probability p be of each successive round being the last one?
4. How many dollars should each of these actions cost:
   1. Deciding to seek FOAF information from a neighbor?
   2. Refusing to provide FOAF information to a neighbor when requested?
   3. Providing incorrect FOAF information to a neighbor when requested?
   4. Performing one rewire operation (drop a current neighbor, pick up a new neighbor?)
5. How many rewire operations should be permissible each round?
6. What should the token limit be for prompts to the LLM? (you can specify an answer in characters instead of tokens if you prefer)
7. What should happen if an LLM agent exceeds the token limit?
8. How many dollars should players whose real-world initials are “GM” be penalized at the start of each round?

You may give ideas about how any of the above should be handled by sending me an email with subject line “DATA 420 IPD rule preferences” no later than noon on Thursday, April 16th. The class will vote on all rules in class that day.

Only one rule preference email (which may itself contain any number of rule preferences) will be accepted per student. If there is another aspect of the rulebook that you’d like to weigh in on, please mention that in your email as well.

Wow, congratulations to Miranda for being singled out for this honor!

My recovery is going quite well so far, but all the same I think I need another slow day before I’m ready to re-enter the rat race which is UMW campus. So office hours are canceled for Monday, April 13th, but I plan on resuming class and office hours as scheduled on Tuesday, April 14th, at the usual times.

For anybody following this developing story, my surgery seems to have gone well and the doctor is pleased. He did think, however, that Tuesday might be too aggressive to plan on a return to the classroom. I’m going to see how I feel. I might come in that day if I feel good enough (and that my voice can sustain a pair of 2-hour classes), or I might do Zoom, or cancel altogether. Stay tuned on this website for the latest details!

Here’s the example we did in class of using batch_run:

from mesa.batchrunner import batch_run

    ... your amazing simulation model here ...

params = {
    "N": 150,
    "leave_prob": np.arange(0,1,.1),
    "edge_prob": np.arange(0,.5,.1),
    "prob_blue": 0.5,
    "fig":None,
    "ax":None,
    "do_plot":False,
    "seed":None,
}

if __name__ == "__main__":
    results = batch_run(
        model_cls=Society,
        parameters=params,
        rng=range(10),
        max_steps=None,
        data_collection_period=-1,
        number_processes=None,
    )
   print(pd.DataFrame(results).groupby(['leave_prob','edge_prob'])['Step'].mean())

This is the version that does a double-parameter-sweep on two i.v.’s (independent variables). One way to plot this intelligently, btw, is to use a heat map, which has the two i.v.’s on either axis and then uses the color of each cell to display the d.v. Here’s my preferred method (which uses the seaborn plotting library, which you must pip install):

heat = (
    results.groupby(["iv_1", "iv_2"])["dv"]
       .mean()
       .reset_index()
       .pivot(index="iv_1", columns="iv_2", values="dv")
)
sns.heatmap(heat)

Aetna came through at the eleventh hour today, and said they would cover my surgery. So I’ll be under the knife tomorrow morning bright and early, and will be canceling all my campus appointments (including class and office hours) for this week.

If all goes well, I’ll be back in the saddle on Monday, and will have office hours that day as usual. Stay tuned for details.

A few people have inquired as to whether they can do homeworks 5 and 6 on the cpsc server. The answer is yes, with two caveats:

• Do realize that like all remote servers, the cpsc server is “headless,” meaning you get a command line to it but no graphical display. Thus you won’t be able to run the animation and see it run. You can, however: (1) save plots (with the plt.savefig("filename.png") command) and download them to your machine to view, and also of course (2) run time-intensive parameter sweeps.
• The default version of Python on cpsc is really old (version 3.10) and incompatible with the latest Mesa release (which uses the discrete_space system we’ve been relying on). There is, however, an updated version, which you can run by typing “python3.12” instead of “python“.

Homeworks #5 & #6 have been framed as one large, combined assignment, and are worth +80XP together. Even though this monster consists of two related-but-separate parts (which you can attack in either order) you’ll be turning in just one submission, as the assignment explains.

Observation: if procrastinating on a big assignment is bad, procrastinating on a double-assignment is twice as bad!

Have fun with this!

I’ve posted the raw data for the class graph in two edge lists:

• class_graph.csv — the original “people whose names you remember” graph
• class_graph_less_dense.csv — the second “people you’ve had a conversation with” graph

Each one of them contains rows with two comma-separated entries: the rememberer, and the remembered.

networkx can load such a .csv file into a graph via:

g = nx.read_edgelist("name-of-file.csv", delimiter=",")

as illustrated in the class_graph.py file I also pushed.



Have fun!

Using the “have had a conversation with” criterion, our class graph looks a lot more manageable (click to enlarge):

And here’s the degree distro:

On Tuesday (tomorrow), office hours will be:

1. Online, instead of in person (see Canvas for the usual Zoom link), and
2. 10:30-11:30am instead of the usual 10-11am.

Clone this repo:

https://github.com/divilian/IPD-LLM

and run the program as follows (for instance):

$ python cli.py 150 \
    --agent-fracs TFT 0.3 Mean 0.3 Sucker 0.4 \
    --p-same 0.1 --p-diff 0.01 \
    --num-iter 100 --plot

You can run “python cli.py -h“, too, to see all the command line arguments and what they mean.

Bonus points to anyone who can vary the parameters such that (1) they maintain at least 5% of all three non-LLM agent types (Mean, Sucker, and TFT), and (2) the tit-for-tat agents, on average, come out the winners.

Quiz #5 has been posted to Canvas, and is due as the month of March expires. It’s timed at 60 minutes and otherwise has all the normal rules. Good luck!

You guys did a really good job of remembering each others’ names. Clearly this was the wrong bar to set for creating an edge in our class graph, because the thing is super dense (click to enlarge):

I’d like to get a more meaningful graph for analysis. So, since I’m giving away XP like candy, here’s another way to earn an additional +1XP: send me an email with subject line “DATA 420 class graph” that has the first name of every student in the class that you can remember having a conversation with. It might have been a one-on-one conversation, or a group conversation in which you and the other student were both present.

The conversation can be of any length. For instance, this counts as “a conversation”:

Them: Yo.
You: S’up.

(If you’ve never had a conversation with anyone else in the class, that’s fine; you can just say “nobody” in your email.)

I’m expecting that with this new, higher bar for edges, most students will have a degree of between 0 and 8, instead of this craziness:

If you’re interested in pursuing a masters degree at Virginia Tech, consider checking out this information session about the program.

I’ve now posted the reflection paper assignment I promised, and I hope many of you will consider completing it. It’s due in a month, so this should give you plenty of time to get your creative juices flowing and to digest many things that are in the book.

One word of warning: grading for this paper will be rigorous. I am not looking for two pages of crap, or for two pages of a stream of consciousness. If you turn in crap, expect to get 5 or 10 XP for your grade, depending on the level of crap. To get a score more towards the maximum, I’m looking for excellence. Take pride in your work, give yourself enough time to succeed, and bounce ideas off myself or others, and you should be well-positioned.

Homework #4 is posted, and is due next week. There will be no extensions to this one, so don’t expect one. Time to get started!

Click for a close-up!

What do people think Samuel is doing in the picture? Here are the results: