Teaching Logbook

My teaching logbook consists primarily of a sheet recording all of the classes I teach, including:

• Term
• Course (number and title)
• Section (number, grad/undergrad, modality)
• Enrollment
• Student eval summary (instructor rating course rating)

This table can then almost be copy-pasted as-is into portfolio or dossier evidence sheets, and makes it easy to generate statistics (e.g., my department wants to see the average instructor & course ratings broken down by modality).

Research Logbook

My research logbook has two different primary log tables in it: one for submitted papers, and another for submitted proposals.

=LET(refdt, MAX([@[Last Active]], [@[Last Submitted]], [@[Last Published]]),
     IF(AND(refdt < TODAY(), refdt >= DATE(2000, 1, 1)),
        DATEDIF(refdt, TODAY(), "m"), 0))

Logging Service

These ideas can also be extended to logging service activity. I do not currently use spreadsheets for service logging, but it’s a good idea. I do log my reviewing activity in an Obsidian document, and automatically extract the review section of my CV from that task list.

Concluding Thoughts

As with saving documentation and evidence, it’s easier to summarize and report on your activity if you keep records of it as you go. A few spreadsheets can make that work, and also support personal academic analytics as well as make it easier to prepare things like NSF COA lists.

Programming

The only particularly interesting change in my programming toolset this year has been leaning more on ZSH shell scripts for various automation things and even nontrivial logic. I wrote dumptruck(8), an encrypted backup management tool I use for backing up some of my servers, in Z Shell. I also replaced several simpler TypeScript programs and with ZSH scripts, and wrote myself a small standard library to share code across the various scripts and repositories I use.

Shell script is notoriously difficult to write correctly, but both Bash and ZSH provide quite a few features to make it easier if you’re willing to depend on them. I target Bash for most scripts in repositories others need to work on but fully rely on ZSH for things that are mostly me. It’s my standard login shell and I have it installed on all my machines, so the dependency isn’t a problem.

I also moved most of my TypeScript/JavaScript code from Deno back to Node. There are many things I like about Deno as a platform, including the first-class TypeScript support, but deployment and building is a pain (they really only support their own binaries, and keeping the Conda packages working was rather a lot of work; last I touched them they had some outstanding build failures that I couldn’t find workarounds for). They’re a VC-funded product, too, which seems to bring various forms of misalignment between community and funder needs. I was also extremely annoyed by their deployment of an AI bot on their support forums to barf incorrect information as a first reply to user questions. I’ve been toying with the idea of using Haskell + Hakyll to build my website again.

Git Hosting

Last year, I was using soft-serve for hosting my private Git repositories (esp. with many LFS files), and early this year I set up Laminar to run CI jobs to rebuild my website. They’re both nice pieces of software, and fine to use if you don’t need anything more than command-line Git hosting and CI jobs.

However, I started needing more than that. This month I migrated to a self-hosted Forgejo instance for my local Git hosting along with Woodpecker for CI. Now that I have the Woodpecker runners stable and the relevant permissions and caching to reliably and efficiently rebuild my website, I’m pretty happy with this setup. Forgejo also provides package repository support for probably a dozen different package systems, including Debian apt, RPM/DNF, Alpine Linux, and Docker, so it serves as a private container image registry as well as Git and Git-LFS hosting. This was my primary impetus for the switch, as I am relying on quite a bit of containerization in my home network setup.

I’ve also started moving some of my public projects from GitHub to Codeberg. GitHub is still super useful, and its performance and community reach are unrivaled, but I’m concerned about the reorganization to put it under the Microsoft AI division. I still keep things that I hope to build community around on GitHub, particularly LensKit, but my little personal things are usually going to Codeberg instead now.

Tasks and Dependencies

Early this year, I discovered mise-en-place, a package that can install software (from multiple different sources, including a registry I hadn’t heard of before called Aqua), manage project environments, and handle task running. I’ve adopted it for managing my home directory software and across many of my projects. It has largely replaced direnv, just, and several uses of pixi or conda in my workflows. Simple task workflows are easier to write in just, but as tasks get more complex, mise’s support for writing tasks as standalone shell scripts provides a better complexity scale-up story. For several of my projects now, the interesting project-local development and management scripts are ZSH scripts written to be run by mise.

Mise provides help, dependencies, and command-line parsing (through usage, which can also be used on its own), so it’s pretty easy to write interesting and reasonably well-documented automations with Mise.

Mise also introduced me to sops, which turns out to be a great way to manage secrets in a project repository. I’ve now been using it, both with Mise and on its own, to manage secrets for several projects (including this website).

As both uv and published Python packages have improved in quality, I’ve stopped using Conda or Pixi for most projects. I’ll still use Pixi for projects with significant non-Python dependencies I want version-locked, but installing scientific Python packages from PyPI works a lot better now than it did a few years ago, and I have slightly fewer difficulties with it than I do with Pixi/Conda. Standard Python installations with uv’s dependency locking work great for a lot of things. For solo projects, I then use Mise or Aqua to pull in other dependencies if needed (or put them in a container).

Home Computing Platform

Concurrently with moving to Forgejo, I also reworked our home networking environment. I plan to write a separate post about this once a few more pieces are finished, but I moved both our home and cloud servers from Debian to AlmaLinux to take advantage of bootc. Some years ago I used NixOS, and wanted for some time a way to declaratively manage my system configurations with a more normal Linux and without the weirdness of the Nix expression language. I’ve experimented with things like Ansible and PyInfra to automate management, but true declarative management they are not.

Turns out bootc scratches that itch! It lets me manage system state as bootable containers, so I write a Containerfile (and associated scripts, configuration, etc.) to assemble my system as an OCI (Docker) image, and bootc installs and upgrades the system with that image. Mutable local state lives in /var (and /etc by default, but I’m working towards the recommended configuration of making /etc fully immutable/ephemeral as well).

When this migration is finished, I think I’ll have achieved my home admin goal: a git repository defining the software load and configurations for the various servers on our network, making changes through configuration edits + deploys. Forgejo’s container registry support plays a vital role here, as the container-based systems install and update from it.

I also replaced Tailscale with Nebula. Tailscale is great, I highly recommend it and have few complaints. I mainly wanted slightly different firewall / access control capabilities than it provides; I also have a small amount of nervousness about two single-points-of-failure (Tailscale and GitHub authentication) for my network security, so managing my overlay network in a self-hosted Git repository of machine definitions and certificates is a nice benefit. Nebula also provides front-line security for sensitive network services: anything I don’t want chatted at by unsecured devices is only on the Nebula network, even within our LAN, so it’s secured from the various IoT devices and other things that might be on the network. I’ll be using WireGuard to provide myself a roaming VPN for travel and off-site access (accessing Nebula from outside our home network is no problem, but it doesn’t support exit nodes like Tailscale, so a full exit to protect from dodgy Wi-Fi needs another solution).

With Windows 10 end-of-life this year, I switched our desktop over to Fedora. It runs (older) Steam games fine, and we don’t have a burning need to upgrade the machine.

Document Preparation

This year I also started exploring Typst, and love it. My use of LaTeX has increasingly been begrudging over the last few years, and Typst is the most promising entrant in a while to try to deliver strong typesetting capabilities with a programming language that isn’t screamingly old in its design principles. My CV is now in Typst, and I’m using it for other documents that I don’t need to collaborate on. I plan to use it for a lot of my course materials next term (and possibly also my lecture slides, we will see — I like PowerPoint, but PowerPoint’s lack of style support makes it annoyingly difficult to consistently typeset code snippets).

It isn’t ready to replace LaTeX for writing papers for ACM conferences and journals, but I wouldn’t be surprised if it gets there in the next five years (whether ACM’s backend will support it is another question, whose positive resolution is less likely).

I have always found it far more painful than worthwhile to attempt to customize LaTeX appearance beyond changing fonts and the page/layout customization supported by the Memoir class. With Typst, making a fully-custom layout and theme for my CV, and another one to implement Drexel letterhead, was pleasant work.

Web Serving

I was not happy with my web server situation this year:

• Apache is reliable for many things, and its interesting mod_speling module allowed me to self-host my website while maintaining URL compatibility with my past hosting on Netlify, which has the interesting and questionable behavior of forcing all URLs to lowercase. However, its “managed domain” support for automatic HTTPS is glitchy and annoying, requiring multiple server reboots to bring a new domain online.
• Caddy has outstanding automatic HTTPS support, and supports 95% of my web serving needs, but it doesn’t have an equivalent to mod_speling, and I don’t know Go so writing a plugin to add that support was going to require learning both a new programming language and the Caddy API. And while I’m rarely one to shy away from a new programming language, I don’t find Go to be a very interesting language (although it’s used for a lot of great software!), so learning it isn’t a priority for me.

So I did the obvious thing, and wrote a (small) web server. The Opinionated Page Server (opserve) serves static files using a pre-generated manifest of files (with their types and hashes), with support for case-folding. It probably isn’t very useful to others, and is currently undocumented, but it does what I need it to and happily serves up my website from a container. It also gave me an excuse to play with some more async Rust. The whole thing took maybe a couple of days to write (using Trillium). It only supports plain HTTP, so I have Caddy in front of it as an SSL-terminating reverse proxy (and Bunny CDN in front of that).

Music

I continue my vinyl music collection and listening hobby, and this year upgraded my stereo receiver: replaced the Sony STR-DH180 with a used Marantz NR1200. The Marantz has noticeably better sound quality, especially for anything digital (even though the Sony supported hi-fi audio over Bluetooth, its quality was pretty poor). I’m still looking to upgrade my speakers, or at least the subwoofer, but I’m pretty happy with most of the sound now.

I replaced the Shure open-back headphones I was using at home with another pair of Sennheiser HD-600s (I already had a pair in my Drexel office), for both weight and sound reasons.

Coming Attractions

I’m always tweaking and improving (or breaking) things, as regular readers may have noticed 🙃. In the next months, I hope to finish the network upgrade (particularly adding the router to the declaratively-managed set and implementing SSO, which is the last major piece needed to switch to ephemeral /etc on managed machines), and I’ll probably write some more random scripts and Rust programs.

Design Goals

Given a spreadsheet containing application information, our goal is to randomly assign 2 reviewers (out of $N$) to each distinct application. We also want to do this in a way that is robust to reordering applications in the sheet, changing or replacing cells, etc., so that the assignments remain stable.

The easy way to achieve this would be to use the application ID as the seed to a random number generator, and then use that RNG to select reviewers. Unfortunately, Excel’s RAND function does not support explicit seeds, so we can’t do that.

Enter the LCG

If Excel doesn’t give us a seedable RNG, can we make one? Yes!

A linear congruential generator is one of the simplest pseudorandom number generator designs, and it’s simple enough to be implemented as Excel formulas. The basic ideas is to start with a seed $X_0$, and then compute subsequent numbers by $X_i = (a X_{i-1} + c) \bmod m$. It isn’t a great random number generator, but it’s good enough for what we need.

Using the application number as the seed, instead of just generating a number for each application, makes our assignments stable: given an application number, we can directly compute its assignments.

Unfortunately, it isn’t quite that simple. An LCG on its own outputs numbers in the range $[0,m)$, but we only have $N$ reviewers. To select two reviewers, we need both the two random numbers, and we need to convert them into the range $[0,n)$ (and for the second, skip over the reviewer we picked in the first step).

To implement this all of this, our workbook has two sheets: the applicant info sheet and the “AssignRNG” assignment worksheet. “AssignRNG” implements the LCG, split out into different columns of a table (RVRNG) that allow us to inspect the LCG’s operation, and compute each of the reviewers.

On the right, you’ll find the parameters, with named cells holding their values (i.e., $m$ is in the cell P.m). These parameters are:

• $m$: the LCG modulus
• $a$: the LCG multiplier
• $c$: the LCG increment / additive constant
• off ($G$): an additive offset for converting entry numbers to seeds
• mult ($H$): a multiplier for converting entry numbers to seeds

The first three parameters are the core LCG parameters. Their values can be a bit finicky; Excel has limited numeric range and precision, so the values used by many implementations are too large for accurate generation in Excel. The Wikipedia article has a table of values from different implementations. These are from random0, an old implementation with a low period (the generator will start repeating more quickly than you would want for a lot of purposes), but they are in-range for Excel and good enough for our purposes, especially since we are only generating two random numbers from each seed.

The last two parameters are not LCG parameters, but control the seed generation process. This control allows us to change the randomization each year. The right portion of the sheet also contains the list of reviewers, counts of the number of times each is assigned in the RNG table, and counts of how often each pair of reviewers is assigned. These counts allow lightweight randomization checks.

The RNG itself is a table (named RVRNG) with 7 columns:

1. Index is the row number in the table. Each application in the applications sheet is assigned a sequential application number, starting with 1; we look up that number in this column to determine that application’s assignment.

2. V1 and V2 are the first and second random numbers drawn from an LCG seeded with the Index. They are computed as follows:

   $$\begin{align*} s & = G (i + H) \\ V_1 & = (a s + c) \bmod m \\ V_2 & = (a V_1 + c) \bmod m \\ \end{align*}$$

3. I1 and I2 are the first and second reviewer indices. They are computed by $I_1 = V_1 \bmod N$ and $I_2 = V_2 \bmod (N-1)$. We use $N-1$ as the second modulus because we have already used one reviewer in the first selection, so we need to select from the remaining reviewers.

4. R1 selects the first reviewer, by using $I_1 + 1$ to pick a reviewer from the list of reviewers.

5. R2 selects the second reviewer, by using $I_2 + k$ to pick a reviewer from the list of reviewers, where $k=1$ if $I_2 < I_1$ and $k=2$ otherwise, to skip $I_1$ in the list.

Together, those cells select a pair of reviewers from the list, ensuring the same reviewer is not selected twice.

Mapping to Applications

Back in the application tracker, we each application has an application number, assigned when it is added to the spreadsheet and never changed. We then look up the assignment for that application number, and include it in the sheet. To do this in “wide” format, where each reviewer has a column that gets marked “X” for applications assigned to them, we use the following formula (assuming the reviewer column is E):

=IF(OR(VLOOKUP([@Entry], RVRNG, 6, FALSE)=E$1,
       VLOOKUP([@Entry], RVRNG, 7, FALSE)=E$1),
    "X", "")

Final Remarks

I’ve been through a couple of iterations to get to this point, but it seems to be a pretty robust and solid way to assign reviewers within Excel, without needing to import assignments from other tools. So long as application numbers are not changed, assignments are stable as the application table is sorted, cells are changed (e.g., replacing the “X” with reviewer comments), and most other manipulations.

Hope this is helpful! The Excel sheet is available here (local download).

Preliminaries

I’m going to use my usual notation in this blog post: we have a set $U$ of $M$ users who interact with $N$ items forming set $I$.

For convenience, let $m_i = |U_i|$ be the number of users who have interacted with item $i \in I$. We can directly sort by this value, in descending order, to find the most popular items (those items who have been interacted with by the most users). We can also frame overall popularity as the probability that a user will interact with that item (assuming, for the moment, that all users are equally likely or representative of future incoming users):

$$\operatorname{P}[i] = \operatorname{P}_{u \in U}[i \in I_u] = \frac{m_i}{M}$$

When we have explicit ratings, user $u$ gives item $i$ the rating $r_{ui}$, and the average rating for an item is $\bar{r}_{ui}$.

To recommend items, we compute some score $s(i|\dots)$ and pick the items with the highest score, or use these scors in some more sophisticated ranker such as a sampler or diversifier.

Related Products

If we want to move beyond overall popularity to contextual probability, where the context is a reference item $j$ (e.g., the item the user is looking at right now), the first thing we can try is just the conditional probability:

$$s(i|j) = \operatorname{P}[i|j] = \frac{m_{ij}}{m_j}$$

This is convenient and straightforward: for some candidate item $i$, what is the probability the user will interact with $i$ given that they have interacted with $j$? We can then use those probabilities to rank the recommendations. Joe Konstan and I taught this in the early videos in the Coursera Recommender Systems MOOC, and I keep teaching it in my in-person recommender systems classes today.

Conditional probability has an interesting problem, however: what if $i$ is very popular? If a candidate item is highly popular, it will have a high conditional probability regardless of the reference item. We sometimes call this the “banana problem”: lots of people buy bananas, so for any other item in the grocery store, there’s a reasonably good probability the user will also buy bananas. Further, most people already know about bananas, so recommending bananas is less likely to give them new information than recommending some less-common item that frequently co-occurs with the reference item.

One way to address this — and the method we taught in the MOOC — is to use lift:

$$\begin{align*} s(i|j) = \operatorname{Lift}(i,j) & = \frac{\operatorname{P}[i,j]}{\operatorname{P}[i]\operatorname{P}[j]} \\ & = M \frac{m_{ij}}{m_i m_j} \end{align*}$$

Lift measures the degree of statistical relatedness of two events. If the two events (items, in our case) are independent, then $\operatorname{P}[i,j] = \operatorname{P}[i]\operatorname{P}[j]$, and $\operatorname{Lift}(i,j)=1$. Higher lift values arise when the items co-occur more frequently than we would expect by chance if the items were independent, indicating that they are statistically related (lower values arise when they co-occur less frequently). If we want lift scores to be better-behaved numerically, perhaps to combine with other scores in a hybrid model, we can use log lift, but this doesn’t change the ordering of potentially-related items.

Lift has a different problem, though: what if $i$ is very unpopular, and therefore $m_i$ and $\operatorname{P}[i]$ are small? Perhaps it is so unpopular, that its only interactions co-occur with interactions with $j$? In this case, lift will be very high, but has low support: we are making very confident conclusions of item relatedless on the basis of a small number of interactions (sometimes only a single interaction).

What can we do about this?

Sidebar: Bias

The bias-variance tradeoff is our key to solving this problem. To see how that might work, let’s go to a different model that we also taught in the MOOC and discussed in our old survey, and I picked up in turn from FunkSVD, I think: the bias model.

If we want to compute the average rating for an item, the basic way to do this is with a simple mean:

$$\bar{r}_i = \frac{1}{|U_i|} \sum_{u \in U_i} r_{ui}$$

The mean has a similar problem as lift — if one user really liked a movie and rated it 5 stars, it will have an average of 5, beating out movies that a lot of people rated and gave a respectable average of 4.7 over, say, 10K ratings.

An easy fix for this problem is to introduce some damping or bias based on the global average rating¹:

$$\dot{\bar{r}}_i = \frac{\left(\sum_{u \in U_i} r_{ui}\right) + \gamma \bar{r}}{|U_i| + \gamma}$$

There are at least two ways to think about this new term $\gamma$:

• A priori, we start each item out with $\gamma$ virtual ratings equal to the global mean rating. This reflects a prior assumption that, absent infromation about user preference, items are probably average. As the item accrues more real ratings, those dominate the virtual ratings and the damped or biased mean converges towards the empirical mean. This is conceptually similar to Laplace smoothing for naïve Bayesian modeling.
• If you push through the calculus, this turns out to be equivalent to computing the posterior expected value of Bayesian inference for the item mean rating with an emirical prior and a Gaussian likelihood function. Specifically, the prior for the inference is a Gaussian distribution with an empirical mean ($\mu_0 = \bar{r}$) and variance derived from the damping strength $\gamma$.

Either way we look at it, the result is to bias the mean, decreasing its variance (in the bias-variance sense) and decreasing its susceptibility to deriving high scores from little information.

The bias model provides another more flexible mechanism to introduce this bias that has the benefit of also accounting for users’ differing use of the rating scale. We can learn global, user, and item biases and combine them to compute a personalized mean:

$$\begin{align*} b_{ui} & = b_0 + b_i + b_u \\ b_0 & = \bar{r} = \frac{1}{|R|} \sum_{r_{ui} \in R} r_{ui} \\ b_i & = \frac{\sum_{u \in U_i} (r_{ui} - b_0)}{|U_i| + \alpha} \\ b_u & = \frac{\sum_{i \in I_u} (r_{ui} - b_0 - b_i)}{|I_u| + \beta} \end{align*}$$

In this version, we center the data at each step, computing the item and user biases from the residuals of the previous step: the item bias $b_i$ captures how much better or worse this item is than the global average rating across all items, and the user bias $b_u$ captures how much more positive or negative this user is than average.

We further bias these biases towards 0 with bias damping factors $\alpha$ and $\beta$. Since the item and user biases are computed from mean-centered data, 0 is now a neutral value, so we can apply the damping factor only in the denominator and obtain our biased biases that require more ratings to learn a high estimate of an item’s quality.

The bias model also yields a third interpretation of the damping or bias: while the precise parameter values might differ slightly due to the step-by-step computation above, the resulting model is very similar to what we would learn if we treated rating prediction as a ridge regression problem with user and item IDs as categorical variables. I sometimes do that, particularly when using the bias model as one component of a more sophisticated regularized scoring model.

The point of all of this for our original problem is that we can decrease the problematic behavior of our scores with respect to low-information items by biasing the scores. We can implement this biasing by starting each item out with “virtual interactions” expressing neutral preference. The more virtual interactions we supply, the more empirical interactions are needed to support a high score.

Biasing Lift

This raises a question: how do we apply the same principle to probabilities and lift? What would it mean to start each item out with some number of “neutral” interactions for the purpose of lift?²

Working this out is a little less obvious than starting each item with some neutral ratings. Both because we need to define “neutral”, and because there will be some additional terms involved (the number of neutral interactions per item is not sufficient for the math to work out).

Let’s define “neutral” as “independent”: our goal is for each item to start out with $\kappa$ virtual interactions that are independent of any other item’s virtual interactions. To compute probabilities, we also need to know the number of virtual users ($K$). We can set this directly, but it is more natural to derive it from $n_K$, the (average) number of interactions supplied by each virtual user. We’ll see at the end that we can actually ignore $n_K$ and $K$ and derive a sufficient approximation of biased lift with only $\kappa$, but we’ll use them for now.

If each item has $\kappa$ virtual interactions, we need $\kappa N$ total virtual interactions. If each virtual user supplies $n_K$ virtual independent interactions, then the virtual user count $K = \frac{\kappa N}{n_K}$. With these in hand, we can define the prior probability of an item, or the probability that one of our virtual users will interact with it:

$$\operatorname{P}_K[i] = \frac{\kappa}{K}$$

Since all virtual interactions are independent, we can define the prior joint probability of individual item pairs:

$$\operatorname{P}_K[i,j] = \operatorname{P}_K[i] \operatorname{P}_K[j] = \frac{\kappa^2}{K^2}$$

To define a biased probability $\dot{\operatorname{P}}[i]$ that incorporates our virtual clicks, an item with $m_i$ real clicks will have an additional $\kappa$ virtual clicks from a pool of $K$ virtual users. This yields:

$$\dot{\operatorname{P}}[i] = \frac{m_i + \kappa}{M + K}$$

Next, we will define the biased joint probability $\dot{\operatorname{P}}[i,j]$. For this, we need to know expected number of virtual co-occurrences a pair of items will have: of our $K$ virtual users, how many will have interacted with both items? This is $K \operatorname{P}_K[i,j] = \frac{\kappa^2}{K}$, allowing us to compute the biased joint probability:

$$\dot{\operatorname{P}}[i,j] = \frac{m_{ij} + \frac{\kappa^2}{K}}{M + K}$$

We can plug these biased joint and unconditinoal probabilities into the lift formula to get biased lift:

$$\begin{align*} \operatorname{BiasedLift}[i,j] & = \frac{\dot{\operatorname{P}}[i,j]}{\dot{\operatorname{P}}[i]\dot{\operatorname{P}}[j]} \\ & = \frac{m_{ij} + \frac{\kappa^2}{K}}{M + K} \cdot \frac{M+K}{m_i + \kappa} \cdot \frac{M+K}{m_j + \kappa} \\ & = (M + K) \frac{m_{ij} + \frac{\kappa^2}{K}}{(m_i + \kappa)(m_j + \kappa)} \\ & \approx M \frac{m_{ij}}{(m_i + \kappa)(m_j + \kappa)} \end{align*}$$

The last simplification is not a precise computation of the biased lift, but can be used to identify the items pairs with the highest lift. We selected $\kappa$ to be a constant, and $K$ is fixed given the data set (since it is computed from fixed $N$ and the constants $\kappa$ and $n_K$). Therefore, adding $K$ or terms involving only $K$ and $\kappa$ do change the order of item pairs within a single data set.

Further, $K \gg \kappa^2$, so adding $\frac{\kappa^2}{K}$ in the numerator is adding a very small quantity, and can be effectively ignored for approximation purposes.

Since the final ordering score no longer depends on $K$, we actually do not need to pick $n_K$ if we just want to use biased lift to select the most related recommendations for a reference item. We can select $\kappa$ (higher values increase the number of real co-occurrances needed to obtain a high biased lift score), and sort by biased scores computed directly from interaction counts:

$$s(i|j) = M \frac{m_{ij}}{(m_i + \kappa)(m_j + \kappa)}$$

And it works! I don’t know how well it works yet, overall, but in my preliminary trials in class, it did exactly what I hoped, computing related-book recommendations that weren’t dominated by either most-popular-overall or unpopular-but-that-one-user-also-read-the-reference-book recommendations. I’ll probably add it to LensKit soon.

Hardware

My core hardware is the same as year; there are a few tweaks or additions:

• Bought a modern gaming rig at work for ML development and testing to increase my lab’s compute capacity and allow me to work on recommender software development while the datacenter machine is busy with bigger jobs and student work. It’s an HP Omen 45L with an i9 14900K and GeForce RTX 4090. After the first of the year, I will also be adding a Geekworm PiKVM for remote management.

• The RockPro64 board in my home NAS died, so I replaced it with a cheap NAS/HTPC machine built on the Intel N100 chipset (link). It’s been working great. I also added a Sabrent 4-bay USB 3.2 drive enclosure with a couple of 16TB drives to expand my data storage; now that it’s Intel-based, I switched the main file system from XFS back to (mirrored) ZFS for increased flexibility with external storage expansion.

• Bought a Steam Deck for gaming (and software testing).

• Bought a 144TB 8-bay Thunderbolt RAID for university data archival and slow storage.

I’ve also been working with Drexel CCI IT to set up the ability to monitor power consumption for my work compute resources; I’ll write more about the details of that another time.

Productivity and Work Management

After experimenting with Logseq for a few months, I have switched to Obsidian for project and task tracking. Logseq’s data model might be a better fit for what I need, but the lack of good documentation for its query language was increasingly frustrating. Obsidian lets me write my custom query and reporting logic in plain JavaScript.

Operating Systems

Nothing new here; still macOS on endpoints, Debian on personal servers, and Ubuntu (grudgingly) on work servers. I’ve got a Windows 11 ARM install in Parallels on my Drexel MacBook, and we still have Windows on a desktop at home. That machine will get a refresh with either Debian or Fedora in 2025, since it is too old to run Windows 11.

I’ve started using Lima for managing Linux VMs on Macs (it provides a WSL2-like experience), and Colima for hosting Docker containers. I’m very happy with both; in particular, Colima’s seamless support for x86_64 Docker containers via Rosetta 2 is very nice.

Data and Code Management

I’ve been moving away from git-annex for data management; it’s nice, but it’s also more manual than I would like for a lot of cases, and while it works ok on macOS, it’s very Linux-oriented.

For synchronizing large data archives (e.g., my academic portfolio or teaching archive), I’ve gone back to Syncthing; once set up, it (mostly) just works, and doesn’t require any manual steps for keeping data in sync.

For assets in my website, I’ve switched back to git-lfs. I set up Charm’s soft-serve to store git + git-lfs data without GitHub’s storage and bandwidth quotas, and after a bit of juggling to figure out how to build it into a Podman container with Tailscale support, it’s been excellent.

I’m mostly using WebDAV for other kinds of data servers these days, especially for DVC remotes and my Kopia backups. Caddy has been great for a lightweight, efficient server (as well as a local development server for web work), and I’m using Apache for more traditional setups (my web site is published with Apache, as is my research group’s data repository). They’re both mostly fine, each has capabilities the other lacks. I did have problems when trying to use Apache mod_dav as a Kopia remote, though — Kopia makes so many small transfers Apache was spending all its time authenticating them, whereas Caddy handles the load without problems.

I’ve switched from my own scripts on top of a Git repo to chezmoi for managing home directory config files.

Development and Analytics

My development stack is mostly stable these days; mostly Python + Rust + TypeScript (usually in Deno). I’m not using Tcl as much as I was (and no longer use it in my website). Interesting additions include:

• just for task running in various projects.
• Pixi for installing Conda-based software environments; it has a better project and lockfile UX than Conda and conda-lock.
• Using PyTorch much more, including as the core of most LensKit models.
• DuckDB for large single-node data processing; so far, it’s working better than other things I’ve tried, including Polars and DataFusion, and has a much more stable developer experience.
• uv for non-Conda Python project management.

Tools and Utilities

I’ve picked up or changed a few of the smaller bits in my toolset as well:

• Switched from Barkeeper to Ice after Bartender was sold without transparency.
• Maccy for clipboard history; works as well as or better than Paste for my use cases and is cheap.
• Adopted Parallels Toolbox for a number of tools, including on systems without Parallels. Particular tools I use from it include the system monitor, the disk cleaner, and the uninstaller. I don’t like the Parallels menu bar hider or clipboard history, they are significantly buggier than Ice or Maccy.
• Replaced iStatMenus with the Parallels system monitor, for now; I was having Bluetooth monitoring performance problems with iStatMenus. May revisit this.
• Atuin for command-line history in zsh.
• Spaceship for my ZSH prompt.

Future

I’m sure my stack will evolve in the next year, but I don’t have any specific changes I’m looking to make.

Microphones

Audio quality is, in my opinion, the biggest boost to video recording or conference quality — visuals are good and important, but sound quality is vital to making sure you are clearly understood. A good quality microphone setup will do a few useful things:

• Clearly capture your voice.
• Not capture other sounds in the background.

There are other things you probably want to consider as well, such as form factor, size and weight, and price; different ones will also fit different spaces better. I currently use 3 different microphones for my home office, university office, and on the go. Microphones are also a place where Wirecutter’s recommendations are pretty good.

I also recommend a microphone setup that connects to your computer with USB rather than the microphone jack: this keeps the analog audio processing circuitry separate from your computer to reduce EM noise interference with your audio signal. MacBooks have pretty clean audio processing, but PCs can be a mess in that regard.

For capturing just your voice, either for a recording or a conference call, you probably want a cardioid microphone: the microphone’s pickup pattern focuses in front of the mic, with little response to sound from the sides or rear; this improves sound isolation (while requiring you to be somewhat careful how you position the mic). For other contexts, you need other pickup patterns: for a group of people, for instance, you need an omnidirectional microphone.

If you watch video podcasts, you probably see 2–3 microphones a lot: the Blue Yeti, the Shure MV7, and the MV7’s older and larger brother, the Shure SM7. Both the Yeti and the MV7 are excellent microphones. I haven’t used the SM7, but assume it is quite good; it is most often used for vocal recording in studio environments, while it is my understanding that the MV7 is designed for use in less sonically refined spaces like offices with poor sound isolation, and from my experience it is very good in them.

Cameras

MacBooks, iPhones, and iPads, as well as higher-end Android phones, all have very good cameras. However, if you are using an external monitor, or have a PC laptop, you probably want a webcam. A good camera makes it easier for others on the call or watching the video to see you, and also provides a better image for background removal (either with AI-powered virtual backgrounds or traditional chroma-key).

For an effective camera, I like to see a few things:

• Full HD (1080p) resolution — even though final video is most often in smaller windows on Zoom or shrunk down in a lecture video, having full HD gives me a high-resolution original for video editing, and lets me use the camera video at full frame for intros/outros and other contexts where I’m not combining with other visual elements (I render all my lectures and other recorded videos at full HD resolution).
• Clear, crisp image without significant noise or grain in normal office lighting. Supplementary lighting is useful for a quality boost in recordings that need to meet a higher quality standard, but I don’t want to need to shine extra lights at my face for all of my calls and utility recordings.
• Reasonably accurate and vibrant color. This will vary a lot; Apple does significant image processing to ensure that their products’ camera images look clear and vibrant. Other cameras often have fuzzier or less saturated colors, and a cheap off-brand webcam we bought for home use when name brands were hard to get in 2020 has truly terrible color reproduction.

I’ve used a few different external webcams in recent years:

• Logitech Brio 4K Pro (MSRP $200)
• Logitech C920s (MSRP $70)
• Anker PowerConf C200 (MSRP $60)
• Logitech C615 (MSRP $50)

I used a Brio as my primary webcam at Boise State from late 2021 onward. To save some startup funds at Drexel, I went with the C920s for my university office (Wirecutter’s former pick), and I recently bought the Anker C200, Wirecutter’s budget pick, for my home office. The C615 (or its predecessor) was the first webcam I bought at Boise State back in 2016.

Of these, the Brio has the best image quality by far, but at a steep price. I’m not very happy with the C920s; the image is pretty grainy, and it isn’t enough better than the less expensive cameras to really feel worth it. The Logi Capture app that is necessary to change camera settings, like the focal length, also doesn’t support modern Macs (there’s a new “Logi Tune” that I haven’t tried yet). For its price point, I’m very happy with the Anker, and the built-in shutter is pretty neat (and easier to use than the flip-cover on the C920s).

My recommendation, therefore, is to get the Anker C200 for a stationary office setup. I don’t have any experience with the newer, less expensive (MSRP $130) Brio cameras that Wirecutter recommends; they’re probably good, but it’s a question of whether they are worth the premium over the Anker. From time to time, I’ve considered the Razer Kiyo to get the built-in light, but haven’t found it necessary yet (and external lighting is pretty easy to set up).

On the go, just use your laptop camera, at least if it’s a MacBook. If you have a low-quality PC camera, there are portable webcams you can get to clip on a laptop display, but I have no idea how good they are. It would probably work just as well to put your phone on a tripod and use that, either by joining the call from the phone or using an app to use it as a webcam for your laptop (more on that in the next section).

Lighting

Lighting is crucial to high-quality photography and videography; above a baseline camera quality threshold (which is sadly not met by a lot of built-in or budget webcams), better lighting probably improves quality more than better cameras. There are a lot of usable, inexpensive lighting options on places like Amazon; LED light panels and ring lights can help a lot with clear face capture, although the glare and lighting intensity can be quite distracting if you aren’t used to it. Diffuse, indirect lighting is good and helps reduce glare, but the equipment is often more expensive.

Before Camtasia got support for AI-based background removal, I used chroma-key for my lecture videos, and that depends a lot on good, uniform lighting on the background. I bought a few cheap 5–7” USB-powered LED lighting panels, and they got the job done, but weren’t very good; if I were setting up that kind of lighting again, I would probably invest in a more expensive panel. I haven’t personally found ring lights to be very useful, but a lot of people use them.

I don’t have any dedicated lighting right now. If I were engaging in a significant buildout like our MOOC or my data science video class again, I would probably invest in lighting, but for Zoom calls and the odd recorded presentation or lecture, I find my office lights to be adequate.

Software

For videoconferencing, the software is pretty set: Zoom, Teams, Google Meet, or whatever your meeting is using.

For video recording and editing, I use Camtasia. Its user experience is heavily optimized for lecture and demonstration videos, so while it lacks a lot of the flexibility in terms of effects and editing that a more general package like Final Cut Pro or Adobe Premier would offer, it’s much easier to use for academics and has very good support for screen capture, including slides in videos, annotating the screen (e.g. putting in callouts to highlight important screen elements), and basic transitions and effects. I wouldn’t call it the most polished piece of software, but it gets the job done.

I don’t do a huge amount of editing on most of my videos, but do clean up significant false starts (and when I detect such a false start while recording, I’ll usually pause for several seconds and restart at a clear point like the beginning of the slide to give myself a clear point to edit in the final production). I usually also do a bit with how the slides and speaker videos are arranged (often replacing my speaker background with transparency or another image), and sometimes put in some intro/outro effects and a title card. TechSmith recently added virtual background replacement to Camtasia (in 2020, the only way to do it was with a green screen and chroma-key).

TechSmith Audiate is a very useful application that complements Camtasia. Audiate uses speech-to-text to transcribe your video’s audio. You can then edit the resulting text to clean up transcription errors (so the transcript can be used for higher-quality captions), but the really interesting thing is that you can delete text such as disfluencies and false starts. When you then export the project back to Camtasia, it includes an edit list to edit those bits out of the video track as well. Polishing up video in this way is very time-consuming when you try to do it directly in the video editor, but Audiate makes it practical, and lets you do the caption cleanup and video edit in one pass through your recording. If you want linguistically-clean recordings and don’t have budget for dedicated captioning and video editing labor, it’s a pretty good option. I don’t have precise measurements, but it probably takes me about 1–1.5 minutes of editing time per minute of video to polish a video with Audiate. The biggest downside to Audiate is its price, and while many universities have site license to Camtasia and/or SnagIt, Audiate site licenses are less common; I’m therefore not currently using it, but would get it again for a significant recording effort that demands the higher level of polish it affords.

I still want to provide good-quality captions for my videos, though, so I’m using MacWhisper for transcription and captioning. Its speech-to-text capabilities are quite accurate (better than anything else I’ve used, including Audiate), and its automatic segmenting logic is also very good (Audiate was not great at that, and also didn’t provide very good manual segmenting controls when last I used it). My usual captioning workflow is this:

1. Export the audio track from Camtasia (Export → Audio Only).
2. Import and transcribe the audio track with MacWhisper.
3. Export the transcription to SRT-format subtitles.
4. Import the subtitles back into Camtasia.

Any further edits, or partial exports, also correctly affect the subtitles after this. If there is some egregiously bad segmenting, I’ll sometimes tweak up the subtitles using Camtasia’s subtitle editor; while Camtasia’s editor is extremely tedious for drafting subtitles, it isn’t bad for light editing of subtitles that are already there.

I edit and export all of my videos at full HD (1920×1080) resolution, typically to a Local File that I then upload to Panopto, Kaltura, YouTube, Bunny, or wherever I am publishing the video in question. With the subtitles in place, Camtasia’s default for a local file export is to “burn” the subtitles into the video (called open captioning); I usually turn this off, re-export the subtitles to an SRT or VTT file, and upload them separately as a caption file to the video hosting service after uploading the video. This allows the viewer to control whether captions are displayed or not (closed captioning).

Headphones and Monitors

Ok, all of that is about creating and sharing the video. What viewing and listening during production, and hearing and seeing the people you’re calling with?

Most of this is covered by general thinking about what kind of monitor and headphones or speakers you want. I recommend getting a relatively good monitor so the color reproduction is decent; this makes it easier to catch significant color or lighting problems in your video. I use the 27” 4K ASUS ProArt monitor; this is a professional-grade color-calibrated monitor. Full color calibration isn’t necessary for basic educational videos (and I don’t do color-calibrated work), but you do want pretty good color calibration. Dell UltraSharp monitors are pretty good, I’ve used those in the past; you mostly want to stay away from the entry-level monitors (especially things costing $200 or less). I have never used an Apple Studio Monitor, but don’t think they’re likely worth the very expensive premium they command — a pro-level ASUS or Dell monitor is fine, and costs $300–500 instead of $1500 or whatever.

For audio, I strongly recommend a good pair of headphones. While some calling software has pretty good software echo cancellation these days, using headphones while on call means there isn’t echo that the software needs to try to cancel. If you like listening to music while working, a good set of headphones will also usually be higher-quality sound than laptop or cheap desktop speakers.

Selecting headphones is a pretty personalized decision. A few things to think about in identifying the ones that will work for you:

• How do you want them to fit on your ears? Besides earbuds, the two major designs are circumaural where the headphone pad goes around your ears, and supra-aural where the pad goes on your ears. There are also very different supra-aural designs; small, often circular headphones like the Evolve rest gently on your ears, mostly towards the center, while larger and heavier ones press more firmly on the outer parts of your earlobes. Your particular ear size and geometry will also make a significant difference on how they fit.
• Do you want sound isolation? Curcumaural and large supra-aural closed-back headphones block out a good deal of background noise to provide good isolation on the sound that you hear. They also block out a lot of your own voice, so it can sound weird to talk while wearing them. It’s also harder to calibrate your vocal volume since you can’t hear yourself very well (sidetone helps with this). Open-back headphones do not isolate: the back of the headphone (away from your ear) is open, usually a plastic or metal mesh, so ambient sound can come in. Earbuds are also available in both sound-isolating and non-isolating varieties, depending on whether they are designed to seal your ear, in addition to having or lacking active noise cancellation.

What you connect the headphones to also makes a significant difference. I’ve mentioned “sidetone” a few times; this refers to signal from the microphone going into your earpiece (in pro-audio terms, it’s usually called “monitoring”; sidetone is what it’s usually called in telephony). It’s helpful for regulating your vocal volume, since our brains rely on auditory feedback for that process. Sidetone or monitoring is especially important with sound-isolating or noise-cancelling headphones or earphones, because they block or cancel your natural voice feedback along with the background noise. Any of the USB microphones I mentioned above actually have a headphones jack on them as well, so you can plug in your headphones to get direct monitoring: microphone signal directly goes to the headphones without going through the digital processing or your computer, so you get zero-latency monitoring of your voice. USB audio interfaces for podcasting or recording, like the Vocaster I use, also provide this feature. You can configure your computer to do monitoring with the headphones plugged in to your headphones jack, but this introduces a small amount of latency (the computer digitally encodes the audio signal, processes it through its audio stack, and decodes it back into an audio signal for the headphones); we’re talking about milliseconds at most, but it’s noticeable.

I personally find it very difficult to talk naturally with sound-isolating headphones and have had mixed success relying entirely on sidetone, so I prefer to use circumaural open-back headphones at my desk and non-sealing earbuds on the go (I do use the earphones’ sidetone, and I’ve started to get used to it, but I don’t usually aim for my highest-quality recording in that setting). The specific gear I run:

• Sennheiser HD600 open-back circumaural headphones in my university office (MSRP $450, I got them on sale for $250–300). Sennheiser is one of the best-known names in headphones (like Shure is the go-to for microphones), for good reason. I like using reference headphones (flat frequency response designed for professional monitoring and production), and the HD600 is in that category. This isn’t a particularly common taste, and is influenced by some time I spent doing audio production in my younger days; many people will probably find some of Sennheiser’s other models, like the Momentum or the HD560, to have a more familiar tonal quality.

  I’m extremely happy with the HD600s — they are comfortable and secure while also feeling light on my head, and sound fantastic. They also fit comfortably around my ears.

• Shure SRH1440 open-back circumaural headphones for my home office. These were the first open-back headphones I bought. I like their sound a lot (and they definitely sound better than anything I had purchased before), but the find the HD600 to both sound better and feel more comfortable (the SRH1440 is noticeably heavier and the pads are relatively thick, so it feels a bit like my head is in part of a football helmet).

• Jabra Elite 85t earbuds for travel & on-the-go calling. I talked about these more under microphones, but they’re a great (although not cheap) set of day-to-day true-wireless earphones. They are unsealed, which is not common for this level of earphone, so I’m a lot more comfortable calling on them than on sealed earphones. They also have both sidetone and hear-through, making them quite versatile for a range of listening and calling situations. Most of the Jabra models (at least at the higher-end) have sidetone, so if you want sealed earphones, they have several options.

I no longer have the Jabra Evolve 40 (they were owned by the university), but as noted a couple times above, they’re a pretty good and semi-portable headset. I wouldn’t want to spend a lot of time listening to music on them, and they aren’t outstanding for listening while doing video editing and post-production, but they’re fine and worth the money.

One last headphone to note is the Sony MDR-7506. These are a professional-grade closed-back supra-aural headphone that Sony has had in production for over 30 years now. For my ears, they aren’t the most comfortable for extended periods of listening, but they sound great and are built to last. They also fold up into a relatively bulky bag for some portability (could toss them in a suitcase, but they’d take up quite a bit of a backpack). They’re a workhorse in the professional live sound world, and earn that reputation. If I started to do live sound work again for some reason, I would definitely buy a pair. In the office, I prefer the Sennheiser.

I don’t have particular recommendations for other types or formats of headphones. Sennheiser generally makes good headphones, so it’s hard to go wrong looking through their product lineup for something that meets your needs; Jabra is generally good as well, but is definitely more optimized for office use than music or video production.

Additional Bits

That covers most of it. There are a few other things that can be useful:

• For extensive video editing, a shuttle controller is extremely useful. This lets you easily “scrub” backwards and forwards in the video to find the precise points to edit, and most of them provide additional buttons that you can map to keyboard shortcuts to perform common operations like splitting a video element or deleting selected video. Microsoft’s Surface Dial is also supposed to function as a shuttle, but I don’t have any experience with it.

• Quick access to a mute button is helpful on calls. Zoom provides some kind of global keyboard shortcut to mute/unmute on most platforms. My keyboard is configurable, so I have set the volume knob to send that key signal when I press it in. In the office, I also have the hardware mute button on the Vocaster.

• Video hosting can be a little trick. Both of my recent universities have contracted with a hosting platform (Kaltura at Drexel, Panopto at Boise State); that’s the best option for class videos, since it integrates with the LMS and also complies with the university’s data storage and access policies. For other videos such as talks or promotional videos, I often publish to YouTube for visibility and discoverability. When embedding video content in my website, however, I don’t like that option: it embeds Google code (and sends traffic to that page to Google), and I also don’t have any control over the ads or the post-video recommendations. This last point is especially important for educational videos, because I wouldn’t want the recommender sending viewers or students off to some other source I haven’t vetted and which I may be specifically avoiding. Since my site is relatively low-traffic, I use Bunny Stream to host embedded videos here. They don’t cost a lot, re-encode to a range of quality levels, and their player is pretty decent.

• If you really want high-quality background replacement, then a good lighting setup and a green screen will produce better results than using the “virtual background” feature of Camtasia or the videoconference tool. However, for most academic purposes, it isn’t worth the extra cost, space, and hassle in my opinion.

Wrapup

That’s… probably more than you were hoping to read, but I hope it is helpful. Go out and make great videos!

Student Achievements

• My first Ph.D. student Amifa Raj successfully defended in June, and is now an applied scientist at Microsoft.
• My second Ph.D. student Ngozi Ihemelandu successfully defended in December, and is now on the job market.
• Srabanti Guha successfully completed her M.S. project and defense in May. Since Drexel info sci’s M.S. programs are not research-based, she is probably my last M.S. advisee for the foreseeable future.
• Undergrad RA Christine Pinney first-authored and presented a full paper at CHIIR 2023 on gender in IR research, in collaboration with Amifa, Alex Hanna, and myself.
• Amifa’s internship project (on which I collaborated) published as a short paper in SIGIR 2023.
• Ngozi’s work on statistical inference for recommender system experiments published as a short paper in SIGIR 2023.
• Ngozi’s work on candidate set sampling in recsys evaluation published at WI-IAT 2023.
• Amifa’s work on measuring fairness in grid layouts presented at FAccTRec 2023.
• Ngozi’s paper on multiple comparison correction in search & recsys experiments was accepted as a short paper for ECIR 2024.
• Amifa’s paper on grid-aware reranking for fairness was accepted to the IR for Good track at ECIR 2024.
• Preprint with Amifa on integrated browsing models for IR metrics.

Other Research Achievements

• Got a new grant (w/ Joe Konstan, Bart Knijnenburg, Robin Burke, and Ed Malthouse) to build a personalized news recommendation platform to support user-facing research on recommender systems.
• Published a short piece on multi-party information seeking with Sole & Katherine.
• Published an article in Transactions on Recommender Systems with Ben Carterette and Fernando Diaz on distributional evaluation of recommender systems.
• Collaborated on a demo (led by Tobias Vente and Joeran Beel) of an auto-tuning framework for LensKit.
• The many-authored paper on values in recommender systems (led by Jonathan Stray) that I contributed to is out in Transactions on Recommender Systems.
• Contributed to a position piece led by Alexandra Olteanu on the need for impact statements in responsible AI research.
• Our paper (with Sole Pera, Henriette Cramer, and Lex Beattie) on consumer impacts of IR systems was accepted to the IR for Good track at ECIR 2024.
• Submitted several more papers that unfortunately were not published.
• Started a couple new collaborations that will hopefully produce interesting results in the next year or three.
• Gave a seminar talk for University of Washington RAISE.
• Gave a seminar talk for UT Austin.
• Gave a seminar talk for University of Glasgow.
• Spoke (virtually) at an ICA post-conference.

This year I made a total of 10 paper submissions and posted 2 preprints (not counting preprints of accepted material); we had 4 rejects to get our 6 conference publications (some of which were resubmissions of work rejected last year)².

Teaching

• Taught Recommender Systems in spring term, as my last class at Boise State.
• Taught INFO 659 (Intro to Data Analytics) for my first class at Drexel.
• Begun preparations for teaching Drexel’s recommender systems class, DSCI 641.

Service & Organization

• Program committees: FAccT (AC), SIGIR, RecSys (SPC), ECIR.
• Co-organized the 6th FAccTRec workshop.
• Numerous journal reviews.
• 2 international research proposal reviews.
• Final year of FAccT executive committee service.
• TheWebConf and SIGIR best paper committees.
• Joined the editorial board for Foundations and Trends in Information Retrieval.
• Drexel IS Ph.D. committee (ongoing).
• Drexel IS faculty search committee (ongoing).
• Traveled to CHIIR (Austin, TX), SIGIR (Taipei), FAccT (Chicago), and RecSys (Singapore).

Wrapup

I’m pretty sure I’ve probably forgotten a thing or two here. It’s helpful to me to sit and reflect on what I’ve gotten done in the course of a year. I’m especially glad to have such a long section of student accomplishments to list. And hopefully some reporting on rejection levels is helpful to others³.

And thank you very much to my excellent collaborators who have made all of this possible.

For the next year, there are of course many new things in planning (or progress), and I look forward to starting with some new Ph.D. students at Drexel.

Endpoints

I’m now primarily on Apple (MacBook Air M2 at home, 14” MBP w/ M2 Pro for work). This is working well, with much better performance and battery life than when I was on Windows (even on an i7 Surface Laptop).

I also maintain a small Windows remote desktop machine at work for software and class instruction testing, and got an iPad Air for reading and paper markup. Plus our old Windows desktop I mostly use for gaming.

I now use a 27” 4K ASUS ProArt display. It looks great, and also works as a docking station if you don’t need a lot of ports or an ethernet connection. So far, the single 27” monitor with the laptop open on a stand is working; don’t have a pressing need to get a second full-size monitor.

I’m still using a Kensington Expert Mouse, now paired with a Keychron K15 Pro low-profile Alice keyboard. I’m liking the Keychron a lot; it doesn’t have quite the slope of the Microsoft ergonomics I’ve been using for the last several years, but it doesn’t seem to be triggering RSI recurrence. For mobile, nothing has changed; still on iPhone and Apple Watch, and don’t anticipate switching.

At work, I also use a CalDigit TS4 docking station to have some extra ports and Ethernet, and have a 4TB Crucial SSD for expanded storage on my MacBook.

Video Recording & Conferencing

I didn’t feel like I was getting a lot of value out of the Logitech Brio’s extra resolution, so when setting up at Drexel I got a C920s instead. It is good enough, but not great; I think the Anker PowerConf C200 I have at home is better.

For audio, I’m very pleased with my new setup: a Shure MV7X connected to a Focusrite Vocaster USB interface, with Sennheiser HD 600 for listening. Call and recording audio quality is outstanding, and the headphones sound very good both for calls and music.

I’m still using Camtasia for recording and editing lecture videos. I have added OBS to my toolkit, along with the iPhone app, for capture from my phone camera when I need another camera (e.g. for a document camera). Another useful addition has been MacWhisper for generating video captions. Its speech recognition accuracy and segmenting logic are both very good, and it is significantly less expensive than Audiate (although without the text-based video editing capabilities).

Interactive Software

My general interactive computing has seen some changes, due to moving to macOS and a new university that doesn’t use GSuite. I have a subscription to SetApp, which yields a number of useful utilities.

Command-Line Software

A lot of my computing is in the terminal. I tried fish for a while this year, but am now back on zsh. I’ve been trying oh-my-zsh, but I’m not sure about some of its configuration choices yet. Several augmentation utilities fill out my shell profile:

• starship for my prompt.
• direnv for managing per-project environment variables (including actiating a project’s Conda environment).
• zoxide to navigate directories.

Beyond that, a very non-exhaustive list of tools I find useful:

• nano for terminal text editing.
• fd to search directory trees.
• eza (successor to exa) for file listing. My terminal config aliases l to eza and ll to eza -l.
• bat to view text & code files with syntax highlighting.
• ripgrep to search file content.
• dust and ncdu for analyzing disk usage.
• jless to view JSON files.
• tmux for terminal multiplexing.
• watchexec to auto-rerun things on file changes (e.g. continuously rebuilding my website while working on it).
• age for file encryption.
• minisign when I need cryptographic signatures.
• tokei to count source code.

Data Management

I’ve made some significant changes in my data management across the board this last year, so that seems to merit a section of its own.

I’m now storing university data on OneDrive, as that’s what Drexel provides. I also use OneDrive for most of my data syncing between my MacBook and iPad, although I share some files with iCloud. Each account has 5TB available, so it works pretty well. At home, I mostly use iCloud, and use an iCloud shared folder to share files between work and personal computers (I use separate Apple accounts for my home and work machines). I also store all photos in iCloud, and use shared albums to send files between my phone and university MacBook.

I use git for all of my source code storage and version management. To have greater control over when and where data files are synchronized and stored, I’ve started using git-annex for data archival (as well as the assets for my website). I still use syncthing for a few folders, but not as much as I used to.

We’ve switched our home endpoint backup from Backblaze to Kopia, to have them more under our control and to manage costs as Backblaze pricing has increased. It’s configured to store backups on our NAS, which synchronizes them to cloud storage regularly.

I’m still using dvc for experiment data. Recent versions have introduced some annoying regressions, but I don’t have a better solution yet.

Programming and Data Analysis

Visual Studio Code is still my primary programming editor across all languages I use.

I’m still using Python and Rust as my primary research languages. I’ve started dabbling in R again, as we use it in the class I taught this fall; I’m not sure long-term if what the R/Python balance will be for statistical analysis and visualization. I’ve also started using Quarto in place of Jupyter in some projects, and for rendering my data tools documentation.

The Polars library has been a useful addition for data processing, both in Python and Rust; it’s replaced Pandas for larger-scale data processing.

I’ve started using more TypeScript in my JavaScript programming, and that’s been quite useful; my website is entirely TypeScript now. TypeScript’s type system is one of the more expressive I’ve used, which has been fun, but it also has several escape hatches that turn the guarantees one would usually expect from a typechecker into more of set of best-effort guidelines. It still gives me higher confidence in code correctness. I expect a lot of my JavaScript work will be in TypeScript going forward, mostly with Deno when not in the browser.

Finally (on the general-purpose front), I’ve also been writing more Tcl in various places. I find it severely underappreciated and quite like it as a glue and scripting language, and the Jim interpreter makes it highly portable.

As noted above, I’m still using DVC for research data management and analysis pipeline automation. To work around the limitations of DVC’s pipeline language, I’ve moved away from the templates and weird TCL scripts I’ve experimented with in the past and now use jsonnet to generate non-trivial pipelines (see the Book Data Tools for an example).

One significant change in my programming across languages is that I have started using autoformatters when they are available. For Rust, this is the standard rust fmt; for Python, I’m using Ruff (it’s also replaced flake8 as my primary linter); and for TypeScript & JavaScript, I’m using deno fmt. I’m also using either Prettier or formatters built in to VS Code extensions for other languages like yaml and jsonnet.

I’m still using Conda — usually with conda-forge — for installing research software and Python development dependencies. It remains the best solution I’ve found for reproducible, cross-platform software environment installation.

Now that I’m back on Mac, I’m using Dash again for reading documentation, although I still do quite a bit of that in the browser.

Other Software & Services

In no particular order:

• Banktivity for personal finance tracking.
• Unsplash and The Noun Project to source images.
• A number of other things that don’t come to mind right now.
• Typeface for selecting fonts.

University Compute

I’m still getting my university compute needs and resources figured out, but I’ve started with one server node in our data center, with dual 64-core Epyc 7662, 256GB RAM, and an Nvidia A40 GPU.

Home Network

I completely overhauled our home network this year to get better Wi-Fi, upgraded network storage, and a firewall that hasn’t taken up conscientious objection to software updates. I’ve also standardized on Debian Stable across our servers.

It was time to replace our old QNAP NAS; its Atom CPU was old enough that it didn’t have hardware-accelerated crypto, and was having increasing trouble keeping up with the amount of data I wanted to run through it. I replaced it this year with a RockPro64 in Pine’s NAS case, with 2 8TB Toshiba hard drives. This machine runs Debian Stable, with syncthing, Samba, and git-annex for its primary data transfer roles. It also serves as our primary backup host with Kopia Backup.

I also bought an R5C to serve as our firewall & router. This little box has an RK3568 SOC and 2 2.5Gbps Ethernet ports on the PCIe bus. It’s also running Debian stable, and thanks to inindev’s images is able to run a stock Debian install instead of Friendly’s images with difficult-to-update kernels. A 5-port switch and a TP-Link TL-WA3001 Wifi6 router complete our home network.

I’m using Tailscale for VPN connections between home, cloud, and my devices when on the move. It works great, and is much easier than managing Wireguard myself (or messing with our old router’s OpenVPN support). I also use a Tornado VPS server to host my website (behind Bunny CDN) and handle other coordination between home and remote devices.

Shape of the Problem

LensKit supports parallelism in two different places: model training and batch evaluation. Model training uses the threading support of the underlying compute engine (BLAS and Numba, although I’m looking to get away from Numba). Batch evaluation uses the Python standard library’s ProcessPoolExecutor through a custom wrapper layer (more on this later).

In batch evaluation, we have a problem that is common to many machine learning batch “inference”¹ problems: we have a large model object that is read-only (except for ephemeral caches), and multiple threads or processes need to compute results using this model object (in our case, computing recommendations for the thousands or millions of test users). Since this model is large, it presents two constraints:

• We only want one copy in memory, no matter how many workers we use (if we eventually support multi-node evaluation, we would want one copy per node).
• We don’t want to have to re-serialize it for each task (user).

Usual API design

Typical Python parallel processing APIs are designed around submitting tasks with arguments to the executor pool. At a surface level, our use case maps reasonably well to the typical map function provided by e.g. Executor:

def map(func, *iterables):
    # apply func to items in zip(iterables), in parallel

Most Python parallel APIs provide a function like this, and differ primarily in their implementation details and optimizations. However, there is one element of this API that does not match our use case: the function needs access to the large shared model object in addition to the per-task arguments it gets from iterables.

Current Solution

The ProcessPoolExecutor works on top of multiprocessing.Pool, which allows us to provide a worker setup function that will be called to instantiate each worker thread, as well as provide arguments to the setup function. This conceptually maps quite well: if we have a way to put a model in shared memory, we can use worker setup functions to store it in a global variable, and then supply a task function to map that fetches the shared model and passes it, along with the per-task arguments, to the batch evaluation function.

In Python 3.8, the pickle module gained support for the new Protocol 5 pickling format, which allows application code to provide a custom callback for serializing anything that implements the Python buffer API, which includes the NumPy arrays that make up the bulk of our model’s saved data. If we “serialize” those by copying them to shared memory, then the remaining object structure is a relatively small pickle. The binpickle library leverages this to provide an on-disk serialization format that supports memory-mapping buffer contents.

I created LensKit’s Invoker framework to abstract all of this this. Its ModelOpInvoker implementations take a model object, and do the following:

• Serialize the model to shared memory (if it isn’t already serialized). It supports both Python’s SharedMemory and binpickle, which was created for the purpose and can be used to share through memory-mapped disk files if /dev/shm has insufficient space for the shared memory.
• Deserialize the model in the worker processes, using shared memory for the buffers.
• Wire up logging so that log messages are properly passed back to the lead process.
• Provide the model object to a “model op”, a function that takes the model and the task (e.g. user ID) and returns the results.

It works pretty well, all things considered. There are a few outstanding problems, though:

• I’m maintaining a parallel processing library.
• Naïve use still has 2 copies of the model in memory, because we cannot move the model’s buffers, we can only copy them. I have workarounds, but they’re not pretty, and require binpickle (basically, serializing to a memory-mappable binpickle doesn’t require the output to be in memory, so you serialize to disk, drop the original model object, and then have the invoker use the pre-serialized model, which its’ capable of doing).
• Excessive worker process logging can overwhelm the logging pipe.

So I spent some time this week trying to see if someone else has built a good solution I can use instead of continuing to maintain the invoker library.

Current Status

So far as I can tell, there really hasn’t been a lot of movement towards a solution for this specific problem shape. I also may not be looking in the right places, but I see surprisingly little uptake of Pickle 5’s ability to facilitate shared memory.

The things I’ve looked at in my current exploration include:

• ProcessPoolExecutor, didn’t expect anything new here, and it’s what we’re already using.
• joblib, which I used in an earlier version of LensKit. Joblib focuses on providing a very clean and robust parallel implementation of map, and admits that this comes at the expense of other functionality; there is no way to provide per-worker model data (at least not without configuring your own executor). It uses loky and cloudpickle to work around old bugs in Python’s concurrency library, improve performance, and increase the number of things that can be pickled; the relevant Python concurrency bug has been fixed in the versions of Python I support, and I haven’t found pickle’s limitations to be a problem in our code paths. Joblib also provides dump/load routines that allow memory-mapped Numpy arrays, but it’s a hacky solution that predates Pickle 5. binpickle was born from “what if we did Joblib’s dump/load but on top of Pickle 5?”. For my purposes in LensKit, joblib doesn’t provide much value on top of Python’s standard library, so it isn’t worth the dependency.
• mpire supports large shared models, but it does so through fork, so that feature doesn’t work on Windows (or with any other multiprocessing spawn model). Good Windows support is a priority for LensKit.
• ipyparallel claims support for “zero-copy” buffers, but the devil is in the details. This zero-copy support only works when you actually pass the NumPy array to the map or push function, not when it is an implementation data structure of an object you pass (with a few exceptions). The serialization layer is extensible, so I could use Pickle5 to serialize anything wtih zero-copy buffers, but it gets to the next problem: the zero copies are in the process of preparing the data to send, but as near as I can tell, it is still copied over the zeromq pipe. Therefore it isn’t zero-copy in the sense that I need, but rather 1 (instead of 2) copies in the driver process and one copy in each of the worker processes. Supporting true shared model objects still requires the same serialization logic I use to work with the standard library.
• dask.distributed and ray are both interesting, but very much seem to want you to work within their framework. LensKit is designed to fit with whatever tools the user wants to use, particularly fro model implementation, so forcing evaluation into a heavier framework like them would be a paradigm change. It also appears like I would still need the custom shared memory serialization logic, although that may not be entirely correct. I also just haven’t had good success getting actual speedups from Dask when I have tried it in the past.
• torch.multiprocessing does go there, for PyTorch tensors — when you send an object to a worker, it moves the tensor into shared memory and allows the workers to access it. Only works for PyTorch, not numpy or other buffer-based APIs.

That’s what I’ve been able to learn so far. Multiple operations with single large shared model object just does not seem to be a well-supported pattern outside of PyTorch and heavyweight cloud- and cluster-oriented frameworks. With LensKit’s goals of good single-node performance and minimal constraints on users’ environments, there doesn’t seem to be a better solution than what I’m doing now.

Sidebar: Subinterpreters

Python recently added “subinterpreters”, which are multiple isolated Python interpreters in a single process, and Python 3.12 allows each subinterpreter to have its own global interpreter lock. This seems to open the door for a very good solution: parallelize with threads, giving each thread its own subinterpreter; share the buffer memory between subinterpreters and keep using pickle for the object structures and other inter-thread communication. However, there is significant work to upgrade extension modules to support subinterpreters, and it doesn’t look like a priority for NumPy yet.

Moving Forward

It looks like I need to keep maintaining a parallel compute API for large shared model objects in order to enable LensKit’s parallel evaluation. I’m looking to make a few changes to make it more usable and ergonomic, and encourage adoption (and co-maintenance?) by others:

• Create a new library for the invoker code so it can be maintained separately and used by non-LensKit projects. It will continue to support both Python shared memory and binpickle mapping, depending on the use case. I may also consider adding support for plasma, but I’m not sure that would gain us much on top of Python’s shared memory API.
• Add support to Binpickle to deserialize directly to shared memory, so users don’t need to have separate compressed BinPickle files for model storage and mappable ones for parallelism, unless they specifically need the increased memory capacity afforded by BinPickle mapping.
• Add progress bar support.
• Add optional “destructive pickling” that tries to dismantle NumPy arrays (by resizing them to 0) once their bytes have been transferred, to enable “move to shared memory” semantics. This is possible because the buffer callback API in Pickle 5 makes the buffer’s owner available. Destructive pickling will leave objects in unusable states and must be used with care, but it seems worth trying for the common cases of models that will not be used after they have been serialized.
• Eventually: consider adding support for ipyparallel, with the same custom deserialization logic currently used for process pool executors. When used with memory-mappable binpickle files stored on NFS, this will allow multi-node parallelism with one model copy per node.

I’ll update this post with links once the new library is up and available. Hopefully people will find it useful!

Teaching

• Spring — CS 230 (Ethics)
• Fall — CS 533 (Intro to Data Science)

Active Grants

• NSF CAREER

Publications

Hardware & OS

Organizing these posts can be a bit random, so let’s start with systems themselves.

Interactive Software

Across my various devices, I tend to rely on the same software for my interactive (graphical) computing:

• Chrome for work browsing, Firefox for personal.
• Office for editing, spreadsheets, and presentations; I do use Google Docs for work stuff that is lightweight and/or needs high collaboration.
• VS Code for text editing.
• Overleaf for LaTeX document preparation.
• Paperpile for reference management.
• DynaList for tracking work notes and my runway document.
• FreeCommander for Windows file management, Marta on macOS (when I am not just using the system’s native file browser).
• The Affinity suite for graphics & photo editing.
• Visio for diagramming (when PowerPoint or diagrams.net won’t do). I’m not using Grapholite much any more. After switching to Mac I will look at OmniGraffle.
• Drawboard PDF for tablet-based PDF reading and annotation; Acrobat Pro on laptop/desktop.
• 1Password for password management.
• iTerm 2 for my macOS terminal (Microsoft Terminal on Windows).

Command Line

I do a fair amount of work from the command line, both for running the software and analyses I’m doing and also for general file and data manipulations. Key pieces of this load:

• zsh on *nix (including WSL), PowerShell on Windows.
• nano for quick terminal editing; I’ve also been experimenting with Helix and Micro but the nano keybindings are pretty engrained now.
• tmux for terminal multiplexing / disconnection.
• mosh for connecting to servers at home, will probably start using it more across the board. I’m now just using OpenSSH’s direct PKCS#11 support for my YubiKey.
• direnv to provide ergonomic project-specific configuration.
• zoxide (on *nix) and ZLocation (PowerShell) for directory navigation.
• htop for performance monitoring.
• bat for file viewing.
• fd for finding files.
• ripgrep for searching files.
• ndcu and dust for managing disk usage.
• exa for listing files.
• pandoc for Markdown processing.

Data Infrastructure

On the personal side, we moved off of Microsoft for our primary data infrastructure. E-mail is now on Fastmail and we’re using iCloud for file sharing, calendars, contacts, tasks, etc. Microsoft was working fine for us, but with moving towards macOS it wasn’t making as much sense, and we were able to cut our infrastructure & services bill a bit. Using Syncthing for data transfer between endpoints and the NAS, Samba/CIFS for direct access to NAS storage, and Backblaze for endpoint backups.

For work, there are a few things in play:

• Google Drive for cloud/synchronized document storage (and all the Google Doc storage, of course).
• git and GitHub for source code and text document management. Most of my documents either live in Google Drive or a Git repo.
• Data Version Control (dvc) for managing research data and pipelines.
• Minio (S3-compatible storage server) for storing and synchronizing DVC-managed data.
• rsync and robocopy for CLI data transfer; WinSCP for GUI transfer.
• Kopia for endpoint backup, using a university CIFS drive as the backup target.

Programming and Science

I am still using Python + Pandas + NumPy as my primary software development environment for scientific computing & open-source work. I wouldn’t say I love it, but it’s widely-known, gets the job done, and has libraries for a lot of things. I’ve been working on filling a few gaps myself, such as the seedbank package for random number generator initialization. For plotting I’m mostly using plotnine, but sometimes seaborn and matplotlib.

I use Rust for high-performance data processing code; this year, I finished rewriting the Book Data Tools to be completely in Rust (except for notebooks computing statistics on the data set) instead of a combination of Rust, Python, and SQL.

JavaScript is my primary language for web stuff; this web site is generated with a custom JavaScript codebase. I’ve been experimenting with Deno some as an alternative to node.js, and might start doing more with it. I also reach for JavaScript when doing some data processing that is heavily web-native, like extracting data from a Twitter archive dump.

I’ve also started playing around a bit more with TCL for lightweight scripting. I’ve used it for a while for helping with some setup bits for my Unix shell environment, and have now written some admin scripts in it. It definitely has some warts and missing holes, but it’s a rather pleasant language to do simple things in, and it’s fantastic for building eDSLs.

I use Conda/Mamba (typically through the Mambaforge distribution these days) for managing my primary scientific software environments, as it allows me to get Python, Rust, R, and other tools as needed. These days, if something isn’t in conda-forge or available on a pretty basic OS install, I don’t depend on it in my research projects, so that students and collaborators can get all the requirements from environment.yml.

For projects that aren’t in Conda, I use rustup for managing my Rust versions and asdf for managing most other interpreter environments (Python, Node, Ruby, etc.). I’ve been using pip-tools to pre-resolve Python dependencies for lack of a better solution.

Media Production

I’m not recording as many videos as I did the year I did the build-out of CS 533, but I do still record and edit a fair number.

Camtasia is my go-to recording & video production software; it’s much easier to use for my use case (educational videos) than something like Preimier. I also have a USB shuttle to help with precise navigation while editing.

The game-changing addition this year was TechSmith Audiate. It’s an audio editor and transcriber that allows you to edit an audio track by editing its text transcription. Corrections just correct the transcript without editing the underlying audio; deletions edit the audio track as well. This way I can more aggressively trim my videos to delete blank spaces, false starts, etc.; previously, I only edited out big gaps, because more detailed editing to clean up the speech is very labor-intensive. It integrates with Camtasia, so I can export the audio track from Camtasia to Audiate, clean it up, and round-trip back with an edit list that Camtasia uses to trim the corresponding video. This cleanup process also gives me working captions at the same time. I haven’t gone back and re-edited all my videos with this, but it makes it much less time-taking to produce both a cleaner video and edited captions for new work.

For recording hardware, I’m using a Logitech Brio as my primary webcam, and a Blue Yeticaster as the microphone. I have a green screen that can clip to the back of my chair and some USB lights to facilitate chroma-key work. I also use my phone sometimes, with the Camo software that turns a phone into a webcam. This, combined with a desk-mounted boom arm and a QuadLock mount, allows me to do document camera work in my office either on video or on Zoom without needing to record and edit in a separate video stream. I also use my phone for mobile video recording.

I’m still using Unsplash and The Noun Project for sourcing most of my artwork & icons, with the occasional addition from OpenClipArt.

Analog Information

I’m still using Leuchtturm1917 A5 notebooks and a collection of fountain pens for my daily and weekly productivity management, note-taking, and journaling. I use a Pentel Orenz mechanical pencil for working out maths, sketching diagrams, and other things where I need to be able to erase.

I also bought a turntable this year and began collecting vinyl music. Tactile, analog engagement helps me ground myself and feel more connected to what I’m doing; it’s been effective in my journaling, and I wanted to bring that to a more purposeful connection to music (in some contexts — I still make heavy use of streaming as well). I’ve been putting together an eclectic collection including P!nk, Sturgill Simpson, The New York Rock & Roll Orchestra, Chicago Transit Authority (before they changed their name — found that album for $1 in the bargain bin), Carly Rae Jepsen, and many others. My weekly ritual has now extended to include spinning an album to go with my whiskey and longhand planning.

Why Keep Deadlines?

I generally cannot do away with deadlines in my classes. Deadlines allow me (or the grader) to grade in a predictable timeframe, with all the assignments in; it’s easier to maintain consistency when grading all submissions in a single effort. They also provide a cutoff for working on the assignment so I can share my solution with the class. For some classes with assignments that produce more varied student responses, this wouldn’t matter near as much, but most of my classes are programming or data analysis classes where everyone is working on the same problem for most of the assignments.

I did try a deadline-free approach one year for my graduate recommender systems course — all material was due at the end of the term — but the effect was that almost all students didn’t start working on the assignments until the last week or two of class, and had significant difficulty completing them in time. Students benefit from deadlines to structure their own work as well.

For all of these reasons, it wouldn’t work to do away with deadlines entirely in my teaching. I also don’t think it would work to just have deadlines with unlimited free extensions — it might guide some students to submission, but makes the deadlines relatively meaningless.

I do not enforce deadlines for the purpose of trying to simulate the “real world” or something like that. The deadlines are for the purposes of maintaining smooth operation of the course.

Principle: Empowering Students

My guiding principle for assignment logistics is that students know better than I what is going on in their life and how it affects their work. There are many different reasons a student may need an extension on an assignment deadline, including but definitely not limited to:

• illness
• accident or emergency
• last-minute child care needs
• car got towed while they’re eating supper before finishing the assignment
• difficulty understanding material, need another trip to office hours

Further, I don’t really want to be in the position of adjudicating what counts as a legitimate basis for an extension and what does not, or obtaining backup justification, or anything like that. It isn’t fun, it’s an unnecessary intrusion on students’ lives, it doesn’t advance learning in any way, and it requires relatively arbitrary decision-making.

Different students have different needs. So I want policies that will meet a wide range of student needs, and empower students to do what they need to in order to succeed in both the class and their lives.

Resources for Success

This led me, when I was preparing my first syllabus in Preparing Future Faculty, to frame the question differently: instead of rules and exceptions, can I design the class to provide my students resources that they can deploy as their particular situation requires?

The fundamental concept I landed on here was to provide late days as a resource. In their current form, they work as follows:

For the assignments, you have a budget of 4 late days to use throughout the semester, at your discretion.

• Each late day extends an assignment deadline by 24 hours with no penalty.
• Late days are indivisible, so submitting an assignment 12 hours late uses an entire late day.
• You may use up to 3 late days on a single assignment.

When submitting an assignment using a late day, state with your submission the number of days you are using. I appreciate it if you notify me (via a Piazza private message) prior to the deadline that you are planning to submit late, but do not require you to do so.

I adjust the total number of late days based on the number of assignments in the course and the level of the students; this version is from a graduate course with 7 assignments. The number is relatively arbitrary, but I’ve found 4 to generally work pretty well for such a class. The limit on late days per assignment is to put a shorter end-point on when all students have turned in an assignment. With deadlines on Sunday night, a 3-day limit means I can teach through my solution in class on Thursday.

The purpose of this design is to enable students to make the decisions they need to in order to succeed, within a framework that encourages them to think about those needs and select appropriate tradeoffs. They could use most of their late days on an early assignment just because they want more time, but that creates the risk of running out of days later.

I round out this policy by also dropping their lowest assignment score (or sometimes the lowest two scores, depending on the course). If they’re out of late days, or something comes up that would require more than the allowable limit, they can just blow off any one assignment without penalty. Sometimes, when students talk with me about a late assignment that they’re having difficulty finishing, I’ll encourage them to just not submit it and save the late days for later. I have also, on occasion, zeroed out a previously-submitted assignment and refunded the late days for use on a later one.

Remaining Extensions

Late days and the dropped assignment cover the vast majority of needs for extensions, so I very rarely grant extensions. About once every 2-3 terms a student will have needs beyond those accounted for in the base course design, and I’ll work with them on a solution (additional extensions, etc.), but for the standard extension requests I can point students to the policy and they can take the extra time without needing to justify themselves.

Alternatives

I have experimented with escalating penalties for late work — 5% the first day, then 10%, then 25% — but that has felt more punitive. Resources that can be used without grade penalty is, in my opinion, a more success-oriented approach to managing course deadlines.

Assignment Timing

There’s also a lot of discourse sometimes about when assignments should be due. I typically make my assignments due at midnight on Sunday.

I use midnight as the deadline to allow students to work into the evening while discouraging them from pulling all-nighters. When I’ve had assignments due at the beginning of the next day (8 or 9 AM), I see quite a few submissions at 4 or 5 AM. I’m lax on the precise time, though, interpreting midnight as “before you go to bed”; if an assignment is an hour or two late, I don’t count it as late.

The day is a little trickier. I’ve sometimes used other days, such as Tuesday. If a course has class on Tuesday afternoon, I’ll sometimes make the deadlines Tuesday at noon.

Some argue against Sunday deadlines, on the idea that it encourages students to work on the weekends. I don’t find this argument persuasive, as students have different needs and different lives. Many of our students work during the week, especially upper-division and masters students, and the weekend is the time they have set to work on their course assignments. If I make the assignment on Friday, then students who have plenty of time during the week have almost an entire week longer to work on the assignments than the students who have to balance their studies with a full-time job and family duties. In keeping with my goal of empowering students to make the decisions they need for their own lives, I put the deadlines either on Sunday night or early in the week, with the expectation that if weekend work isn’t good for a student, they can choose finish it on Friday, and the difference in effective time for the assignment isn’t as large as it is if I used a Friday deadline.

Student Response

Students have been overwhelmingly positive about this policy. There’s sometimes a little confusion early on, or the first time a student actually needs an extension (since they usually haven’t had the policy in their other classes), but once I’ve explained the policy I have never had a student complain about it. I have had a student say that all CS courses should use my late day policy 😊.

If you find this policy useful, feel free to adopt it in your courses. If you want to attribute it to me, a link to this blog post would be fine. So far as I know, this particular policy was original or at least independently developed; I don’t remember referencing any directly similar policies while designing it, except perhaps some general flexible extension policies, and was primarily informed by universal design and what I was learning in class about how to accommodate a wide range of student needs.

Academic Review

I’d like to start with a quick overview for new or prospective academics. There’s the well-known Big Evaluation, when you apply for tenure & promotion; and a few years after that the application to Full Professor. Many institutions have a pre-tenure reappointment review at around year 3. You will also likely have one or two annual reviews; for pre-tenure academics, these serve as a check-in on your tenure progress, and for all academics they’re used for performance evaluation and determining merit raises (in institutions that use the evaluation for raises).

When I was at Texas State University, tenure-track faculty had to provide data for two reviews each year, because the progress / reappointment review and performance review were separate processes with separate deadlines. It was weird.

But for each of these reviews, you need to provide a complete log of your activities, in the format required by your institution (often in addition to keeping it current on your CV). You also need supporting documentation. What precise documentation is required will vary from institution to institution, but you can’t go wrong by saving everything, even if you don’t need to submit it all to the institution.

Current CV

I keep my CV continuously updated as a part of my web site. Some of the technical information is here; the web pages for my publications serve as single-source-of-truth for publication information, and my CV itself is a Markdown file that pulls in the publication information, along with other content, for rendering to HTML. WeasyPrint converts that to a PDF that I can save, share, upload, etc.

Some institutions require you to highlight new things. Texas State did that; I dealt with that by including the year in an HTML class or data attribute on the relevant HTML <li> elements, and a bit of CSS to recolor everything for a particular year. These days I would probably just mark up the PDF in Acrobat.

Keeping my CV current as I get new publications means my annual CV update is mostly adding new teaching & updating my service. Huge time-saver at review time; time sink overall, but it’s my pet time sink and I’m rather proud of it.

Logging Work

Different types of work have different logging needs. Paper acceptances sort of log themselves, but you need to keep track of service work, grant applications, teaching, etc. I have different logs for different bits:

• All reviewing work is logged in a DynaList notebook, by type; this is also where I keep track of deadlines and my overall reviewing workload.
• I have a spreadsheet for grant proposals that lets me look at my overall funding rate, etc.; I included a copy of this in my tenure application.
• Teaching I just update in my CV every year; our faculty effort reporting system automatically gets it from the registrar’s office.

Many universities have online systems for tracking work; Boise State uses Faculty 180. You will need to enter your work into this system for review, but I find it completely useless for my own use. I recommend keeping your own records, and then updating the effort tracker at review time. This lets you keep them in a format that’s useful for you; also, you can keep copies of them if you ever need them for a dispute, you get fired, or you’re looking for a different position. Keeping your own copies of vital professional records (so long as no FERPA or otherwise-protected information is implicated) on your own private drive is good practice.

Documenting Work

I maintain a Portfolio folder that I use for documenting all my work: copies of my annual review reflections, documentation for my accomplishments, copies of my CV each year, tenure application materials, and whatever else is relevant to documenting my professional career. I’ve gone back and filled it in all the way back to the beginning of my Ph.D.

This folder is organized by year (calendar year, since we do annual reviews on calendar year cycles at Boise State). Within each year, I have several things:

• That year’s reflections from my annual eval (Word documents)
• End-of-year CV
• A Papers directory with PDFs from all papers published (or in some cases accepted) that year
• A Proposals directory with every grant proposal submitted
• A Teaching directory with saved teaching resources
  • Syllabi
  • Student evals
  • Representative documents (assignments, final exam, project description, etc.)
• A Service directory with documentation & evidence for major service assignments (e.g. thank-you letters for program committees, etc.). If the only evidence available has confidential information (such as the titles of papers or proposals reviewed), I print it to a PDF and use Acrobat’s Redact tool to remove the private info.
• A Talks directory with documentation for invited talks.
• Other files to document random things (like my ACM Senior Member designation).

After receiving a year’s evaluations (from chair, tenure committee, dean, etc.), I save PDF copies into the year’s directory as well.

I try to drop things into this folder as they happen, but usually need to spend an hour or two each year reviewing it and hunting down any missing files. Making sure it’s up to date as a part of my annual eval, though, means it’s already there and I don’t need to check for missing things when preparing for one of the big evaluations.

In addition to the portfolio documents, I also have each class in a separate working folder, organized by year and term, and keep those indefinitely. All documents I create for the class, except things like source code and web content that live in a GitHub repository, are there, and I keep them indefinitely. If I need another document from a particular class, it’s almost always in that folder, the class GitHub, or Canvas (and most of my content on Canvas is links to documents or web pages, very little except exams lives natively there).

A little organization in advance can save a headaches later.

Scope

Before we get to the organization of the dissertation document itself, I want to spend just a little time on the scope of a dissertation.

A dissertation should contain the research content of approximately three good papers (full conference or journal papers). There are variations on this — sometimes there’s 4 papers, sometimes two short papers replace a full paper — but it’s the basic idea. Not all papers need to be published: at least one should, but the publication process is fickle sometimes. The ideal is probably to have one published, another published or accepted, and a third under review (or in preparation for a deadline shortly after defense).

These three papers will also often not be the only papers you write in the course of your Ph.D — while additional papers are not needed to earn the degree, they’re usually necessary to have a competitive application when pursuing research-oriented jobs.

The 3–4 papers should also be on a theme so you can tell a coherent story of your dissertation work. Three disconnected papers on different topics are hard to sell, and make it difficult for you to make a clear pitch of research directions when you’re on the job market. There isn’t one story for a dissertation, but there are a few general shapes that tend to work well:

The Hammer

A “hammer” dissertation focuses on a particular toolset and demonstrates its applicability across a range of problems; each paper is applying a similar tool to a different problem, and the overall synthesis draws the resulting knowledge together to show where the tool does and does not work, what’s needed to make it effective, and how to adapt it to a range of settings. You have a hammer, and you show that it can pound ordinary nails, U-shaped nails, and some weird five-pointed thing.

The Problem

A “problem” dissertation focuses on a particular problem or problem area and does a deep multi-paper exploration. The papers may apply different techniques to the same problem, or they may examine different stages of the problem or different perspectives on it. The key thing here is that the problem, rather than the tools, are the unifying theme: in the analogy you’re trying to study the problem of fastening things to wood, and study using a hammer and nails, a screwdriver, and glue. Or maybe you study different things to do with wood and similar materials, like pounding, carving, and painting.

The Hybrid

A hybrid dissertation is between the problem and the hammer: it focuses primarily on one application or problem for 2 of the papers, and spends the third showing that the techniques and/or knowledge are also applicable to a related problem. You focus on the problem of fastening things to wooden objects, and then show the hammer can also be used for something else.

There are likely other workable designs as well, but most coherent stories for 3–4 papers will probably fit one of these patterns, more or less.

Dissertation Outline

So you have some papers, and an overall narrative to show how they form a connected and coherent body of work. What does the actual document look like?

There are some variations, but I expect most dissertations I advise to have an outline approximately like this:

1. Introduction. The first chapter casts your overall vision: defines your topic and the terms needed to understand it, presents your story, and previews your contributions. In particular, it sets up your organizing theme (either the hammer or the problem you’re solving). By the end of it, the reader should know (1) what you’re trying to do (including your organizing principle), (2) why it matters, and (3) your core contributions. The rest of the dissertation is to then convince them that you actually make the contributions you claim.

2. Background & Related Work. The second chapter is your primary literature survey. This serves two distinct but related roles¹: first, it covers the necessary background for a reader who is competent in computer science broadly, but not your specific specialty, to understand the rest of your work. Second, it positions your dissertation work in the broader research space, and in particular other work on your problem and related or precursor problems.

   This is the literature survey for your whole dissertation. Some later chapters may also contain small background and/or related work sections that survey work specifically supporting that chapter’s unique work, but the common elements should usually be factored out into Chapter 2. In some dissertations, you may present most of the background in Chapter 1, so Chapter 2 is just related work, but in my experience there’s still background that’s needed in Chapter 2.

3. Common Infrastructure (optional). In some dissertations, you’ll have some resource, such as software or a data set, that you use throughout the entire dissertation. If it doesn’t make sense to describe it in Ch. 2, it can be useful to spend Chapter 3 describing this resource in some detail. In some cases, if it is an original resource, it may also be a paper, particularly if your discipline has venues for publishing resources such as the SIGIR Resource Track or the NeurIPS Datasets & Benchmarks track. Many dissertations won’t have a dedicated chapter for this, though.

4. Research Content. The next 3–4 chapters present your primary research content. Each of your component papers usually becomes a chapter; much of the content can be reused from the paper, but you usually need to make a few changes:

   • Rewrite the intro so it flows narratively as a chapter of a book rather than a standalone paper, including discussions of how it relates to the other chapters (particularly the previous chapters)
   • Rewrite the background and/or related work so that any ideas shared with the other papers in the dissertation are moved to Chapter 2, and the content chapter only has background that’s specific to that chapter’s methods (or the methods that are introduced for use in later chapters). You don’t want to introduce the same related work in three different chapters.
   • Expand the writing to include useful details that you had to drop from the published version for length reasons, further charts and results that shed deeper insight on your findings, etc. In some cases, you may need to re-run or update experiments, particularly if your methods have improved in later portions of the work. What this looks like will differ a lot between dissertations; in my own, Chapter 3 described our research software, which evolved quite a bit from the first published version to what was released as I finished grad school, so I largely rewrote that chapter to describe the software as it was at the end, not the first version.

5. Conclusion. Your last chapter ties it all together: given the vision outlined in Ch. 1, and the work presented in the research content chapters, what do we know about your topic now that we didn’t know before you started the Ph.D? What are the next steps to advance knowledge beyond what you’ve accomplished in the dissertation?

6. Appendices. Some dissertations have appendices; their use varies. I’ve seen them used for additional research content outside the main narrative flow, such as another paper the student wrote. They’re also useful for additional supporting evidence for the research content that would break the flow too much if you included it in the chapter, but you want to make available to readers who wish to check your work more thoroughly. This can include documentation for software you developed, more complete output from statistical models, supplementary charts, etc. If anything is needed to understand one of your research results, however, it should go in the main chapter, not an appendix.

There are some variations on the themes — I didn’t have a 1:1 relationship between papers and chapters in my own dissertation — but for a typical computer science dissertation, this outline will usually work pretty well, and strikes a useful balance (in my opinion) between a pure staple dissertation that does no integration, and a complete rewrite of all the material.

Planning

When you’re planning out your dissertation work, particularly around the proposal stage, that’s what you’re planning to write. Make sure you leave plenty of time for the writing — it can take longer than you expect, and while the dissertation doesn’t need to be your best writing, it should be reasonably good and definitely needs to be clear and readable.

Things I’ve Kept

My stack has stabilized quite a bit over the last few years; many things are the same, or slightly upgraded, from 2019. Highlights:

• Surface devices for portability (Pro 4 at home, Go 2 at work). In 2020 I upgraded from the original Go to the Go 2, which is significant upgrade in performance.
• Dell desktops (2016 XPS i7 at home, 2019 Precision i9 at work)
• iPhone 13 Mini and Apple Watch (I upgraded from the SE to the 11 Pro at the beginning of 2020, and am now on the 13 Mini; only had it for a few days but quite happy with it).
• Windows 10 on the client (with Windows 11 on my Go 2).
• RHEL 8 on the work server.
• Google for work e-mail; Office 365 for personal e-mail, docs, and storage.
• Chrome for work-related browsing (linked to university Gmail).
• Signal and WhatsApp for most personal communication.
• Slack and Hangouts for work messaging.
• Word and Google Docs for writing; Paperpile for citation management.
• Python for most programming; Rust for targeted speed improvements, JavaScript for web-related things.
• Editing in VS Code almost exclusively; a little Nano in the terminal.
• Git and DVC for code and data management.
• PostgreSQL whenever I need a relational (or JSON) database.
• Paint.net for lightweight image edits.
• Grapholite and Visio for diagraming.
• Drawboard PDF for PDF reading and markup.
• 1Password for password management.
• Leuchtturm1917 A5 hardbound notebooks and a collection of fountain pens.

Small Changes

• Switched from Edge back to Firefox for personal browsing. Edge doesn’t scale well to large numbers of tabs, and I believe Microsoft’s shopping assistant is actively harmful to small Internet businesses.
• iPhone upgrade mentioned above.
• WinCryptSSHAgent for YubiKey/SSH support. Considering moving to PuTTY-CAC
• Increased usage of Rust for data processing.

Graphics and Document Software

I still use Paint.net for lightweight raster image editing, but have switched to Serif’s Affinity suite for more substantial work (Photos for raster images, Designer for vector images, and Publisher for advanced document preparation). They’re much less expensive than Adobe’s suite, seem generally more robust and easier to use than the open-source ones I had ben using (Krita, Inkscape, and Scribus).

I’ve also switched from FoxIt Pro to Adobe Acrobat Pro for PDF manipulation at work, now that Boise State has an Acrobat site license.

Bicycling

My Novara Gotham commuter bike suffered frame failure mid-fall. I am replacing it with a Tout Terrain Amber Road (frame-only, will build up with the components from the Gotham, several of which I had upgraded). The result will be a steel-frame commuter with an Alfine 8-speed drivetrain with dynamo hub and Gates carbon drive.

I’m still using the Wahoo ELEMNT Bolt for my road bike rides, but have switched from Strava to RideWithGPS for my cycle tracking. The ELEMNT Mini didn’t work out, both because it wasn’t very reliable, and Wahoo’s software update on the app seemed to break the Mini after they discontinued the product; rather than trying another dedicated computer and hoping it would work, I got a QuadLock handlebar mount to just use my phone for mapping and logging. However, on my commuter, it’s pretty crucial to be able to quickly see the current time along with speed and ride statistics, so I can see how long I have until an appointment at my destination. Strava doesn’t display a clock and doesn’t allow the live interface to be customized. RideWithGPS has a customizable live display including a clock.

Home Network

I’ve standardized my home network devices (NAS and media widget) on Alpine Linux.

The “media widget” is a Raspberry Pi Zero W with a HiFiBerry DAC+ Zero attached, using librespot to play Spotify on our stereo. It also has shairport-sync installed for AirPlay, but we don’t actually use that very often. I tried Raspbian on it for a while, but had update problems with a slow SD card; upgrading the SD card fixed that, but switching to Alpine in “data” mode has worked very well and has a pretty simple upgrade path that doesn’t tax the Pi Zero’s meager processor as much. With only minimal necessary software installed, the system takes about 50–60 MiB of memory, which leaves enough of the board’s 512 MiB free for system operations. I packaged (and maintain) librespot for the Alpine testing repository, so the install and updates are generally pretty smooth. I am very intrigued by the Pi Zero 2 W, but am not in a hurry to upgrade.

The NAS is still our old QNAP 2-bay NAS, running Alpine for consistency across our network (and to give an environment with more resources to test Alpine things for use on the Zero). I don’t actually use the NAS for much anymore; I was using it to backup teaching materials, but need to find new software to facilitate that since the Boise State network blocks Syncthing for unknown reasons. If and when it dies, I don’t know if I will replace it or not; would probably do something on a 64-bit Pi platform.

Current State

The current published version of the tools uses PostgreSQL for data storage and integration, which is pretty good in principle but has significant impedance mismatch with my use of DVC to automate the data processing and integration process. It has an ugly set of hacks involving status files and two DVC stages for almost every logical stage of the data integration pipeline (because my other egregious hack of monkey-patching DVC to support a custom URI schema to track PostgreSQL data status as if it were external data broke with DVC upgrades). The result was a repository layout and set of program interactions that are very non-obvious, so it’s harder than it needs to be for others to extend the tools.

It’s also harder than it needs to be to manage different version of the data, as we have to have different PostgreSQL databases, keep them straight (and synchronized with Git branches), and it all takes a lot of space.

PostgreSQL has served us well, but it’s time to move on.

Upgrade Pathway

In the time since I built the original data integration, the Apache Arrow project has made significant progress on the Rust version of Arrow, along with the DataFusion query engine that supports SQL queries (with a lot of PostgreSQL syntax) over Parquet files.

I have used this to rebuild the integration primarily in Rust and SQL, using custom Rust code for data processing and DataFusion to implement many of the integrations. This fits much better with DVC’s file-based input/output model, as each stage reads input files (from CSV, compressed JSON, MARC, an intermediate Parquet file, or whatever) and produces output in CSV or Parquet format. We can use DVC as intended, without needing to document our own weird pipeline quirks on top of the data structures and integration logic itself.

The result is much faster (2 hours instead of 12), takes less space, and removes both the impedance mismatch and some of the complexity of installation (as users no longer require a PostgreSQL installation). It does mean extending it usually requires writing some new Rust, but we’re generally using our dependences as they’re intended to be used, which should help a lot with that learning curve.

Spotting Problems

One of the things I teach my students about data integration is to always track the coverage or the success rate of each link in their integration pipeline. I do that in this project: for each source of book or rating data, I measure the fraction of records that fail along each stage of the pipeline:

1. Can’t find a book
2. Book doesn’t have authors
3. Can’t find information about authors
4. Author doesn’t have a gender record

This is all reported in the statistics notebook, and allows us to have a good view of the data coverage.

I watched these statistics closely while I was reworking the code, and ensuring we had (approximately) the same coverage was one of the major criteria for making sure I hadn’t broken the logic.

When I was pretty much done, though, I had a problem: the GoodReads coverage was broken. All my other coverages (Library of Congress, BookCrossing, and Amazon) were fine — slightly better, actually, because I had improved some of the ISBN parsing and resolution logic, and had updated the source data to use more current versions of OpenLibrary and VIAF — but GoodReads had gone from about 58% unresolved books to 68%. I fixed two or three other bugs (including working around an annoying bug in the Rust implementation of Parquet) to get there, but the GoodReads coverage wouldn’t budge.

Finding the Problem

After digging around for a while (the details are rather boring), I finally found the problem: the original PostgreSQL integration code had a bug in the GoodReads interaction data integration, where I used an inner join where I should have used a left join. The result is that if a GoodReads book did not have an ISBN, its ratings and interactions were dropped entirely, instead of included as interactions with unknown books. A lot of these are either older books or Kindle exclusives (or editions that haven’t been linked with their works yet); it was a rather large number of books, but they don’t seem among the more active books.

I verified this by replicating the erroneous join logic in the new code, and my coverage jumped back up to where it was supposed to be. Glad I found the problem, but disappointingly it was due to a bug in the version that supported the published results. Once verifying the bug I reverted the change, because including all the records is the correct decision for our purposes.

How Big is the Problem?

This data integration is the data set backing our signature book gender paper. Since the bug only affected books that didn’t have links elsewhere in the database, and most of the analysis focuses on books for which we have data on the author’s gender, I expected it wouldn’t have much impact on the published results: it would affect our GoodReads coverage statistics, and the collaborative filters would be trained on a larger data set (one that includes interactions with these unknown books).

But we need to make sure.

The paper is supported by reproduction scripts that use DVC to automate the entire experiment process: run dvc repro on a large machine, and a day or three later you’ll have the results. This has had quite a few advantages, including allowing us to do a clean re-run and report the compute resources needed to reproduce the experiments in the paper. My team is also using this repository as the basis for further research, so I have been working on upgrading it to use the new version of the data integration instead of pulling from the PostgreSQL database.

So, I imported the new data into the book gender experiment and re-ran it.

As I hoped, the results stayed the same. The coverage plots changed to reflect the new data coverage, but nothing else changed beyond the expected variance due to randomization.

Sigh of relief.

Lessons Learned

I spend a lot of time polishing data and experiment pipelines — sometimes too much time — but the result here was finding an interesting bug, correcting it, and being able to verify that it didn’t invalidate any of our experimental results.

There are many reasons why reproducible pipelines are important, but this is one. It takes time and effort to achieve, but the resulting improvements in robustness and confidence in results are worth significant investment, in my opinion.

Next Steps

I still have some more polishing and documentation to do before releasing the new code, but I plan to update the public version of book data tools to use the new system shortly. I’ll also be pushing a new version of the scripts for the paper that reproduce its results using the current integration, as an example of how to use it. Stay tuned!

The Week

The most central of my rituals is on Sunday evenings. I make a practice of not working on Sundays; in the morning we go (or “go”, during COVID) to church, and in the afternoon I relax, often with a video game.

Sunday evening I begin to prepare for the next week. I fetch my notebook and my pens, typically pour myself a whiskey (tonight I’m drinking a bourbon from Wyoming Whiskey), and reflect on what I need to do the next week. If I’ve been able to rest over the weekend, my mind is starting to think of the next week’s work, and committing those thoughts to paper helps me put them aside (“this is for tomorrow, and it’s written down so I won’t forget”) and start the week with better rest; if I haven’t, it helps me get mentally ready for the week ahead.

I use this time to fill out my Weekly Plan and the (initial) To-Do and Scheduled sections of my Daily Log for Monday (see my notebook description for more details on what these contain). I consult both my calendar and my runway when planning this, to see if there are things coming up that I need to touch this week, if only to check on their status. This practice helps me contain my thoughts, and makes it easier to go to sleep Sunday night knowing that I’ve written down the things I remembered over the weekend that I should do, and I have a plan of action to begin the week. If I don’t make my plans on Sunday, it’s easy to lie awake pondering the things I need to do. Monday morning, I can wake up and meet the day.

Daily To-Dos

I operate day-to-day with a Daily Log. Before beginning the day’s work (or occasionally after a first-thing meeting), I write down my schedule and a set of Desired Outcomes for the day. The details of this ritual are a little more variable; it’s usually in the morning, but I also commonly do it as I wrap up work the day before. The day before option is often helpful when I’m feeling behind, as making a plan for the next day’s work makes it easier to call today’s work done.

Re-Copying Lists

In both my daily and weekly work planning, I am regularly writing down the same items, either because they are recurring, or because they didn’t get done the first time. This physical act of copying them, as noted by Bob Greenberg, helps me feel connected to my work, and forces a regular implicit review of what exactly it is that I’m trying to do.

I’ve used various digital to-do lists, including the rather flexible auto-prioritization logic of TaskWarrior, and for me, nothing beats regular, tangible contact with my workload.

Longer-Term Rituals

At the beginning of each semester, I take stock of what my goals are for the term, what I hope to accomplish for myself and my students. I review and update my runway, breaking out the new term’s months if needed and assessing the planned and committed work.

I used to do annual plans as well, but with the runway giving me a continuous big-picture view, the semester plans seem adequate to the role the annual review used to serve. For a few years, I had a practice of writing a blog post at the end of the year summarizing what I had accomplished with links to public outcomes (see e.g. 2018); I may resume this practice in the future.

Conclusion

Regular, often physical, contact with my workload helps me keep on top of what I need to do. It also helps my mental and emotional health, as planned times to review my work and plan the next week let me offload things occupying (anxious) brain cycles and rest soundly in the knowledge that they’re captured in a form more reliable than neurons. When I’m feeling particularly underwater, the act of writing down what I need to do helps me find my way out and think, concretely, about what to do next.

Different things will work for different people, but I find analog rituals to be crucial and these are ones that work for me at this time in my life. In a year or five I may well be doing something differently. We will see what the future brings.

If you’re feeling completely overwhelmed and having difficulty planning your way through the next day, I found One Minute To-Do List helpful in that situation — it starts with a short analog ritual, and provides useful tactics for taking back some control of your workload.

Runway Organization

I keep my runway in a document in DynaList, an online outliner (similar to Workflowy but with multiple separate files). I have a section for each upcoming term (spring, summer, and fall), at least through the next year. The current term, and occasionally the next term, is broken down with a subsection for each month.

Within these sections, I keep a list of the things coming up that month. Most of these are deadlines or commitments, but some are other planned activities or ticklers (e.g. a reminder to re-activate a currently dormant project). If something has a known deadline, I include that (as a Dynalist date).

When something is done, I mark it as completed; at the end of a period, I mark the whole period (month or term) completed after dealing with the unfinished items (moving them to a later period or deleting them if they are no longer relevant).

Adding to the Runway

Whenever I commit to something with a time period, like a talk or a conference PC, I put in the runway.

When a CFP comes out that I want to target, I put it in the runway. If it’s more of an aspirational target, I’ll put a question mark after it.

When I have a project that I’m ready to actually start working on, I’ll make sure key pieces and the tentative target are in the runway.

When I create the section for a new term that contains deadlines I typically target (FAccT, RecSys, increasingly SIGIR), I put those in, even if the exact deadline isn’t available yet.

If something comes up that’s further out than my runway currently extends, I create a new section for that time block (e.g. upcoming semester or summer).

Using the Runway

The result of this practice is that, in addition to the weekly and daily plans in my notebook, I have continuous telemetry on where I sit with respect to my planned work for the next year (at least).

When I am sitting down to plan my week (a typical Sunday evening ritual; I’ll write more about my rituals in a separate post), I can review the runway to see what’s on my horizon that I need to be making progress on, and should think about adding to the week’s Desired Outcomes.

When I get a new review request, I can look at the runway and see what else I’ve already committed to during that time. I try not to be on more than one regular conference PC in any given month.

Reviewing the Runway

At the beginning of each term, part of my term planning process is to flesh out that term’s outline into per-month details, and review for things I might be missing (as well as things I should drop or defer due to overcommitment).

I also review it as needed and try to make sure it stays current throughout the term; I re-evaluate at least once a month to make sure the current term still reflects my goals and the term’s surprises to date.

Wrapping Up

I’ve been using the runway document for a little over a year now, and it has significantly improved my ability to keep track of my work over time and make more realistic plans on a longer time horizon. I wish I had started this practice years ago — work is less overwhelming when I have it slotted in to specific times and can think ahead about what to expect in future months and terms. I finally feel like I actually have strategic control of my future work, in addition to the tactical control the other pieces of my productivity strategy afford. Time and commitments are also a more useful way to manage future planning at this time than the Kanban-style board modeling the research pipeline.

The runway is a plan, not a prison. I don’t treat it as a firm commitment to myself, and adjust it liberally as my priorities change, new opportunities arise, and projects take more or less time than expected. It does help make space for things, though; if a new collaboration possibility comes up, for example, I can think more concretely about what I might need to give up to make it work, or communicate up front about a realistic timeline for my involvement.

I’m not entirely sure “runway” is really the best name for this document, because I’m always accelerating and never taking off. Sometimes it feels a bit more like the little dog, building his railway track just in time to ride on it, but it’s more like he’s got the controls for a robot that’s laying it out a few more feet in advance.

Python Code

The make_plot function takes care of three things:

1. Chaining the ggplot calls together (since the syntax is slightly less friendly in Python)
2. Applying the theme I’m using for the notebook, along with additional theme options
3. Saving the plots to both PDF and high-DPI PNG
4. Returning the plot for notebook drawing

This function is built for plotnine, a Grammar of Graphics plotting library for Python that I currently use for most of my statistical visualization. It should be possible to write a similar function for raw Matplotlib, or for Plotly, but I have not yet done so.

It uses a global variable _fig_dir to decide where to put the figures. The extra keyword arguments (kwargs) are passed directly to another theme call, to make per-figure theme customizations easy.

Code:

import plotnine as pn
def make_plot(data, aes, *args, file=None, height=5, width=7, theme=theme_paper(), **kwargs):
    plt = pn.ggplot(data, aes)
    for a in args:
        plt = plt + a
    plt = plt + theme + pn.theme(**kwargs)
    if file is not None:
        outf = _fig_dir / file
        if outf.suffix:
            warnings.warn('file has suffix, ignoring')
        plt.save(outf.with_suffix('.pdf'), height=height, width=width)
        plt.save(outf.with_suffix('.png'), height=height, width=width, dpi=300)
    return plt

This can be used like this:

make_plot(data, pn.aes('DataSet', 'value', fill='gender'),
          pn.geom_bar(stat='identity'),
          pn.scale_fill_brewer('qual', 'Dark2'),
          pn.labs(x='Data Set', y='% of Books', fill='Gender'),
          pn.scale_y_continuous(labels=lbl_pct),
          file='frac-known-books', width=4, height=2.5)

The width and height are in inches.

And here’s theme_paper, a custom theme that extends theme_minimal with some text cleanups:

class theme_paper(pn.theme_minimal):
    def __init__(self):
        pn.theme_minimal.__init__(self, base_family='Open Sans')
        self.add_theme(pn.theme(
            axis_title=pn.element_text(size=10),
            axis_title_y=pn.element_text(margin={'r': 12}),
            panel_border=pn.element_rect(color='gainsboro', size=1, fill=None)
        ), inplace=True)

I use these functions in the book author gender code.

R Code

I also have an R vesion from some older projects, before I switched to Python. This one requires you to use + yourself; it doesn’t have any automatic ggplot calls.

make_plot = function(plot, file=NA, width=5, height=3, ...) {
    if (!is.na(file)) {
        png(paste(file, "png", sep="."), width=width, height=height, units='in', res=600, ...)
        print(plot)
        dev.off()
        cairo_pdf(paste(file, "pdf", sep="."), width=width, height=height, ...)
        print(plot)
        dev.off()
    }
    plot
}

You can use it like this:

make_plot(ggplot(frame, aes(x=DataSet, y=value, fill=gender))
    + geom_bar(stat='identity')
    + scale_fill_brewer('qual', 'Dark2')
    + labs(x='Data Set', y='% of Books', fill='Gender')
    + scale_y_continuous(labels=lbl_pct),
    file="frac-known-books", width=4, height=2.5)

I also don’t have automatic theming in the R version, but it would be easy to add.