{ "version": "https://jsonfeed.org/version/1.1", "title": "lukasschwab.me \u2022 blog", "expired": false, "home_page_url": "https://lukasschwab.me/blog/", "feed_url": "https://lukasschwab.me/blog/feed.json", "items": [ { "id": "./gen/comparing-vertex-cover-algorithms.html", "url": "https://lukasschwab.me/blog/./gen/comparing-vertex-cover-algorithms.html", "title": "Comparing Vertex Cover Algorithms", "content_html": " Comparing Vertex Cover Algorithms « blog

Comparing Vertex Cover Algorithms

In \u201cGraphs at Work\u201d I claim

To improve the customer experience, what you need to understand is the distribution of strategy performances for a given algorithm on real-world graphs, not the theoretical worst case.

Of course, that\u2019s a simplification. Theoretical worst-case performance does impact the customer experience, either on accident or because some adversary wants to throttle your program. Worry less if your \u201ccustomer concerns are well-isolated,\u201d but they rarely are.

By \u201creal-world graphs\u201d I suggested a family of associations graphs, distinct from the family of all possible associations graphs, might describe the bulk of customer desires. For example, if the typical customer associates Contacts, Deals, and other objects via a mutual association with a Company, those customers\u2019 \u201creal-world graphs\u201d will approximate stars with the Company at the center.

Parameterizing those graph families, you can simulate the candidate algorithms\u2019 performance against a wide range of possible customer configurations (even configurations impossible in practice, e.g.\u00a0because of a limit on the number of HubSpot types).

Experiment space

All the uncertainty about what\u2019s \u201creal-world\u201d translates into a huge experimental space. Any given experiment embeds a set of assumptions about what the user probably wants and what they have in HubSpot. All the experiments taken together are only useful if they describe the problem well enough, in all its dimensions, to react to user information as it comes in.

Most of the variables here are discrete categories \u2014 different vertex cover algorithms, different topological families of graphs, and different weighting schemes for vertices. For these purposes, there isn\u2019t a continuous space between Vazirani\u2019s algorithm and Lavrov\u2019s.

Some of those categories introduce scalar variables, like the size of a random graph or the probability two vertices are adjacent.

Vertex cover algorithms

An unweighted vertex cover algorithm will never k-approximate a minimal weighted vertex cover. Unweighted algorithms behave as if all vertices have the same weight, then minimize the number of vertices in the cover Vreq. When you add weights to G\u2004=\u2004{V,\u2006E}, Vreq might be arbitrarily bad: each vertex in Vreq can be arbitrarily expensive, and vertices in V\u2005\u2212\u2005Vreq can be arbitrarily cheap.

Obviously the best an unweighted algorithm can perform on a weighted graph is to accidentally suggest the weighted optimal strategy.

How should we expect it to perform given a random distribution of weights? I think of a single graph with its topology-determined minimum unweighted vertex cover Vreq. The expected weight of that vertex cover is \u03bc\u2225Vreq\u2225 where \u03bc is the mean of the weight distribution.

What if weights aren\u2019t distributed randomly, and instead correlate with graph topology? I\u2019m not sure what to expect. That question is easier to simulate than to theorize.

I implement three of the algorithms discussed in my last post.

clever

The greedy algorithm I previously implemented in Elixir. I came up with it independently, but I take the name from Michel Zito.1

The punchline: this algorithm isn\u2019t clever. It performs so badly on tricky graphs it doesn\u2019t k-approximate minimal vertex covers. To this algorithm\u2019s credit, it has an intuitive heuristic: add the highest-degree vertex in G to Vreq, remove it from G, and recurse for a minimal vertex cover on what remains.

lavrov

the \u201c2-approximating [unweighted] vertex cover\u201d variation suggested by Mikhail Lavrov:

Start with Vreq\u2004=\u2004\u2205. Then, as long as Vreq doesn\u2019t cover every edge, we pick an uncovered edge and add both endpoints to Vreq.2

My implementation is an even greedier variant, at the cost of some runtime complexity: rather than arbitrarily picking an uncovered edge, it picks the uncovered edge with the highest combined degree between its incident vertices. Lavrov proves the 2-approximation.

vazirani
The only weighted vertex cover algorithm I implement. Rather than removing vertices outright, it relaxes virtual weights t(v) for the vertices incident on a \u201ctaken\u201d (uncovered) edge, then adds one or both to Vreq depending on the result. My implementation leaves edge-selection arbitrary, per Vazirani\u2019s definition.3

As the only k-approximation of a minimum weighted vertex cover, it\u2019s safe to assume vazirani should perform the most stably. That doens\u2019t moot the experiment. As I noted, there are graphs where it yields a worse result than clever!

Topologies

Well, what\u2019s a \u201creal-world graph?\u201d In the absence of customer data \u2014 I don\u2019t have any \u2014 we can try simulating several parameterized families of graphs instead of just guessing. I implement two families.

random
A graph with n vertices. Any pair v1,\u2006v2\u2004\u2208\u2004V are connected with probability p. Notably, these graphs have a nondeterministic total connectivity \u2014 there\u2019s always a p2n chance of getting a complete graph! \u2014 but that\u2019s just a reason to simulate repeatedly.
tricky

\u201cTricky\u201d graphs follow a construction from Lavrov designed to force the clever algorithm into arbitrarily bad vertex covers. Instead of taking a number of vertices, the construction starts with integers a and k\u2004\u2264\u2004a; then

On one side of the graph, put n vertices, for some (large) n. Call the set of these vertices A.

On the other side of the graph, put k\u2005\u2212\u20051 \u201cblocks\u201d of vertices, numbered B2,\u2006B3,\u2006...,\u2006Bk. Block Bi will have \u230an/i\u230b vertices.

Each vertex in A gets one edge to a vertex in Bi, and they are distributed as evenly as possible: each vertex in Bi ends up with either i or i\u2005+\u20051 neighbors in A.4

From Lavrov (2020): a \u201ctricky\u201d graph with n\u2004=\u200420 and k\u2004=\u20045. The edges adjacent on B3 illustrate each B-group is adjacent to all of A.

What makes it tricky? Initially the highest-degree vertices are in Bk. When clever has included those in the cover Vreq, the highest-degree vertices are in Bk\u2005\u2212\u20051. This continues until Vreq\u2004=\u2004B2\u2005\u222a\u2005B3\u2005\u222a\u2005...\u2005\u222a\u2005Bk, even though there\u2019s a smaller vertex cover staring right at us: A!

These graphs have a\u2005\u22c5\u2005Hk vertices, where Hk is the kth harmonic number: \u2225A\u2225\u2004=\u2004a and \u2225Bi\u2225\u2004=\u2004(i\u2005\u2212\u20051)a.

There are potentially many more families to explore. Like I mention above, stars might capture HubSpot\u2019s ontology for built-in types well: it\u2019s a CRM, so pretty much everything associates, directly or indirectly, with a Company. Graphs composed of stars, joined at the leaves, might reflect HubSpot customers using custom objects to hang subontologies off of the built-in types: a Company\u2019s association with a Deal indirectly associates it with a collection of Deal-specific custom object types.

I considered testing a \u201csmall-world\u201d family of graphs, with vertices highly-adjacent to \u201cneighbors\u201d and adjacent to non-\u201cneighbors\u201d with some lower probability, but I\u2019m not sure it\u2019s relevant to the HubSpot problem definition. There\u2019s no natural concept of \u201cneghbors\u201d among HubSpot object types. Remember: customer \u201cconfigurations\u201d are subgraphs of a complete graph of types. Emergent graphs \u2014 especially physical networks \u2014 may well have small-world properties, but it\u2019s less clear why a subtractive configuration-selection process would yield them.

Weight schemes

The cost of requesting the objects of a HubSpot type is the number of members divided by the APIs request page size. How many objects of each type exist? Who knows! Like with graph families, it\u2019s easier to simulate several of them than to evidence-free agonize over which is \u201crealest.\u201d

uniform
Assigns every vertex the same weight, unweighting the vertex cover problem.
random
Assigns each vertex a random weight on the half-open interval [0,\u20061).
degree-negative
A weight scheme where a vertex\u2019s weight is negatively correlated with its degree. To keep weights positive, I use (d(v)\u2005+\u2005\u03f5)\u2005\u2212\u20051, where \u03f5 is a small positive constant to prevent divison-by-zero panics on disconnected vertices.
degree-positive
Each vertex\u2019s weight is its degree.
degree-positive-superlinear
Each vertex\u2019s weight is the square of its degree.

I expect degree-negative and degree-positive to reward and punish the clever algorithm, respectively. clever gobbles high-degree vertices into Vreq oblivious to their weight. In one case that greedy strategy correlates with low weights, in the other case with high.

Project structure

Why reach for Go? My first instict was to use built-in benchmarking to compare algorithms. That is a cool tool (one of my favorite take-home interview hacks) but I don\u2019t wind up using it. Runtime performance is just too dependent on implementation details; I\u2019m better off spending my time building a graph package with a decent interface than, say, using separate graph implementations optimized for each algorithm.

There are other, better reasons to write Go. Static types prevent a suite of bugs I\u2019d encounter in Python \u2014 Go\u2019s a \u201cwhen it compiles, it does what you mean\u201d experience \u2014 and module management is simpler than in Typescript. The standard testing library is pleasant. godoc is built in, and linters abound.

The stand-out advantage, I found, is a convention: the standard Go project layout, which separates reusable library code in pkg subdirectories from applications in cmd, is well-suited to experimentation. I wound up with a project structure like this:

The simulation models depend on one another, but they\u2019re independent of the experiments that use them. pkg/graph exports weighted and unweighted graphs, along with some known graphs used in tests. pkg/cover implements vertex cover algorithms on those graph types.

Both packages include tests. Testing is especially useful in pkg/cover, where it tests two kinds of correctness: that my code behaves how I expect it to, and that my expectations match the algorithm definitions in literature! By testing on known graphs (defined in pkg/graph/fixtures.go), I check my implementations match examples from papers. TestClever_Tricky confirms clever selects Vreq\u2004=\u2004B like Lavrov claims; if it doesn\u2019t, there\u2019s some mistake in the implementation even if it yields valid vertex covers.

Experiments go in subdirectories of cmd. Each directory is space for notes, outputs, experiment-specific helper code, and so on. Because there\u2019s no risk of one experiment impacting the others \u2014 there aren\u2019t interdependencies in cmd \u2014 these are scratchpads.

In summary:

  1. Put reusable simulation components in pkg.
  2. Use unit tests to confirm your implementations against prior art examples.
  3. Explore in cmd.

Next time I might reach for a scientific computing package like Gonum to aggregate data in experiments. It even ships with a graph package, albeit one suited for edge-weighted problems over vertex-weighted ones.5

Simulations

Any pair of experiments should be comparable at a glance. Absolute vertex cover weights are sensitive to graph topology and individual vertex weights, which makes it hard to compare an algorithm\u2019s performance on a ten-vertex graph against another algorithm\u2019s performance on a hundred-vertex graph.

Instead, since my goal is to compare algorithms, I measure relative cover weight against a baseline: vazirani. Because it\u2019s a 2-approximation, it guarantees a bound on any algorithm\u2019s relative performance: if it\u2019s implemented correctly, nothing can do more than 50% better than vazirani.

In the heatmaps below,

Each cell averages ten relative performances: ten times, the simulation generates a graph G with the given parameters, then finds vertex covers on that G with the vazirani and tested algorithms.

For starters, we can visually confirm a behavior predicted in literature: Lavrov\u2019s tricky graphs are designed to arbitrarily damage the clever algorithm\u2019s performance.

As Lavrov suggests, the \u201cclever\u201d algorithm is tricked for large a,\u2006k. (Note: while k is an integer in Lavrov\u2019s tricky graph construction, here it\u2019s expressed as a multiple of the number of vertices in A.) Does the lavrov algorithm do better?

Indeed it does! Strikingly, lavrov performs just as well as vazirani on a uniformly-weigted graph. Upon reflection, I realize vazirani and lavrov are equivalent in a uniformly-weighted context: t(v) always goes to zero for u and v at every step, because both have the same starting weight!

To my surprise, clever outperformed vazirani on tricky graphs when vertex weights corresponded 1:1 to degree.

On reflection, this makes sense too. Any vertex cover of a degree-positive-weighted graph covers every edge, which means it has a minimum weight \u2225E\u2225. By minimizing redundant coverage, clever approximates that optimal cover. Since vazirani is a 2-approximation, the optimal cover is at most 50% better.

What if vertex weights are superlinearly positively correlated with vertex degree?

Huh! To my surprise, except for a low-n low-k blip, clever still generally outperforms vazirani. Even weirder:

I expected degree-negative weighting would reward the clever algorithm for picking high-degree vertices, but it doesn\u2019t!

lavrov overview

A weight-na\u00efve version of vazirani, lavrov achieves parity when it isn\u2019t punished for weight-na\u00efvet\u00e9 (when weights are uniform). Otherwise, vazirani\u2019s performance lower-bounds it; you\u2019d only prefer lavrov if you need its programmatic simplicity.

Weight/Topology random tricky
uniform
random
degree-negative
degree-positive
degree-positive-superlinear

clever overview

As expected, clever does poorly on tricky graphs with uniform and random vertex weights.

Unexpectedly, it performs quite well on tricky graphs where vertex weight correlates positively with vertex degree! In domain-specific terms, that suggests clever is good choice if your HubSpot account\u2019s most-associated types are also your most numerous collections.

I still gut-distrust this conclusion. It\u2019s just so easy to think up a small but plausible associations graph where clever\u2019s dismal because, say, you have hundreds of thousands of Contacts.

Weight/Topology random tricky
uniform
random
degree-negative
degree-positive
degree-positive-superlinear

If you want to poke around the experimental results, check out the full set of interactive graphs. If you want to reproduce them or inspect my implementations, see the source code on GitHub.


  1. Zito, Michele. \u201cVertex Cover.\u201d (2005). Also see Lavrov (2020).\u21a9\ufe0e

  2. Lavrov, Mikhail. \u201cLecture 36: Approximation Algorithms\u201d (2020). 2-3. Note: I\u2019ve modified the definition to share variables with the other algorithms.\u21a9\ufe0e

  3. Vazirani, Vijay V. Approximation Algorithms. Vol. 1. Berlin: Springer, 2001.\u21a9\ufe0e

  4. Lavrov, Mikhail. \u201cLecture 36: Approximation Algorithms\u201d (2020). 2-3. Note: I\u2019ve modified the definition to share variables with the other algorithms.\u21a9\ufe0e

  5. Since there\u2019s a minimal Steiner tree weighted-edge equivalent to this vertex cover problem \u2014 see my addendum in Graphs at Work \u2014 I could compare Steiner tree algorithms using gonum.org/v1/gonum/graph.

    There are advantages to reimplementing graph, though. For example, clever and lavrov both recurse into searches on subgraphs, so my graph package defines an easy-to-call (*Weighted).Without(removed Vertex) function.\u21a9\ufe0e

", "summary": "Go simulations, on several families of graphs, comparing the algorithms discussed in \"Graphs at Work.\" Can you reasonably give up a $k$-approximation (Vazirani) for a simpler weight-na\u00efve strategy?", "date_published": "2022-10-30T22:50:38.644000+00:00" }, { "id": "./gen/graphs-at-work.html", "url": "https://lukasschwab.me/blog/./gen/graphs-at-work.html", "title": "Graphs at Work", "content_html": " Graphs at Work « blog

Graphs at Work

You might never invert a binary tree, but graph theory \u2014 and graph algorithms \u2014 worm their way into software engineering, whether you recognize them or not. This problem wormed its way into my brain last week, where it reminded me API engineering isn\u2019t all just flat-mapping responses. Conceiving this as a graph problem doesn\u2019t just guide implementation: it determines the product\u2019s attributes, which will impact its direction.

One of Sequin\u2019s products is a two-way integration betwen our customers\u2019 HubSpot accounts and their Postgres databases, using the HubSpot API to live-update the database and forward writes to Postgres back to HubSpot. There\u2019s a variety of entities in HubSpot: Companies, Contacts, Deals, and all the custom object types our customers\u2019 hearts desire. (Want to index Cars in HubSpot? Make a custom Car type.) Our customers might have thousands of Contacts, hundreds of Companies, and so on.

Of course, these entities aren\u2019t islands; a Contact works for a Company, a Deal binds a company and is negotiated by a Contact, and so on. Rather than capturing every possible relationship on each type, HubSpot has a separate associations model for connecting two objects. For example, an association might declare \u201cContact Lukas Schwab is associated with Company Sequin.\u201d Those associations are always bidirectional; if I\u2019m associated with Sequin, Sequin\u2019s also associated with me.

Because associations are a core part of HubSpot\u2019s product model, Sequin\u2019s sync needs to scrape them into Postgres. HubSpot\u2019s API makes that a little tricky; I\u2019ll try to explain this in brief, because it\u2019s important to undertanding the task at hand.

If you want to fetch all associations between Contacts and Companies from the HubSpot v3 API, you can use either of two endpoints:

Because HubSpot limits each customer\u2019s daily API usage, the second endpoint is the much more desirable access pattern. Still, we have to pick a request strategy. Given the association types a customer wants synced to Postgres \u2014 type pairs like Contact \u2194\ufe0e Company \u2014 what fromObjectTypes should Sequin list? The project requirements are to design a request strategy which

  1. Systematically pulls all associations the customer desires, specified as pairs of types.
  2. Minimizes the number of requests to HubSpot\u2019s API.

For instance, a customer\u2019s desired association types might look like this:

From To
Contact Company
Deal Contact
Company Deal
Company Car
Company Policy
Car Policy
Company Owner

But it\u2019s easier to start thinking about a simpler configuration:

From To
Contact Company
Deal Contact

This is when I recognized graphs would be good visualizations for request strategies. Make each unique type (in this example, Contact, Company, and Deal) a vertex; then the associations we want to fetch are edges on the graph. The customer configuration is essentially an adjacency list!

The associations we want, expressed as a graph: Contact \u2194\ufe0e Company and Deal \u2194\ufe0e Contact. The Contact type is involved in both associations, but it\u2019s reduced to a single vertex.

Coloring the types you\u2019d list with GET requests gives you a visual representation of each request strategy.

Request strategy Visualization
GET contacts
associations=[company]

GET deals
associations=[contact]
GET contacts
associations=[company, deal]
GET deals
associations=[contact]

GET companies
associations=[contact]

Now that you\u2019re working with graphs, you have access to a pithier expression of the problem. The set of desired associations defines a graph G\u2004=\u2004(V,\u2006E), where the edges E are desired associations and the vertices V are the HubSpot types involved. Our search for a \u201crequest strategy\u201d is a search for an optimal subset Vreq\u2004\u2286\u2004V such that every e\u2004\u2208\u2004E is incident upon \u2014 touching \u2014 at least one object type v\u2004\u2208\u2004Vreq.

All of the strategies in the table above satisfy that condition, but one of them \u2014 and only one valid strategy for that graph! \u2014 only lists objects of a single type, Contacts.

There isn\u2019t always a single-type silver bullet. For some graphs, no single vertex is incident on every edge.

In this associations graph, the simplest valid strategy lists objects of two types: Deal and Company. No single vertex is incident on every edge.

You\u2019re faced with the algorithmic question I faced on Friday: given any graph like this, how can you systematically produce a valid minimal strategy?

Cracking the graph

As a rule of thumb, greedy graph algorithms \u2014 approaches which repeatedly apply a single heuristic to make local decisions \u2014 are easier to stomach. Because the heuristic is the crux of the implementation, you can focus on documenting that. With any luck, it won\u2019t give your code reviewer a headache.

Luckily, there\u2019s an intuitive greedy heuristic for selecting Vreq from V! Consider your start state and end state:

At any intermediate step, what change to the Vreq you\u2019re building makes the most progress towards that end state? The one that adds the most new edges in E to Ereq!

More formally, given an associations graph G\u2004=\u2004(V,\u2006E),

  1. Let Vreq\u2004=\u2004{}, Ereq\u2004=\u2004{}.
  2. While E\u2004\u2260\u2004{}:
  3. Return Vreq.

G shrinks with each step, so this is guaranteed to terminate for any finite associations graph (and, since there\u2019s a finite limit to the number of object types in a HubSpot account, G is guaranteed to be finite \u2014 phew). I\u2019ll call this the clever-greedy algorithm, after Zito.2

This has a straightforward \u2014 if inefficient \u2014 recursive implementation,3 but don\u2019t sweat it if you don\u2019t want to read code. You can use the visualizations to see successive applications of the algorithm chip away at G as it builds the set of types Vreq.

Step Vreq Ereq Subgraph Evaluation
0 \u2205 \u2205 Company and Deal both have three incident edges. Take Deal.
1 {Deal} Deal\u2194\ufe0eContact
Deal\u2194\ufe0eCompany
Deal\u2194\ufe0ePolicy
Company has two incident edges; Contact and Policy each have one. Take Company.
2 {Deal, Company} Deal\u2194\ufe0eContact
Deal\u2194\ufe0eCompany
Deal\u2194\ufe0ePolicy
Company\u2194\ufe0eContact
Company\u2194\ufe0ePolicy
There aren\u2019t any more edges in G, so we have a valid Vreq!

Problem solved, right? You reframed the problem in terms of graphs, which led you to a greedy algorithm \u2014 just a couple-dozen lines of code \u2014 that\u2019ll produce a valid, seemingly minimal request strategy every time.

There\u2019s one issue. In practice, what\u2019s optimal? What do the project requirements say about optimality?

  1. Systematically pulls all associations the customer desires, specified as pairs of types.
  2. Minimizes the number of requests to HubSpot\u2019s API.

Are you minimizing API usage with the greedy strategy? Not necessarily! You\u2019re minimizing the number of types you have to list, true, but the API cost of listing a type is nonuniform. Because each GET request only returns a page of \u2264100 objects, the number of API requests we have to make for a strategy scales with the total number of objects of the types in our strategy.

Do fewer types mean fewer objects? That\u2019s not an unreasonable guess if you know nothing about the system, but you don\u2019t have to look far for a counterexample. Consider a tweaked version of our initial simple associations graph, where you have additional info about the number of objects in HubSpot: there are 300 Contacts, 100 Companies, and 100 Deals.

Strategy Vreq API cost
{Contact} 300 contacts.
\u2234 Three requests.
{Deal, Company} 100 Deals.
100 Companies.
\u2234 Two requests.

The first strategy is simpler \u2014 it only involves listing objects of one type \u2014 but it\u2019s costlier. The project requires optimizing API cost, not strategy simplicity!

Can you get there by modifying the greedy strategy? That\u2019s not clear. The simple heuristic (\u201cpick the vertex that adds the most new edges in E to Ereq\u201d) also needs to factor in each vertex\u2019s cost. Sometimes a voluminous type is worth requesting because it\u2019s incident on many edges; sometimes it isn\u2019t. In complex graphs, it\u2019s hard to figure out locally whether a given vertex will wind up being a good pick or a bad one. Hmm.

The graph cracks back

Looks can be deceiving! This puzzle is a minimal vertex cover problem disguised as a HubSpot API question. Introducing cost minimization makes it a weighted minimal vertex cover problem. With or without weights, the optimization problem is NP-hard: as far as the career math-freaks know, there isn\u2019t a solution that works in polynomial time. Any general algorithm for truly optimizing your HubSpot associations-fetching is going to be really, really slow. Bummer!

If you don\u2019t care about performance, there\u2019s a brute-force solution in O(2n):

  1. Filter the 2n subsets of V for vertex covers of G, the valid API strategies.
  2. For each, calculate the API cost.
  3. Take the lowest-cost strategy.

What\u2019s wrong with the clever-greedy algorithm? Provided a graph designed to trip it up, it can\u2019t even guarantee results within any constant-factor multiple of the optimal weight!4 There are other greedy algorithms for vertex cover guaranteeing results no larger than twice the minimum (2-approximations).5 My favorite is Savage\u2019s, which simply prunes the leaves from a spanning tree.6

It stands to reason that introducing weights makes this harder; unfortunately, since we\u2019re working with weighted vertices, you can\u2019t just apply Savage\u2019s reasoning to a minimum weighted spanning tree (which you could build with Prim\u2019s algorithm). There are 2-approximation algorithms for weighted vertex cover, like this one adapted from Vazirani:7

2.11 Consider the following algorithm for the weighted vertex cover problem. For each vertex v, t(v) is initialized to its weight), and when t(v) drops to 0, v is picked in the cover.

Algorithm 2.17

  1. Initialization:
    • Vreq\u2004\u2190\u2004\u2205
    • \u2200v\u2004\u2208\u2004V,\u2006t(v)\u2004\u2190\u2004weight(v)
  2. While Vreq is not a vertex cover do:
    • Pick an uncovered edge, say (u,\u2006v). Let m\u2004=\u2004min\u2006(t(u),\u2006t(v)).
    • t(u)\u2004\u2190\u2004t(u)\u2005\u2212\u2005m.
    • t(v)\u2004\u2190\u2004t(v)\u2005\u2212\u2005m.
    • Include in Vreq all vertices having t(v)\u2004=\u20040.
  3. Output Vreq.

This has essentially the same halting condition as our greedy algorithm, but it decreases a vertex\u2019s weight t(v) as its neighbors are added to the eventual set cover Vreq. Intuitively, a vertex with a high initial weight is unlikely to ever hit t(v)\u2004=\u20040, so it\u2019s unlikely to wind up in Vreq. An algorithm like Vazirani\u2019s might make a fine solution for getting an okay minimization.

Is it the solution for you? Like the earlier question about whether a \u201csmall\u201d strategy was optimal, this comes down to the requirements \u2014 implicit and explicit \u2014 of the project. The pragmatizing effect of project requirements is really, in some sense, what sets software engineering apart from computer science.

This puzzle explicitly requires you minimize API usage. Unfortunately, neither the clever-greedy algorithm nor Vazirani\u2019s algorithm guarantee optimal outcomes. Vazirani guarantees at worst a 2-approximation of the optimal strategy, whereas the clever-greedy algorithm can\u2019t even promise a constant factor for an unweighted set cover. Practically, though, worst-case performance might not matter! User-provided lists of assosiations to fetch usually won\u2019t be the graphs that trick clever-greedy into poor outcomes. If customer concerns are well-isolated, an adversarial customer can force your algorithm into an inefficient solution\u2026 but they can\u2019t impact anyone else\u2019s experience. To improve the customer experience, what you need to understand is the distribution of strategy performances for a given algorithm on real-world graphs, not the theoretical worst case. Here clever-greedy might be better, even if it assumes constant vertex weights.8

Instead, implicit project requirements \u2014 values inherent to the engineering task \u2014 could reasonably determine what you implement. New engineers will have to read and modify the algorithm you choose; because that time trades off with other work, your customers may be better-served holistically if you pick a simple, maintainable algorithm here over a strong n-approximation. That\u2019s the principle behind minimizing tech debt.

Lehman describes the evolution of programs through a system of three nesting levels:

  1. S-programs implement specifications: \u201creturn the lowest common multiple of two integers.\u201d
  2. P-programs, made up of S-programs, strive to perform optimal behaviors: \u201cplay chess.\u201d
  3. E-programs, made up of P-programs, \u201cmechanize a human or societal activity.\u201d Software prducts, by nature of having collaborators and users, are E-programs.9

The HubSpot project requirements ask you to design a P-program. At several junctures, feeling comfortable working with graphs helped break that task into S-programs: graphs gave us a natural visualization, which helped design a greedy algorithm. Recognizing a well-studied problem in graph theory, the vertex cover, exposed flaws in that greedy algorithm \u2014 the adversarial cases \u2014 and led to the conclusion all known optimal solutions are inefficient.

Critically, though, graph theory is handy for understanding the relationship between P-program and E-program. Discovering an existing 2-approximation like Vazirani\u2019s might be the difference between a worthwhile feature and a waste of time. Having the vocabulary to study the associations graphs you see in practice lets you study whether worst-case performance matters, or if maybe something simpler (unweighted) will usually do better.

In summary, this isn\u2019t just engineering navel gazing; recognizing a minimum weigted vertex cover problem when you see one impacts your bottom line. Losing engineering hours to battling P\u2004\u2260\u2004NP or reading abstruse optimizations won\u2019t keep the lights on.

Brush up on your graph theory, and wield it, but remember you\u2019re building a product \u2014 don\u2019t miss the forest for the trees.

I didn\u2019t recognize the weighted vertex cover problem at first. Intead, I worked through expressing this problem as two other NP-hard optimization problems, distrusting my own conclusions. Here\u2019s a bit more math for anyone curious!

My first theory was that I could calculate the minimum edge cover of a modified line graph L(G) \u2014 a task much easier than NP-hard \u2014 but that was wrong. For our problem, the edges in L(G) are non-independent; you have to take or leave all the Contact edges as a batch. If I successfully reduced minimum vertex cover to minimum edge cover, I would suddenly be very, very popular.

Fiddling with line graphs led me to this more fruitful representation of the three-vertex weighted example discussed above:

The root is an artificial vertex. It\u2019s connected to each vertex in V by an edge weighted by the cost of listing all objects of that type. The final row of vertices represents the association types we want to fetch. An association vertex\u2019s parents are the types at either end of the association. Once you\u2019ve paid the cost of listing Contacts, you have all the associations involving contexts; that holds for any type, so all of these edges are weighted 0.

The optimal request strategy is a subgraph spanning the blue terminal vertices, the subgraph with the smallest total edge weight that does so. Think of it this way: the bottom row of vertices represents the data we need to get to. We can either go through (list) Deals to get the Contact\u2194\ufe0eDeal associations, or we have to go through (list) Contacts.

We\u2019re looking for a minimal Steiner tree in a directed graph \u2014 like a minimal spanning tree, but free to ignore non-terminal vertices. The terminal vertices are in blue above: the root and the desired association types.

The minimal Steiner tree for this simple example has weight 200.

Certain properties of these graphs make them a simpler than the general class of Steiner tree problems (namely, their fixed height guarantees that they\u2019re pseudo-bipartite), but that doesn\u2019t save it from NP-hard status.

Computing minimum-cost r-Steiner trees is NP-hard for r\u2004\u2265\u20044, even if the underlying graph is quasibipartite. The complexity status for r\u2004=\u20043 is unresolved, and the case r\u2004=\u20042 reduces to the minimum-cost spanning tree problem.10

I\u2019ll spare you the details, but r here is one plus the highest degree of a vertex in G \u2014 three for this simple example case, and often greater than or equal to four in practice.

Still, I wondered if there was some other relevant feature of these graphs that simplifies the problem. I looked for a more familiar representation, and found one: a weighted set cover.

For brevity, I\u2019ll use shorthand for the HubSpot types; instead of Contact, Company, Deal, etc., I\u2019ll write A, B, C, and so on. You\u2019re looking for a number of associations specified by unique unordered pairs of those types; for example, {(A,\u2006B),\u2006(B,\u2006C),\u2006(B,\u2006D),\u2006(C,\u2006D)}.

This set of pairs is the universe \ud835\udcb0 to be covered. The sets doing the covering represent the endpoints we can call: for each type, the associations in \ud835\udcb0 incident on that type:

These constitute a family \ud835\udcae of subsets of \ud835\udcb0. The set cover problem is to find a minimal number of members in that family such that their union spans \ud835\udcb0, analogous to the optimal HubSpot API strategy. In this case you can get all desired associations by listing objects of two types: Bset\u22c3Cset \u2004=\u2004{(A,\u2006B),\u2006(B,\u2006C),\u2006(B,\u2006D)}\u22c3{(B,\u2006C),\u2006(C,\u2006D)} \u2004=\u2004{(A,\u2006B),\u2006(B,\u2006C),\u2006(B,\u2006D),\u2006(C,\u2006D)} \u2004=\u2004\ud835\udcb0.

Bad news: finding a minimal set cover is NP-hard.

As you might imagine, the weighted set cover assigns a weight to each member of \ud835\udcae and demands you minimize the weights. For this problem, you\u2019d assign a weight Aset with the API cost of listing all objects of type A\u2026 but don\u2019t bother: solving weighted set cover isn\u2019t any easier.

Interestingly, there are exceptions! Finding a minimum set cover on 2-sets is equivalent to finding a minimum edge cover, which is easier. Tugging on this thread11 led me to discovering the weighted vertex cover problem, which exactly matched my original project definition.

That match confirmed it: any weighted vertex cover problem can be expressed as a question about HubSpot associations, so optimizing this HubSpot API strategy is, in fact, NP-hard.


  1. A request for a page of Contacts, with Company and Deal associations included, takes this form. Note the weird URL-encoding.

    curl --request GET \\\n  --url 'https://api.hubapi.com/crm/v3/objects/contacts?limit=10&associations=contact&associations=deal&associations=company&archived=false' \\\n  --header 'authorization: Bearer YOUR_ACCESS_TOKEN'
    \u21a9\ufe0e
  2. Zito, Michele. \u201cVertex Cover.\u201d (2005). Also see Lavrov (2020).\u21a9\ufe0e

  3. Here\u2019s an Elixir implementation of the greedy strategy for finding Vreq:

    @spec greedy_V_req(list({String.t(), String.t()})) :: list(String.t())\ndef greedy_V_req([]), do: [] # Base case: no edges in G.\n\ndef greedy_V_req(graph) do\n  # Greedily take the highest-degree vertex.\n  taken =\n    graph\n    |> Enum.flat_map(&Tuple.to_list(&1))\n    |> Enum.frequencies()\n    |> Enum.max_by(fn {_k, freq} -> freq end)\n    |> elem(0)\n\n  # Remove taken from graph; recurse.\n  subgraph =\n    graph\n    |> Enum.filter(fn pair -> taken not in Tuple.to_list(pair) end)\n\n  [taken | greedy_V_req(subgraph)]\nend

    &greedy_V_req/1 receives graph as an adjacency list of order-agnostic tuples. To specify the example graph:

    greedy_V_req([\n  {"Contact", "Company"},\n  {"Contact", "Deal"},\n  {"Deal", "Company"},\n  {"Policy", "Deal"},\n  {"Policy", "Company"}\n])

    This implementation is inefficient \u2014 O(|V|2) \u2014 but it isn\u2019t worth optimizing.\u21a9\ufe0e

  4. Lavrov, Mikhail. \u201cLecture 36: Approximation Algorithms\u201d (2020). 1-2.\u21a9\ufe0e

  5. Zito, Michele. \u201cVertex Cover.\u201d (2005). Also see Lavrov (2020).\u21a9\ufe0e

  6. Savage, Carla. \u201cDepth-first search and the vertex cover problem.\u201d Information processing letters 14, no. 5 (1982): 233-235. A very tidy three pages.\u21a9\ufe0e

  7. Vazirani, Vijay V. Approximation Algorithms. Vol. 1. Berlin: Springer, 2001. My adaptation renames C to Vreq for consistency with earlier examples, and removes steps updating c(e) \u2014 our problem doesn\u2019t require we track edge \u201ccosts,\u201d part of Vazirani\u2019s exercise.\u21a9\ufe0e

  8. I want to test this on random graphs. For now, consider the strategy Vazirani\u2019s algorithm picks for this simple weighted graph:

    Vazirani\u2019s algorithm lists all three types, even though, once you\u2019ve listed any pair of types, listing the third is strictly redundant.

    Notably, this four-request solution is still a 2-approximation of the optimal 2-request solution from running clever-greedy on the unweighted graph. You could tweak Vazirani\u2019s last step (\u201cInclude in C all vertices having t(v)\u2004=\u20040\u201d) to guard against trivially-unnecessary additions like this.

    Counterintuitively, one 2-approximation algorithm for the unweighted set cover intentionally includes vertices which redundantly cover a given edge; doing so thwarts the clever-greedy adversarial case.\u21a9\ufe0e

  9. Lehman, Meir M. \u201cPrograms, life cycles, and laws of software evolution.\u201d Proceedings of the IEEE 68, no. 9 (1980): 1060-1076.\u21a9\ufe0e

  10. K\u00f6nemann, Jochen, David Pritchard, and Kunlun Tan. \u201cA partition-based relaxation for Steiner trees.\u201d Mathematical Programming 127, no. 2 (2011): 345-370. The introduction to this paper is the clearest overview of Steiner trees I found.\u21a9\ufe0e

  11. Thanks to David Richerby for answering \u201cSet cover problem with sets of size 2\u201d (July 5, 2017).\u21a9\ufe0e

", "summary": "Last week at work I found a wicked graph problem at the heart of an innocuous API task. This post explains the problem, its relationship to several well-studied discrete math puzzles, and comments on the difference between computer science and software engineering.", "date_published": "2022-09-03T18:23:14.827000+00:00" }, { "id": "./gen/sequin-technical-writing.html", "url": "https://lukasschwab.me/blog/./gen/sequin-technical-writing.html", "title": "Technical Writing for Sequin", "content_html": " Technical Writing for Sequin « blog

Technical Writing for Sequin

I joined Sequin earlier this year. I joined in part because they promised I could wear many hats; they\u2019re true to their word, and I\u2019m moonlighting in content marketing!

I\u2019m focused on improving two aspects of my technical writing:

See Blogging with TikZ, State Machines via Jorge Luis Borges, and Processing PDFs with Cloud Functions for examples of (loosely) technical writing outside my work.

I\u2019ll keep this list up-to-date instead of cross-posting articles.


What to do without HubSpot Webhooks
Published July 11, 2022.
A review of shortcomings in HubSpot\u2019s CRM Webhooks API. Webhooks should be a neat system for triggering event-driven tasks outside HubSpot, but these have a number of shortcomings. So does HubSpot\u2019s low-code automation system, a commonly-recommended alternative.
Stripe metered billing, simplified
Published August 26, 2022.
Stripe\u2019s metered billing strategy lets you bill customers according to how much they use your product. That should let you tether your billing to the value your product provides, at the cost of requiring you constantly report cusotmer usage to Stripe. Reading Stripe data through Sequin saves an API call in each reporting cycle.
", "summary": "An index of writing I publish as part of my work on Sequin --- a data integration tool for software engineers --- usually on Sequin's blog. I'll keep this list up-to-date instead of cross-posting articles.", "date_published": "2022-08-27T19:19:16.957000+00:00" }, { "id": "./gen/recruiting-emails.html", "url": "https://lukasschwab.me/blog/./gen/recruiting-emails.html", "title": "Managing Emails from Recruiters", "content_html": " Managing Emails from Recruiters « blog

Managing Emails from Recruiters

I have a GitHub profile, so I\u2019m doomed to receive emails from tech recruiters until the end of time.

Until recently, I just archived emails that didn\u2019t interest me. An unfortunate proportion of the recruiters take this as an invitation to email me again (and again) to ask if I\u2019ve seen the first. This is a bad strategy on their part.

I\u2019ve taken to sending a short, generic template email explicitly declining their invitations:

Hello\u2013\u2013thanks for reaching out!

I\u2019m not looking for new job opportunities at this time, but I\u2019ve read and labeled your email so I can find it if my priorities change.

Best of luck in your search,
Lukas
labelRecruiting

\u201cRead and labeled?\u201d Sometimes it really is useful to have these emails categorized:

As you might\u2019ve noticed, I include a line in white text at the end of my template: labelRecruiting.

This is the crux of my trick: Gmail filters are evaluated for outgoing mail. I set up a filter on label:sent labelRecruiting that applies the label Recruiting.

Image of the Gmail query panel: label:sent captures all sent mail, and labelRecruiting searches email text for that token.

It takes two clicks to fill the template response. When an inbound email is especially good or especially off-base, I add a short note to that effect before sending. The filter notices the lazily-hidden keyword and applies my label.

As much as it chafes to dignify plainly spammy recruiters with a response, replying does usually prevent pushy follow-ups. The underlying mechanism \u2014 using a hidden string to trigger filter behavior \u2014 should be useful whenever you need to bulk reply-and-categorize. Let me know if you think of something clever!

", "summary": "You can set up a Gmail filter that evaluates for outgoing mail. Hiding a keyword in a template response is a neat time-saver for cold outreach from recruiters.", "date_published": "2022-07-06T00:40:55.341000+00:00" }, { "id": "./gen/technical-debt.html", "url": "https://lukasschwab.me/blog/./gen/technical-debt.html", "title": "Tech Debt and Taxes", "content_html": " Tech Debt and Taxes « blog

Tech Debt and Taxes

In Reframing Tech Debt, Leemay Nassery suggests framing technical housekeeping positively \u2014 as building \u201ctech wealth\u201d \u2014 to convince business and product stakeholders it\u2019s worthwhile.

If I were one of those stakeholders, a self-professed customer-obsessive, I\u2019d balk (rightfully!) at the idea of a secondary system of value independent from the user experience. \u201cWhat does \u2018tech wealth\u2019 mean for a customer?\u201d Fair question!

I think a negative framing is more useful here, but Nassery\u2019s suggestion that, say, refactoring a program yields a resource to reinvest gets at the issue with calling it \u201cdebt.\u201d Technical \u201cdebt\u201d sounds like a fixed cost that can be deferred; more accurately, we should say debt exacts an ongoing tax from go-to-market teams. The costs of quick development may be hidden, but they aren\u2019t externalized.

If you make implementation shortcuts in the name of product velocity, but you don\u2019t have the discipline to clean them up, those shortcuts will eventually be counterproductive to the speed-of-delivery you hoped to prioritize. Changes to a codebase are more often integrative than strictly additive: they have to contend with the code that\u2019s already there. Writing sloppy code today means writing code slowly tomorrow.

In each of these cases, tech debt\u2019s the common enemy of the product stakeholder and even the most navel-gazing, customer-indifferent engineer. While the underlying debt isn\u2019t addressed, the stakeholder pays a tax \u2014 as a delay, or as systematic under-verification that\u2019ll eventually yield bugs.

This dynamic is particular to API and product debt \u2014 unnecessary complexity in how parts of the system interact with each other (interface and data model design, documentation, and testing practices) or the space of customer states (feature flags and billing). These forms of debt tax new projects in proportion to those projects\u2019 complexity; big projects and radical changes in product behavior are proportionally punished. Of course, not all \u201ctech debt\u201d should be prioritized this way; poor database performance might usually be considered debt, but it\u2019s less likely than an unexpressive API to derail your next feature.

The usual refrain \u2014 including in Nassery\u2019s article \u2014 is to preallocate blocks of time for building tech wealth. The \u201ctax\u201d translation from tech into customer impact should let a team prioritize specific tech debt initiatives, according to the same criteria as features.2

Unsurprisingly, it falls to engineers and their managers to use this framing device effectively.

Business and product stakeholders can take some proactive action.

The issue with \u201ctech debt\u201d and \u201ctech wealth\u201d is that they underemphasize software\u2019s impact on a company\u2019s agility, its ongoing ability to adapt to serve its users\u2019 needs. Without recourse to customer value, advocating against technical debt means advocating for a separate and competing system of value, something business and product stakeholders are right to treat with skepticism.

Emphasis on the productivity impact makes technical debt expressible in the primary system of value, where it\u2019s comparable against \u2014 and can be prioritized over \u2014 any other company initiative.

Process Plants: A Handbook for Inherently Safer Design (Kletz 1998) reminded me of this essay. A process plant designer can intervene to prevent hazards (e.g.\u00a0a \u201csnakepit\u201d plant layout), but investors don\u2019t value hazard mitigation for its own sake. Quoting Kirkland,

This cannot all be the fault of the \u201cmoney men\u201d and their failure to understand us. It is often much more our inability to express ourselves in a language they can understand.3

I\u2019m more optimistic than Kirkland. I think the \u201cmoney\u201d stakeholders do understand hazards, but that they\u2019re better-equipped to predict and prioritize money. To that end, Kletz offers napkin math:

Remember that if it costs $1 to fix a problem at the conceptual stage, it will cost

Software could use some equivalent with costs in various stages of planning, development, and deployment. \u201cHazard\u201d may be a better analogy than \u201ctax\u201d for describing the running cost of technical debt.


  1. Take the time to keep your feature flags mutually independent, even if that means running migrations to split or merge flags; make invalid states unrepresentable. If you don\u2019t, each new flag effectively doubles the user interface surface you design and test.\u21a9\ufe0e

  2. Development acceleration should have a double-integral relationship to the value delivered by new development, but expressing this accurately would require some complex discounting.

    Imagine you can divide your fixed engineering time in a quarter between tdebt and tfeat. Customer value delivered looks something like (v\u2005+\u2005tdebt)\u2005\u22c5\u2005tfeat, where v is your initial product velocity. As tfeat approaches zero, the team spends all of its time preparing to deliver value\u2026 but none of its time actually doing so.

    This model is short several coefficients, but hopefully you get the idea.\u21a9\ufe0e

  3. Kirkland, C. J. The Channel Tunnel: Engineering under Private Finance \u2014 Innovation or Frustration (London: Fellowship of Engineering, 1989). Quoted in Kletz, page 157.\u21a9\ufe0e

  4. Kletz, Trevor. Process plants: A handbook for inherently safer design. CRC Press, 2010. Page 165. For specific examples of how \u201cchang[ing] early\u201d saves money, see page 202.\u21a9\ufe0e

", "summary": "An argument for highlighting the customer cost, via product unreliability and feature delays, of technical debt. Getting the time to clean up old bodges is a matter of revealing the constant but oft-hidden productivity tax of leaving them in place.", "date_published": "2022-03-14T02:36:37.514000+00:00" }, { "id": "./gen/memories-of-evans-hall.html", "url": "https://lukasschwab.me/blog/./gen/memories-of-evans-hall.html", "title": "Collected Memories of Evans Hall", "content_html": " Collected Memories of Evans Hall « blog

Collected Memories of Evans Hall

Berkeley\u2019s Evans Hall is unmitigatably ugly, inside and out, even at a great distance; to the discerning student, Evans is the ugliest structure on campus. Last week The Daily Californian reported it\u2019s to be demolished as part of the University\u2019s 2021 Long Range Development Plan.

This news is a long time coming. As the Daily Californian and SFGATE articles discuss, Evans has been widely disliked since it was built to Gardner Dailey\u2019s design in 1971. The University floated a proposal to demolish Evans, and build a complex of low buildings on the site, in 2000; they punted that decision, citing budgetary constraints and a long line of competing seismic priorities.1 Now Evans is on the docket again: it\u2019s just unacceptably bad to look at.

This perspective is true, but it elides a deeper reading of the value in buildings. There\u2019s a sort of memory in Evans Hall itself: of institutional dehumanization; of the studentry\u2019s miniature ingenuity; of milestones, and major pests, in computer science; and of love (yes, even in Evans).

Always the best primer for a UC Berkeley building, The Loafer\u2019s Guide notes Evans Hall\u2019s

main stairwell balcony that faces a concrete wall, eye level with a bare fluorescent light fixture. This building, Davis, and Wurster are Neobrutalist, derived from the French word for \u201cexposed concrete.\u201d Before its completion, the Daily Cal printed an editorial entitled \u201cLove can\u2019t live in an ugly temple.\u201d In addition to design and mechanical problems beginning Day 1 at \u201cFort Evans,\u201d students began to paint and otherwise \u201cadapt\u201d the building. The \u201cDeath of Archimedes\u201d and other unsanctioned artworks are worth checking out.2

John Rhodes, Professor Emeritus of Mathematics, remembers those \u201cunsanctioned artworks\u201d as a kind of antifascist gesture. The department was moved into Evans Hall in 1971, the year it was completed. Striving to do creative work, they felt consigned to a building so austere Rhodes found it dehumanizing.

Sometime after this seminar [on creativity in mathemtatics], I organized a one-hour talk in 60 Evans by an assistant professor in the architecture department entitled Fascism and Architecture, being as Evans Hall was despised by many as a dehumanizing workplace. The subject of the talk was the effect of architecture on politics and the spirit. I remember the speaker showed slides of Hitler waving to crowds from the balcony of buildings in Berlin.

After the applause abated at the end of this well-received talk, I passed out paintbrushes and paint cans. Even Sarah Hallam (senior administrative assistant of the Department for many, many years) grabbed a paintbrush. We rushed upstairs and painted some of the walls on the 7th to 10th floors.

Voil\u00e0! That\u2019s how the \u201cmurals\u201d got there. Some were painted by famous mathematicians \u2014 [William] Thurston and [Dennis] Sullivan. And I know La Mort de Galois on the 7th floor was painted by my son\u2019s uncle, Jack Knutson. I thought perhaps some of you \u2014 young and old \u2014 might like to know the history of these paintings. I hope as many of them can be preserved as possible.3

The painting of Galois included here is dated December 1971; it may be La Mort de Galois, but I\u2019m not sure. Sullivan also reports it was 1971.4 Remember, Evans was completed in \u201971; the math department got mad fast. Sarah Hallam\u2019s retirement in 19785 is a latest date. In any case, I\u2019ll try to photograph these murals the next time I\u2019m on campus.

Evans was only painted its current institutional grey-green in the late 1990s, part of a renovation to prevent \u201clarge pieces of concrete [\u2026] falling off the face of Evans Hall without warning\u201d (yikes).6 Apocryphally, the color was selected to blend the towering blight into the hillside; in practice, this marks the building as a place of great aesthetic shame. B\u00e9ton brut was better.

In 2002, shortly after the repainting, first-year architecture student Alfred Twu ran the Evans Hall Coloring Contest: entrants printed, then recolored, folding scale models of Evans. First and second place went to trompe l\u2019oeil panoramas, braver (and more surreal) imaginings of the hidden-in-plain-sight grey-green. The competition\u2019s a great example of lighthearted Berkeley creative initiative.7 All contest submissions and the vote tally are here.

L. K. Leong\u2019s winning submission to the Evans Hall Coloring Contest (2002). A foreground trees in silhouette shade the walkway around Evans; behind them, low peaks (the Berkeley Hills?) blush orange.8
Tom Barber\u2019s second-place submission to the Evans Hall Coloring Contest (2002). It appears to be a photo of Memorial Glade (the eastward view from Evans) stretched to gargantuan proportions.9

Of course, there\u2019s more to Evans than its aesthetics. The departments it houses have astounding academic records; at work, I regularly use software spawned in its miserable greenish bowels. All that the comes from hardworking people, decades of students and academics, and the building is a testament to their experience inasmuch as they tell stories about its architecture.

The most elaborate such stories are in the CSUA Encyclopedia.10 Though it was written after the Computer Science Undergraduate Association\u2019s relocation to Soda Hall, the Encyclopedia can\u2019t disentangle the CSUA\u2019s history and in-jokes from Evans itself.

Sometimes Evans was a nuisance:

elevatorP
Incantation to summon the elevator to the basement of Evans. Also, a means to subvert the security measures of Evans. Largely obsolete. Usually initiated from the tvi920c in the WEB print room.
In ye olden days, the public computers at Berkeley were grouped in two locations \u2014 in the WEB, located in the basement of Evans, and in E260, located on the second floor of Evans. The WEB was easilly accessible, but E260 was in an area of Evans which was locked in the evenings. Fortunately, the elevators on the south side of the building could be used to subvert the lackluster security measures; unfortunately, the elevators could not be called from the basement. elevatorP was initiated by students in the WEB to call out to their brethren in E260, and so summon elevators down to the WEB. Could also be used to gain access to the former CSUA office in E238.
Lock Collection
Holy CSUA artifacts. A set of locks rumored to have once adorned the secret north door of Evans Hall.
Little is recorded regarding how the Lock Collection might have come into existence, but according to quiet whispers, the north door of Evans might once have been a secret entrance to the hallowed lands of E260 and the CSUA office in E238. It has been suggested that, perhaps, if the lock on the north door went missing, it would have been easier for computer students to attain the computers of the second floor Evans, but archaelogical evidence is too scant to verify this. In all likelihood, the CSUA Lock Collection is actually composed of discarded locks and has no relation to this theory.
Three Evans Route
Secret path through the Evans Security Perimeter.
The administration tried very hard to maintain the Evans Security Barrier, manning the area with campus police and fixing the occasional mysteriously missing lock; one must given them credit for that. However, no one ever said they were very bright in their efforts. Take a case-in-point, the Three Evans Route.
Locked glass doors can be used to prevent student scum from entering the upper reaches of Evans without barring them from the twin nirvana of WEB and vending machines; no question about it. But, what if there were a room, say 3 Evans, which had two entrances, one on either side of these magic glass doors? If there were such a theoretical bone-headed design, it would be very simple to slip past the Perimeter, without having to wait for a doosher to notice the call of elevatorP.
Purely theoretical, and in any case the administration remembered to lock the outer Three Evans doors on occasion.

And sometimes Evans was a place to practice being a nuisance:

CSUA Bat (or CSUA Baseball Bat)
Holy CSUA artifact and the symbol of office for the Vice President. A black Louisville slugger, well-worn and loved. [\u2026] Particularly useful for obsoleting CRTs, gaining votes at politburo meetings, emphasizing points at general meetings, and hitting Evans to relieve stress (the last being its original purpose, according to rumors).
Fencing Club
Saber-rattling bunch of CS students nearly foiled by the UC police. Their motto: \u201cdon\u2019t take a fence.\u201d
Picking up fences and hauling them across campus is very funny. Explaining to UC police that you weren\u2019t doing so is even funnier. Still funnier is getting a chain-link fence, dragging it through the Evans Security Perimeter, and then using it to block off the office of a beloved lecturer. Even the beloved lecturer will agree it\u2019s as funny as anything, while plaintively asking that said chain-link fence please be removed.

And sometimes Evans was more than the sum of its nuisances. Deep in the CSUA Encyclopedia, in the \u201cPoetry\u201d entry, in a full-alphabet acrostick, there\u2019s an extraordinary name.

i is for ikiru, Euphrasia Lavette Alzena Guri Scientia Ventura Ikiru; and then
j is for more ikiru, Alvera Ganbatte Gelasia Curvilinearjky\u2026 van Bezooijen

ikiru pased away in 2011. She took the additional \u201cvan Bezooijen\u201d when she married Eric van Bezooijen (CSUA Encyclopedia contributor) in 1997. Eric\u2019s eulogy for Euphrasia paints a brilliant and complicated picture, though reading it feels uncomfortably like an intrusion on something personal: \u201cI could spend all day telling you the things she gave me.\u201d

\u201cHow did I meet Euphrasia? We met in the early 1990s in Evans Hall room 260, a UC Berkeley computer lab, not the most romantic of all places.\u201d

Actually, I think this captures the romance of architectural memory. Our common architectural referent \u2014 Evans \u2014 makes Eric and Euphrasia real, makes their experience seem immediate. Locating Eric\u2019s story sets up an exchange of meanings: it marks Evans 260 as a place, in Marc Aug\u00e9\u2019s sense.11 That means, in turn, Evans 260\u2019s existence testifies to the reality of their experience. The place is an encounter between us, now, and them, then.

Moreover, Evans\u2019s architectural memory is alive; students are still groaning about the green-grey, about the slow elevators, about the hulking blight. Evans isn\u2019t a museum of the work that was done there; it\u2019s a place to take part in the continuation of that work, to participate (even in some miniscule way) in the history that is the very basis for this memory.

Of course, participation means imagining something better. Working in Evans Hall every day made me unahppy, and would make me unhappy again. The University\u2019s plan to raze Evans Hall is even older than the Coloring Contest plans to dress it up as nature. They\u2019re right to prioritize current and future academics over campus history: UC Berkeley isn\u2019t a museum, it\u2019s a workplace.

One shouldn\u2019t desperately conserve miserable places, but one should read buildings \u2014 beyond their aesthetics and functionality \u2014 for what they remember: our collective memory, in architecture.


  1. Keller, Josh. \u201cNo Stirrings of Pride\u201d in The Chronicle of Higher Education. (2007).\u21a9\ufe0e

  2. Dougherty, Carolyn. \u201cThe Existing Campus\u201d in The Loafer\u2019s Guide to UC Berkeley.\u21a9\ufe0e

  3. Rhodes, John. \u201cA History of the Murals in Evans Hall\u201d in Berkeley Mathematics Newsletter, Fall 2002 Vol. IX, No.\u00a01. (2002)\u21a9\ufe0e

  4. Mosher, Lee and Bill Casselman. \u201cWhat is\u2026 A Train Track?\u201d in Notices of the AMS, Vol. 50, No.\u00a03. (2003). According to Thurston, the mural-painting continued \u2014 over administrative protestation \u2014 after Rhodes\u2019s start:

    Last fall Sullivan wrote to Mosher: \u201cIn 1971 I was a guest of the University of California giving lectures in the Math Dept. At the same time there was a confrontation between the trustees and the graduate students et al.\u00a0The latter planned to continue decorating the walls of the department by painting attractive murals and the trustees forbade it. At tea some students came up and invited me to join their painting the next day. I became enthusiastic when one bearded fellow [Thurston] showed me an incredible drawing of an embedded curve in the triply punctured disk and asked if I thought this would be interesting to paint. I said, \u2018You bet,\u2019 and the next day we spent all afternoon doing it.

    [\u2026]

    Thurston wrote in a note to Mosher that the project \u201cwas in response to a little flurry with administration sanctions of some sort when John Rhodes painted the wall outside his office, I think with a political slogan related to one of the issues of the times (Vietnam war, invasion of Cambodia, People\u2019s Park?).\u201d

    \u21a9\ufe0e
  5. Felde, Marie. \u201cHallam Bequest Benefits Math\u201d in The Berkeleyan, March 13, 1996.

    There\u2019s some disagreement: a departmental chronicle reports Hallam retired six years earlier, in 1972. Interestingly, it also reports she was hired half-time by Evans \u2014 the building\u2019s namesake \u2014 in 1936, for a $400 salary. See Mathematics at Berkeley: A History by Calvin C. Moore (2007).\u21a9\ufe0e

  6. Keller, Josh. \u201cNo Stirrings of Pride\u201d in The Chronicle of Higher Education. (2007).\u21a9\ufe0e

  7. The Evans Hall Coloring Contest is theoretically interesting. Using scale models as a means of participatory architecture would be right at home in Log 50: Model Behavior \u2014 and it would stand out. The authors are largely focused on one-off canonical models (the stuff of architecture exhibitions) or models as devices for iterative studio design, and they all reference the same Borges story and enumerate the issue-theme multi-entendre.

    The Evans Hall Coloring Contest\u2019s paper models are cheaply reproduced, structurally simple, and accessible to nonprofessional participants.\u21a9\ufe0e

  8. Twu, Alfred and contestants. Evans Hall Coloring Contest Winners. (2002).\u21a9\ufe0e

  9. Twu, Alfred and contestants. Evans Hall Coloring Contest Winners. (2002).\u21a9\ufe0e

  10. Appelcline, Shannon with pictures by Eric van Bezooijen. Welcome to soda.csua.berkeley.edu, Now Go Home! or When Geeks Collide. (1994).

    This is really a remarkable document; well worth the read if you have any affiliation with UC Berkeley Computer Science.\u21a9\ufe0e

  11. See vaguely Aug\u00e9, Marc. Non-places: Introduction to an anthropology of supermodernity. (1992).\u21a9\ufe0e

", "summary": "Evans Hall, once home to UC Berkeley's Computer Science department and current home to several others, is slated for demolition. While that's good news, Evans Hall is a point of contact with history, a sort of concrete collective memory.", "date_published": "2022-02-13T23:41:59.793000+00:00" }, { "id": "./gen/hearst-hall-continued-reading.html", "url": "https://lukasschwab.me/blog/./gen/hearst-hall-continued-reading.html", "title": "Hearst Hall: Continued Reading", "content_html": " Hearst Hall: Continued Reading « blog

Hearst Hall: Continued Reading

Writing typically hurries one of my interests out of its \u201call-consuming\u201d phase and into \u201cembarrassed retirement,\u201d but Hearst Hall keeps coming up. I borrowed Kenneth Cardwell\u2019s Bernard Maybeck: Artisan, Architect, Artist from the SF Public Library, and suddenly I\u2019m all-consumed again.

That last post was heavy on my own, uh, sketchily-supported commentary. Cardwell, to my relief, supports a number of those claims. He dedicates the better part of a chapter to Hearst Hall\u2014in Cardwell\u2019s telling of Maybeck\u2019s career, it\u2019s prototypical of his first professional phase: redwood buildings, mixing gothic-evocative forms and prominent structural rhythms (especially of trusses and arches).

The book has its shortcomings. Cardwell praises unrelentingly. When a Maybeck building disappoints, it\u2019s always because the owner had poor taste, or meddled in rennovations, or that Maybeck must\u2019ve been too hands-off. Cardwell refuses to discuss Maybeck in relation to his contemporary architects besides, very briefly, a teacher (Louis-Jules Andr\u00e9) and a student (Julia Morgan). Greene & Greene, who built the William R. Thorsen House in Berkeley, are conspicuously absent. John Galen Howard is treated like a bureaucratic gadfly. Cardwell, after all, dedicated much of his career to Maybeck alone: building an archive of Maybeck\u2019s documents for Cal\u2019s College of Environmental Design. Some auteurism is to be expected.

I won\u2019t introduce any truly new ideas. Instead, I\u2019ll string together excerpts (mostly, but not exclusively, from Cardwell, and with emphasis added) pertinent to the arguments in the last essay: that Hearst Hall\u2019s bare structure is notable; that Maybeck\u2019s philosophy stressed the adaptability of spaces, even to the point of accepting their destruction; and that Maybeck exemplifies certain foundational principles of modernism.

Style and Structure

I did my best to argue the prominence of Hearst Hall\u2019s structure\u2014those laminated archies\u2014in its deisgn, but I only had a handful of interior and exterior photos. Cardwell describes that structure, and the means of its production, in finer detail.

\u201cHearst Hall under construction.\u201d1

The reception pavilion, Hearst Hall, was built on the south side of Channing Way, adjoining Mrs.\u00a0Hearst\u2019s residence on Piedmont Avenue. It was a building sixty feet in width and one hundred forty feet in length, with its principal axis oriented in a north-south direction on a site which sloped gently to the south and west. The structure\u2014the engineering of which was worked out in collaboration with Maybeck\u2019s university colleague Herman Kower\u2014was an exciting and bold concept that employed laminated arches and diaphragms to create sections of the buildings as independent, movable, and re-assembled units. Twelve wooden arches rising fifty-four feet above the ground floor level were paired to form six structural bays. Single arches were used at the walls of the end bays. These arches were constructed of laminated two-by-eight timbers, formed horizontally on the site by nailing the timbers to a radius of approximately sixty feet. Holes were then drilled and bolts placed which were tightened after erection loads had been imposed. The arches were given lateral stability by the floor of the main hall which was twelve feet above the ground floor. it consisted of a diagonal grid of beams and joists, which was left exposed in the ceiling of the lower room. The arches were further tied, at approximately half height, by a horizontal plane surrounding the exterior of the arches, creating a floor for an outdoor gallery. The exterior walls of the building were curtain walls of stud construction with columns at each arch. These columns were tied at the middle to the arch by horizontal members which extended through the exterior to support planting boxes.2

Laminated timber seems a pretty modern technical decision, but I\u2019ll come to a stronger point about concrete later. The real surprise in this passage is about the ties between the arches and the curtain walls extending to form Hearst Hall\u2019s exterior planters. The ogive arches are outwardly apparent in Hearst Hall\u2019s front elevation\u2014that might suffice to argue its structural frankness\u2014but these planters give outward expression to these ties, these much more obscure structural elements. There\u2019s a similar structural extension through the building in Maybeck\u2019s Outdoor Art Club building in Mill Valley.

The design for the Outdoor Art Club (1904) in Mill Valley added a new roof variation. It has a simple rectangular plan spanned by rafters trussed with a king post and collar ties. The ties extend through the roof covering and join a vertical extension of the wall columns, thereby strengthening the weakest part of the rafter chord between the tie and the support. Maybeck probably adopted this form for two reasons: first, the spaciousness of an uninterrupted vertical volume under a roof plane pleased him; second, the adaptation of a king post truss thrusting through the roof fulfilled his desire to indicate the nature of the building\u2019s framing from the exterior.3

The Outdoor Art Club in Mill Valley. Structural members extend through the roof to the building\u2019s exterior, similar to the extended ties that support Hearst Hall planters. A gabled interior volume nests in\u2014and exceeds\u2014a rectangular structural frame.4

Speaking of exterior extensions, the \u201cweird turret-obelisk\u201d dangling from the Hall\u2019s west face serves a purpose:

The tower at the right contained the flue from the heating plant. It also embellished the terrace roof garden gallery.5

It seems Maybeck didn\u2019t bother moving it\u2014the chimney is bare in all photos of Hearst Hall on campus\u2014and I can\u2019t say I blame him. Hearst Hall\u2019s exterior sillhouette isn\u2019t all that charming (an effect worsened in black and white), but the integration of the original chimney seems slapdash, after-the-fact, like a towering wart. I haven\u2019t been able to find any photos of the \u201cterrace roof garden;\u201d it sounds fantastic.

Cardwell sheds some light on the matryoshka spaces-within-spaces effect\u2014not the round arches of electric lights, but the alcoves and the fabric frozen in mid-air:

The alcoves formed by the spaces between the ribs of the arches were hung with Gobelin tapestries and various paintings from the Hearst collection. When the Hall was used for less formal occasions, such as afternoon teas given by Mrs.\u00a0Hearst, a large canopy of Genoese rose velvet was suspended from the arches in the middle of the hall.6

Posh! These tapestries (\u201cof the fifteenth and sixteenth centuries [\u2026] all historic treasures\u201d) were relocated to the Maybeck-designed Hearst manse at Wyntoon, where presumably many were destroyed (with the castle) in a 1929 fire.

Tapestries in Hearst Hall (left)7 and at Wyntoon (right).8 It\u2019s hard to tell in these scans, but the tapestry in the middle Hearst Hall alcove is\u2014post-relocation\u2014also the far tapestry in the Wyntoon photo.

Finally, and importantly, Cardwell identifies some material decisions that don\u2019t quite read in the interior photos.

The interior of the main hall was finished with redwood barn shakes similar to those on the exterior but treated with a sulphate of iron solution which rendered them a silvery-grey. Light came from clerestory windows covered by grilles of turned wood balusters of flemish derivation, typical of Maybeck\u2019s ornamentation. [\u2026] The plain pine floor was covered by two large Chinese rugs of special design executed in grey-blue with a gold border. At night the room was lighted with more than a thousand incandescent bulbs. Two rows of colored lights were mounted directly on each arch of the room, and suspended from each arch were two rows of white lights with milk glass reflectors hung in an arch complementary to the structural members.9

These aren\u2019t frank structural elements, but they\u2019re interesting inasmuch as the interior presented Maybeck a margin of freedom: he could readily have built an interior that elided the Hall\u2019s structure. Instead, the interior surfaces repeat the exterior (shingles! inside!) and lights trace the ogive arches.

Adaptation

My last essay argued Maybeck\u2019s architecture involved a certain openness towards its occupants. I was working from two pieces of evidence: first, that Hearst Hall was primarily a multipurpose social space, and then that Maybeck reflected on the importance of architecture\u2019s \u201cinmates\u201d in his written reflections: \u201cthe real interest must come from those who are to live in it.\u201d

Luckily for me, Hearst Hall isn\u2019t a fluke; this openness was a part of Maybeck\u2019s practice in general, a part especially pronounced in his residential architecture.

In addition to technical and business details, he expressed his ideas on the relationship of a house, its owners, and its furnishing [to a Mr.\u00a0and Mrs.\u00a0Havens, clients]. \u201cWe believe that when you have seen the house and become accustomed to it,\u201d he wrote,

there will be a thousand and one ideas come rushing to your mind to complete it\u2026 We therefore are getting an idea of the cost of things and when you have decided what you will do to put life into the rooms, you will make such a success of it that the thing as a whole will have that interest and personality which even as a plain business proposition will be a good investment\u2026 We also were under the impression that you could get your own personality into your house better if we stopped at the building proper, so as to leave you unhampered in the final work of furnishings.10

Lewis Mumford remembers not every starchitect was this open-minded:

Frank Lloyd Wright, it is said, once turned upon a client\u2014let\u2019s call him John Smith\u2014who had added a few pleasant rugs and comfortable Aalto chairs to Mr.\u00a0Wright\u2019s furnishings, and exclaimed, \u201cYou have ruined this place completely, and you have disgraced me. This is no longer a Frank Lloyd Wright house. It is a John Smith house now.\u201d11

The physical openness of Hearst Hall\u2019s floorplan isn\u2019t unusual given the project parameters: a subdivided main space wouldn\u2019t have suited music or theatrical performances, for example. Later in his career Maybeck advocated for open floorplans in a setting where they were unusual at the time: in residential architecture.

In 1923 Sunset Magazine published an article entitled \u201cThe Maybeck One-Room House,\u201d in which he proposed the incorporation of house and garden into one entity, concentrating time and effort as well as cost on one handsomely proportioned and beautifully furnished space. Service rooms were to be reduced to such a minimal size that they would become mere alcoves.

Maybeck\u2019s argument in Sunset Magazine is essentially that the \u201cOne-Room House\u201d is economical, but it\u2019s predicated on the idea that one \u201cbeautifully furnished space\u201d can serve a variety of functions admirably.

Maybeck\u2019s preference\u2014that users should tailor and modify his work\u2014extends to his urban- and campus-planning. A campus, he contends, is eventually \u201clived in\u201d in the same cozy sense as a house.

Fifty years from now the plan of the University will have become modified and softened; it will be transformed many times, because so easily done. The buildings will have undergone changes, so planned from the beginning that they could grow. This patchwork gives the same feeling to the whole composition as the new stones in a cathedral\u2013\u2013the newness becomes tarnished, the monotony of exactness relieved. By that time the gardens will be older, and in places a vine may soften the harshness of perfection.12

Loss as final adaptation

Maybeck saw an astounding portion of his work destroyed by fire. His office, with his early records and drawings, burned in the fires that followed the 1906 San Francisco earthquake. Innumerable early \u201cGothic houses\u201d were leveled in East Bay blazes. Wyntoon burned. Hearst Hall, of course, burned. Twice Maybeck insists on destruction as a sort of final adaptation\u2014a recapture into a different aesthetic scheme, a radical extension of the vine that softens the harshness of perfection.

The Palace of Fine Arts used the impression of ruination to its advantage from the getgo. Maybeck aimed to set a somber mood, to give the impression of a palace that was always already lost.

[\u2026] Frank Morton Todd gives complete technical and physical descriptions of the structure, but when attempting to describe the character of the building in his volumes, he hesitantly states

[\u2026] The theme itself we might attempt to state as the mortality of grandeur and to describe as having some affinity with our eternal sorrows over the vanity of human wishes\u2026

Some such feeling as this, though vague, must have come to every responsive intelligence that looked across the Fine Arts Lagoon and the Palace itself. It represented the beauty and grandeur of the past. A cloister enclosing nothing, a colonnade without a roof, stairs that ended nowhere, a fane with a lonely votary kneeling at a dying flame, fluted shafts that rose, half hid in vines, from the lush growth of an old swamp\u2026 all these things were in the picture.

It was evident that Maybeck had succeeded in setting a mood. It was also evident that he had achieved his goal of creating a beautiful building. But all the words and the praise heaped on the building fail to explain what it was about the architectural forms that contributed to its universal appeal. Even when Maybeck wrote about the Palace, not one word refers to the building itself, or even to any of its parts. All of his explanation is devoted to the mood appropriate for an art gallery\u2013\u2013which was \u201ca sad and serious matter\u201d\u2013\u2013and this is done through repeated allusions to the haunting character of remanants of past civilizations and natural landscape forms. In his small booklet, The Palace of Fine Arts and Lagoon, he writes: \u201cI find that the keynote of a Fine Arts Palace should be that of Sadness, modified by the feeling that beauty has a soothing influence.\u201d13

Eventually, in the decades after the Exposition, Maybeck\u2019s facsimile-ruination gave way to its less picturesque referent: by the early sixties the structure was physically crumbling\u2026 which didn\u2019t concern Maybeck much at all.

During the period when funds were being gathered for restoration, Maybeck had his own ideas concerning the Palace of Fine Arts. They ranged from demolishing it in order to create an active community center, to heavily planting its site with redwoods in order that children of the future might find bits of ornament and sculpture of a wondrous ruin of a previous generation among the trees.14

Earlier, in 1947, Maybeck suggested encapsulating a ruined cathedral in a transparent geodesic dome to serve as a memorial rather than rebuilding it:

Several week after the conclusion of Fuller\u2019s stay at the University [Buckminster Fuller in 1950], Annie Maybeck telephoned to report that Ben was most anxious to see me. She said \u201cIt has something to do with that man and his domes.\u201d When I arrived at Maybeck\u2019s studio, he had spread before him a copy of Life magazine showing Sir Basil Spence\u2019s proposal for the reconstruction of Coventry Cathedral as a war memorial. Maybeck was visibly agitated. \u201cLook, look,\u201d he said, \u201ca reconstructed Gothic cathedral is not what is needed; what should be done is to preserve the bombed ruins as a memorial.\u201d He then suggested that a large glazed geodesic dome covering the ruins would be the perfect solution and urged me to contact Fuller to initiate such a project.15

Like with the Palace of Fine Arts, the ruins in Coventry had an affect all their own: the walls suggest vaults, the tracery suggests glass, and so on. Rebuilding would mean losing a means of authentically signifying loss. If a memorial gestures at a historic void, its form should gesture at precisely the structure that was lost.

Losing Hearst Hall

It\u2019s surprisingly hard to find the precise date for the fire that destroyed Hearst Hall\u2014even Cardwell just gives the year\u2014perhaps because it\u2019s overshadowed by the resulting rush to commission Hearst Memorial Gym. Cal\u2019s 1923-1924 yearbook mentions the fire in a page anticipating the rebuild:

[Hearst Memorial Gym] has for several years been merely a dream, but at last our dreams have come true because Mr.\u00a0William Randolph Hearst has promised the women of the University a new Building. The other Hearst Hall was used as a place for gathering as well as for a gymnasium, but the new building will be used mainly for a gymnasium. The former hall was a gift from the late Mrs.\u00a0Phoebe Hearst to the University, but it was destroyed during a fire on June 20, 1922.16

The June 24, 1922 Daily Californian also dates the fire to June 20th. William Randolph Hearst promised a replacement, by telegram, the day after the blaze. \u201cIn fireproof materials.\u201d17 Hearst specifically requested Maybeck get the assignment.18

Contemporary Blue & Gold yearbooks are the best testament to Hearst Hall\u2019s adaptability. Plays, concerts, pep rallies, banquets, dances, insensitive \u201cItalian carnivals\u201d\u2026 the Hall was home to student events of all sorts.19 Cardwell calls it \u201cthe center of the campus social and cultural life;\u201d that seems apt.

Steven Finacom penned an article about the Hearst Hall fire on its hundredth anniversary, including a contemporary account of the firefighting response and a poetic description of the building.

Modernism

Cardwell works hard to differentiate Maybeck both from the Arts & Crafts tradition and from International Style modernism. Peeling Maybeck away from Arts & Crafts is definitely the harder task, but Cardwell approaches both the same way: he diagnoses phobias in both traditions and finds Maybeck projects that broke from them.

Even though much of Maybeck\u2019s early work received strong acclaim from people who were supporters of the ideas of John Ruskin and William Morris, he was not a proponent of a crafts revival. Maybeck started his designs on premises more sophisticated than those attributed to the Arts and Crafts movement. Exploitation of structural order, application of visual phenomena, and respect for new materials and industrial processes, as well as a firm belief in the expressive qualities of form were the bases for his work. A building was designed in the manner that Maybeck felt was appropriate for the conditions of the problem given. [\u2026] The unprecedented laminated wood arch framing of Hearst Hall was designed primarily to solve the problems of dismantling and reassembly and secondarily to provide a Gothic allusion of shape. The fact that Maybeck was ready either to adapt historic forms or to explore the boundaries of current construction technology went unnoticed by others advocating nature-inspired ornament and the infusion of handcrafts into the building process.20

Frankly, it\u2019s a stretch to argue Maybeck\u2019s buildings are just incidentally classics of California Arts & Crafts. Maybe one could argue that A&C was already an established Bay Area vernacular style, and so similarities in Maybeck\u2019s projects (like his monomaniacal respect for wood) are conventional. If he articulated it, this argument would seem a litte too convenient: Cardwell\u2019s happy to attribute just about everything else to Maybeck\u2019s genius.

Maybeck did make certain material decisions that\u2019d make an A&C purist wince (in one episode he orders industrial sash windows for a church, to the manufacturer\u2019s alarm). Nonetheless, Maybeck never strayed far from his crafts aesthetic: heavy redwood beams, corbels, and so on. All his technical innovation is subordinate, in his finished projects, to that aesthetic effect. That\u2019s precisely why Cardwell can\u2019t identify him with modernism: a true modernist would avoid the old, the romantic.

The immediate reaction was to look at Maybeck as a forerunner of the modern movement. It is true that some of his architectural innovations broke with the immediate past and used products of a machine technology in fresh and imaginative solutions to building problems. However, in the sense of \u201cout with the old order and in with the new,\u201d Maybeck was not a revolutionary. The architect\u2019s primary function, he reasoned, was to fulfill one of man\u2019s greatest yearnings\u2013\u2013to be surrounded by beauty. As an architect he strove to produce, with modern materials and techniques, an architecture as beautiful and meaningful at that which had been created in past times and for past cultures.21

Their respective phobias aside, I\u2019m not the first to claim there\u2019s an intersection between Arts & Crafts and modernism. Cardwell invokes Nikolaus Pevsner (Pioneers of Modern Design, 1949) to claim \u201cin architecture, particularly in the domestic field, [Arts & Crafts movement] architects produced designs with a basic simplicity and functionalism that have been identified as the beginnings of modern architecture.\u201d Some of their efforts are \u201ceuphemistically called \u2018rustic,\u2019\u201d but Maybeck\u2019s are simple and functional.22

The best argument I found (by far) is Lewis Mumford\u2019s, cited in passing. Pevsner worships Gropius; Mumford loathes Gropius, and absolutely roasts Gropius\u2019s imitators from his column in The New Yorker (1947). For Pevsner, Arts & Crafts posed modernism\u2019s original problem (the unity of structure and decoration); for Mumford, on the other hand, Maybeck\u2019s architecture softens modernism\u2019s hard edge, makes it less afraid of comfort.

Certainly Le Corbusier\u2019s dictum of the twenties\u2013\u2013that the modern house is a machine for living in\u2013\u2013has become old hat; the modern accent is on living, not on the machine. (This change must hit hardest those academic American modernists who imitated Le Corbusier and Mies van der Rohe and Gropius, as their fathers imitated the reigning lights of the Ecole des Beaux-Arts.) Mr.\u00a0Siegfried Giedion, once a leader of the mechanical rigorists, has come out for the monumental and the symbolic, and among the younger people an inclination to play with the \u201cfeeling\u201d elements in design\u2013\u2013with color, texture, even painting and sculpture\u2013\u2013has become insuppressible. \u201cFunctionalism,\u201d writes a rather pained critic in a recent issue of the Architectural Review of London, \u201cthe only real aesthetic faith to which the modern architect could lay claim in the inter-war years, is now, if not repudiated, certainly called into question \u2026 by those who were formerly its most illustrious supporters.\u201d

We are bound to hear more of this development during the next decade, but I am not alarmed by the prospect. What was called functionalism was a one-sided interpretation of function, and it was an interpretation that Louis Sullivan, who popularized the slogan \u201cForm follows function,\u201d never subscribed to. The rigorists placed the mechanical functionings of a building above its human functions; they neglected the feelings, the sentiments, and the interests of the person who was to occupy it. Instead of regarding engineering as a foundation for form, they treated it as an end.

[\u2026]

Well, it was time that some of our architects remembered the non-mechanical and non-formal elements in architecture, that they remembered what a building says as well as what it does. A house, as the Uruguayan architect Julio Vilamaj\u00f3 has put it, should be as personal as one\u2019s clothes and should fit the family life just as well. This is not a new doctrine in the United States. People like Bernhard Maybeck and William Wilson Wurster in California, always practiced it, and they took good care that their houses did not resemble factories or museums. So I don\u2019t propose to join the solemn gentelmen who, aware of this natural reaction against a sterile and abstract modernism, are predicting a return to the graceful sterotypes of the eighteenth century. Rather, I look for the continued spread, to every part of our country, of that native and humane form of modernism which one might call the Bay Region style, a free yet unobtrusive expression of the terrain, the climate, and the way of life on the Coast. That style took root about fifty years ago in Berkeley, California, in the early work of John Galen Howard and Maybeck, and by now, on the Coast, it is simply taken for granted; no one out there is foolish enough to imagine that there is any other proper way of building in our time. The style is actually a product of the meeting of Oriental and Occidental architectural traditions, and it is far more truly a universal style than the so-called international style of the nineteen-thirties, since it permits regional adaptations and modifications. Some of the best examples of this at once native and universal tradition are being built in New England. The change that is now going on in both Europe and America means only that modern architecture is past its adolescent period, with its quixotic purities, its awkward self-consciousness, its assertive dogmatism. The good young architects today are familiar enough with the machine and its products and processes to take them for granted, and so they are ready to relax and enjoy themselves a little. That will be better for all of us.23

I\u2019m surprised that Mumford never mentions Arts & Crafts, either as a name for what he calls \u201cthe Bay Region style\u201d or as a movement. Does he identify Maybeck with it? If not, does he share Cardwell\u2019s reasons\u2014that Maybeck wasn\u2019t puritanical enough about manufactured elements?

Wurster Hall

I haven\u2019t found anything suggesting a direct architectural influence between Hearst and Wurster Halls. This is a little disappointing, but to be expected\u2014Hearst Hall was gone thirty-five years before Wurster Hall was commissioned. Nonetheless, I\u2019m confident in two connections. The first is an indirect connection to that most-publicly-loathed feature of Wurster, its use of concrete. The second connection is direct: the architectural philosophy of William Wurster.

Maybeck frequently used the same materials for his projects\u2019 structures and their interior and exterior textures. For the most part, these were wood\u2014redwood when it was widely available. In at least one tight-budgeted project, the San Francisco Settlement Association\u2019s Boys\u2019 Club (1910), the outer walls and roof were left uncovered in the interior; two sides of the same boards and shingles constituted the outer and inner walls.24 Wurster\u2019s concrete interior is analogous to Hearst Hall\u2019s interior shakes: both buildings frankly reuse their structural materials.

Maybeck even worked with raw concrete interiors. The First Church of Christ, Scientist in Berkeley is highly ornamented, but its enormous carved beams rest on lightly-adorned concrete pillars. Much of the exterior is concrete, hidden as it may be behind quintessentially Maybeck trellises. Hearst Memorial Gym? Concrete. He surfaced his own studio (1 Maybeck Twin Dr, Berkeley) in burlap sacks dipped in the stuff, an experiment in quick construction, and built his son\u2019s house in the Berkeley hills out of pre-cast concrete slabs. \u201cbuilding was designed in the manner that Maybeck felt was appropriate for the conditions of the problem given.\u201d

Maybeck\u2019s bare-burlap-and-concrete studio in the Berkeley hills, now a City of Berkeley Landmark.25

There are strong echoes of Maybeck\u2019s philosophies in William Wurster\u2019s initial dreams for Wurster Hall\u2014most prominently, his dream of a building that will earn its character through use and adaptation.

\u201cI wanted it to look like a ruin that no regent would like\u2026 It\u2019s absolutely unfinished, uncouth, and brilliantly strong\u2026 The Ark [the preceding college of architecture], for instance, is a ripe building; it has been lived in; it\u2019s been used, it\u2019s been beaten up and everything else. It\u2019s arrived. Our building will take twenty years to arrive.\u201d26

Like the inside-and-out shingles of Maybeck\u2019s Boys\u2019 Club, Wurster Hall\u2019s concrete serves a practical purpose: economy.

The decision to construct it of concrete inside and out reflected both economic considerations and the aesthetic of the times. As often happens within the subculture of architecture, the architects were reacting to another building, the Yale School of Architecture, which had been designed by Paul Rudolph. [\u2026] For Wurster such design was anathema; he made his point loud and clear. \u201cI want you to design a ruin,\u201d he would say, pounding the table for emphasis. Wurster\u2019s idea of a \u201cruin\u201d was a building that achieved timelessness through freedom from stylistic quirks. While Wurster Hall has been categorized generally as a Brutalist design, the architects protest that the Brutalist aesthetic did not cause their preoccupation with consistency in the use of materials and forms.27

Exposing ductwork and piping was similarly economical.

One aspect of the design that has been generally misinterpreted is the exposure of the ductwork and of the other mechanical equipment. Far from being an expression of style, the exposure was a means of avoiding the tunnel-like corridors that a dropped ceiling could produce and to obtain the effect of a high ceiling in the rooms. [\u2026] Having the mechanical equipment exposed also made maintenance easier, and was useful, to some extent, in teaching.

Concessions as these might\u2019ve been, Wurster\u2019s architects hardly adhered to Le Corbusier\u2019s \u201cmachine for living\u201d stricture. The hall\u2019s courtyard is a sentimental feature, a callback (albeit an economized on) to the brick courtyard of the old John Galen Howard-designed architecture college (\u201cthe Ark,\u201d now North Gate Hall) Wurster Hall would replace.

[Wurster] and others also considered a courtyard imperative. One of the most cherished parts of the old Architecture building, called the Ark, was the brick-paved court where social and ceremonial occasions took place. The new court was intended to express continuity with the old setting, to be the symbolic heart of the new college building.28

This all seems like a Maybeckish mix of technical pragmatism and personality, albeit in the colder (and more fire-resistant!) concrete vernacular of the fifties. It\u2019s not whimsical, but it\u2019s honest; it was built to be lived in. Wurster\u2019s comment about the Regents is a little insouciant, and I\u2019m not sure Wurster Hall achieves \u201ctimelessness through freedom from stylistic quirks,\u201d but he\u2019s right about how to judge the building: what matters is not the Regents\u2019 kneejerk griping, but how the building \u201carrives\u201d via the life given it by educators and students decades down the line. As Maybeck puts it, \u201cyou could get your own personality into your house better if we stopped at the building proper.\u201d

On sources

My access to resources on Maybeck are more or less limited to what\u2019s available online\u2014Google Books and the Internet Archive have been indispensible\u2014and in the collections of the SF Public Library (which includes Cardwell\u2019s book and reference copies of vintage Blue and Gold yearbooks). The Berkeley Library Digital Collections have scans of Daily Cal papers dating back to 1898.

UC Berkeley holds several collections that might be of interest to someone who really wants to dive deep.

Berkeley\u2019s libraries have been closed because of COVID-19, and won\u2019t reopen until the fall.

One more text stands out, which deserves attention but doesn\u2019t fit neatly into this essay: Ursula K. Le Guin wrote an essay, \u201cLiving in a Work of Art\u201d, about growing up in Maybeck\u2019s Schneider house. She covers a lot of the same ground\u2014even borrows some of the same quotes from Cardwell, whose book she recommends\u2014but she tells of growing up in the house, really living in it, before she understood its architectural significance. \u201cI have trouble distinguishing the ethical from the aesthetic,\u201d she writes,

Is it not fair to say that every building has a morality, in this sense, and not merely a metaphorical one, in the honesty and integrity of its design and materials, or the dishonesty expressed in incompetence, incoherence, shoddiness, fakery, snobbery?

I think I absorbed this morality of the building as I did the smell of redwood or the sense of complex space.

I think the moral conception of the building was as admirable as its aesthetic conception, from which it is, to me, inseparable.29


  1. Cardwell, Kenneth H. Bernard Maybeck: artisan, architect, artist. Santa Barbara and Salt Lake City: Peregrine Smith, 1977. (48)\u21a9\ufe0e

  2. Cardwell, Bernard Maybeck, 46\u201347.\u21a9\ufe0e

  3. Cardwell, Bernard Maybeck, 86.\u21a9\ufe0e

  4. Cardwell, Bernard Maybeck, 86.\u21a9\ufe0e

  5. Cardwell, Bernard Maybeck, 47.\u21a9\ufe0e

  6. Cardwell, Bernard Maybeck, 48.\u21a9\ufe0e

  7. Cardwell, Bernard Maybeck, 49.\u21a9\ufe0e

  8. Wikimedia Commons contributors, \u201cFile:American homes and gardens (1905) (17963343698).jpg,\u201d Wikimedia Commons, the free media repository (accessed July 6, 2021).\u21a9\ufe0e

  9. Cardwell, Bernard Maybeck, 48.\u21a9\ufe0e

  10. Cardwell, Bernard Maybeck, 106.\u21a9\ufe0e

  11. Mumford, Lewis. \u201cThe Sky Line: Status Quo.\u201d The New Yorker, October 11, 1947. (104-110)\u21a9\ufe0e

  12. Cardwell, Bernard Maybeck, 189.\u21a9\ufe0e

  13. Cardwell, Bernard Maybeck, 144.

    Todd\u2019s picture of \u201ca cloister enclosing nothing, a colonnade without a roof, stairs that ended nowhere\u201d reminds me of a very abstract explanation of a cube with a missing octant from Peter Eisenman:

    Traditionally architecture, let\u2019s say the last 400 years of architecture, has been defined by what I call a humanist architecture\u2014that is, that man took the forms of his building and related them back, and gave them some meaning, by virtue of their relationship to some ideal, or more pure state. Something that was symmetrical, something that was contained, something that was whole. Something that was understandable as a more perfect condition than the reality of which man lived in.

    Now, my argument is that that ideal state that we can refer to and get meaning from no longer exists. Man\u2019s position relative to God and nature has changed. Man has unwittingly unleashed natural forces of terrible proportion which threatened to extinguish the civilization. We have no model of architecture which defines that condition that is the opposite of ideal. It is the negative of an ideal state. And I want to try and say, \u201calright, what would a design process be that starts with the negative of an ideal?\u201d

    Take these two objects. These are two partial cubes. They each have an octant cut out of them. They could be seen to be in the process of growing toward something ideal, or disappearing to a point; that is, they are in a state of instability. And I want to try and suspend the architecture between the former classical ideal notion of what architecture was, and this negative which may represent man\u2019s destiny today\u2014that is, the future which is, uh, nothing.

    I\u2019m not convined Eisenman\u2019s example. He calls his cubes, well, \u201cpartial cubes\u201d\u2014they, like referential humanism before them, certainly refer to some ideal (a platonic solid, no less!). I suppose Eisenman isn\u2019t arguing against all reference, but rather the inheritance of humanistic principles\u2026 but then the cube\u2019s missing octant seems superfluous: does the missing octant make it less humanistic or more? This seems to be of grave importance to Peter.

    In any case, Eisenman and Maybeck are using geometric form similarly to signify loss. We imagine the cube\u2019s missing octant, we imagine a roof on the PFA\u2019s colonnade; in both cases, we realize an absence. The rise of a staircase-to-nothing and the slant of a cube\u2019s edges towards the vertex opposite the void (Eisenman touches it to punctuate \u201cdisappearing to a point\u201d) function similarly: they imply a motion towards an end.

    For Eisenman\u2019s view of architecture and cubes, see Michael Blackwood\u2019s Beyond Utopia: Changing Attitudes in American Architecture (1983), 47:36\u201349:27. I found it on Kanopy.\u21a9\ufe0e

  14. Cardwell, Bernard Maybeck, 152.

    Cardwell mentions the October 16, 1915 \u201cPreservation Day\u201d and subsequent formation of a PFA preservation league, but plainly says they failed and that the Palace fell to \u201cforty-five years of disintegration and decay;\u201d (151) he skips, or misses, updates made in the thirties.

    As a part of the process of permanent rehabilitation of the Palace of Fine Arts, [painter Frank Joseph] Van Sloun was commissioned in the spring of 1936 to execute eight large murals in the ceiling of the octagonal rotunda [\u2026] His works are destined to be viewed by countless tourists, for the lagoon and lovely building are always sought by the sight-seer.

    Van Sloun\u2019s murals were destroyed in the Palace\u2019s 1962 demolition and reconstruction. The above quote is from California Art Research, Volume 8 (1937, pages 117-118), which describes them.

    Van Sloun\u2019s \u201cJupiter\u201d panel. Baird, Joseph A., Jr., photographer. Ceiling of Rotunda Dome, 1956. Photograph. From Wikimedia Commons, (accessed July 10, 2021).
    \u21a9\ufe0e
  15. Cardwell, Bernard Maybeck, 10. Spence\u2019s design did preserve the footprint of the old cathedral, which now encloses a courtyard; the new cathedral sits adjacent.\u21a9\ufe0e

  16. 1925 Blue & Gold: A Record of the College Year 1923-1924.\u00a0United States:\u00a0University of California,\u00a01924. (336)\u21a9\ufe0e

  17. \u201cW. R. Hearst to Rebuild Woman\u2019s Gymnasium Given by His Mother.\u201d The Daily Californian. June 24, 1922. (1, 6)\u21a9\ufe0e

  18. \u201cWill Choose Site for a Quadrangle of Women\u2019s Halls.\u201d The Daily Californian. June 27, 1922.

    This article mentions plans to develop a whole women\u2019s quad, and Robert Gordon Sproul (then Comptroller, later President of the University of California) reports that the Regents are \u201cextremely anxious, according to Mr.\u00a0Sproul [Robert Gordon, then Comptroller and later President of the University of California], to embark as soon as practicable upon a policy of supplying dormitories than women.\u201d

    Cardwell mentions that Maybeck had grand plans too (his early sketches included a complex of buildings\u2014not just a gym, but also a domed auditorium, museums, etc.). The University reined in Maybeck and only built the gym. Maybe the Regents were more anxious about the project\u2019s soonness than its scope\u2026\u21a9\ufe0e

  19. 1918 Blue & Gold: A Record of the College Year 1916-1917. \u00a0United States:\u00a0University of California,\u00a01917. (56) \u201cInsensitive \u2018Italian carnivals?\u2019\u201d

    The main floor was arranged with numerous booths, decorated with chili, garlic, and onions, in which artists and fortune-tellers were busily engaged. Italian peasant girls in bandanas, bright-colored aprons and skirts, sold balloons, flowers, and candy. Organ grinders, singers, tarantula and dagger dancers afforded entertainment, and in the gaily decorated annex there was dancing. Down-stairs, peasant girls sold fruit, spaghetti, raviolas, and tamales.

    Yes, tamales.\u21a9\ufe0e

  20. Cardwell, Bernard Maybeck, 83.\u21a9\ufe0e

  21. Cardwell, Bernard Maybeck, 236.\u21a9\ufe0e

  22. Cardwell, Bernard Maybeck, 84.

    I\u2019m a little skeptical of Cardwell\u2019s characterization. Pevsner seems to be praising C. R. Ashbee, and only because he renounced William Morris\u2019s \u201cintellectual Ludditism;\u201d \u201cthe true pioneers of the Modern Movement are those who from the outset stood for machine art.\u201d His description of Morris is blistering, though Morris gets some credit for emphasizing stylistic unity. See Pioneers of Modern Design: from William Morris to Walter Gropius, pages 24\u201326.\u21a9\ufe0e

  23. Mumford, Lewis. \u201cThe Sky Line: Status Quo.\u201d The New Yorker, October 11, 1947. (104-110)\u21a9\ufe0e

  24. Cardwell, Bernard Maybeck, 160.\u21a9\ufe0e

  25. Cardwell, Bernard Maybeck, 215.\u21a9\ufe0e

  26. Woodbridge, Sally. \u201cReflections on the Founding: Wurster Hall and The College of Environmental Design [Two Place Tales].\u201d Places 1, no. 4 (1984).\u21a9\ufe0e

  27. Woodbridge, Sally. \u201cReflections on the Founding: Wurster Hall and The College of Environmental Design [Two Place Tales].\u201d Places 1, no. 4 (1984).\u21a9\ufe0e

  28. Woodbridge, Sally. \u201cReflections on the Founding: Wurster Hall and The College of Environmental Design [Two Place Tales].\u201d Places 1, no. 4 (1984).\u21a9\ufe0e

  29. Le Guin, Ursula K. \u201cLiving in a Work of Art.\u201d Words Are My Matter: Writings About Life and Books, 2000-2016, with a Journal of a Writer\u2019s Week. Small Beer Press, 2016. (61\u201362)\u21a9\ufe0e

", "summary": "Extended notes on *Hearst Hall.* This essay reviews *Bernard Maybeck: Artisan, Architect, Artist* by rehashing old arguments with new evidence: extended excerpts from that text, from Sally Woodbridge's history of Wurster Hall, and zingers (\"quixotic purities...\") from Lewis Mumford.", "date_published": "2021-07-04T21:04:06+00:00" }, { "id": "./gen/subtraction-tests.html", "url": "https://lukasschwab.me/blog/./gen/subtraction-tests.html", "title": "Genres and Subtraction Tests", "content_html": " Genres and Subtraction Tests « blog

Genres and Subtraction Tests

The Subtraction Test isn\u2019t a mental model in itself, but it relates to one: genre. Given some corpus of texts, a genre is a pragmatic class of some members in that corpus\u2014that is, texts don\u2019t innately belong to a genre, but we organize them (a posteriori and inductively1) into learned genres because it\u2019s analytically useful to do so.

Genres are hazily, even personally, defined; Genre Theory aims to explain why some things belong and other things don\u2019t. Rick Altman imagines an illustrative argument about musicals:

A: I mean, what do you do with Elvis Presley films? You can hardly call them musicals.

B: Why not? They\u2019re loaded with songs and they\u2019ve got a narrative that ties the numbers together, don\u2019t they?

A: Yeah, I suppose. I guess you\u2019d have to call Fun in Acapulco a musical, but it\u2019s sure no Singin\u2019 in the Rain. Now there\u2019s a real musical.2

Altman goes on to decompose film genre features into syntactic features (structural elements, like that a Western involve a governmental frontier, whether that be the borders of a town or some intergalactic outer limit) and semantic features (characteristic symbols: dust and dusters for the Western, song and dance for the musical, and so on). I like this breakdown for film\u2014syntaxes link apparently distant texts\u2014but it doesn\u2019t generalize to the genre mental-model.

We can make Altman more generic by setting aside the feature categories (syntactic and semantic).

Suppose we have a genre G. The simplest definition for the genre would be to decompose it into a complete set of necessary features {f1,\u2006f2,\u2006\u2026fn}, where every feature must exist in a text T if it\u2019s to be included in the genre. That is, T\u2004\u2208\u2004G\u2004\u21d4\u2004{f1,\u2006f2,\u2006\u2026fn}\u2004\u2286\u2004the features of T.3 Using this to filter texts gives us the \u201cinclusive list,\u201d

an unwieldy list of texts corresponding to a simple, tautological definition of the genre (e.g., western=film that takes place in the American West, or musical=film with diegetic music).4

The inclusive list is unwieldy because it doesn\u2019t leave much room for nuance, e.g.\u00a0to say that casting Elvis makes something less characteristic of the musical genre. We can complicate the definitions by adding unnecessary features and weighting them,5 but that gives us unwieldy definitions and a false sense of precision.

When I say we learn genres inductively, I mean we learn the genre holistically: rather than accumulating a set of atomic features {f1,\u2006f2,\u2006\u2026fn} that characterize a cluster of texts, we learn a genre duck test: \u201cif it looks like a [musical] and quacks like a [musical], it\u2019s a [musical].\u201d Formally, a test function duckG(T)\u2004\u2192\u2004{true,\u2006false} returns true if and only if the text T belongs to the genre G.

A Subtraction Test (my invented term) is a way of revealing relationships between particular features of a text and the genres to which the text belongs.

Imagine decomposing a text T into its features: T\u2004=\u2004{t1,\u2006t2,\u2006\u2026tn}. Subtraction-testing means comparing duckG(T) and duckG(T\u2005\\\u2005{ti}): does removing the feature ti change whether the text is a member of G?

duckG(T) duckG(T\u2005\\\u2005{ti}) Implication for G Example
true true ti is not necessary. High Noon without revolvers is still a Western.
\u2234 a Western needn\u2019t include revolvers.
true false ti may be necessary. Annie without musical numbers is no longer a musical.
\u2234 a Musical may need musical numbers.
false true \u00acti may be sufficient. I don\u2019t know, can you think of one?
false false \u00acti is not sufficient. No Country for Old Men without milk is still not a musical.
\u2234 milklessness does not a Musical make.

Note how the implications are qualified: subtraction tests are confident when subtracting a feature does not impact a text\u2019s classification. When the classification changes\u2014either from true\u2004\u2192\u2004false (indicating necessity) or from false\u2004\u2192\u2004true (sufficiency)\u2014we learn something about the directional influence of the feature but can\u2019t make too strong a claim. Maybe you don\u2019t think Star Wars is a Western, but you imagine it\u2019d pass if it didn\u2019t have the space travel; that means you think space travel makes a film less Western-y, but it obviously doesn\u2019t mean every space-travel-free film is a Western: Annie ain\u2019t.

Notably, this works for multi-feature sets as well as single features: we can check duckG(T\u2005\\\u2005{ti,\u2006tj}). Nothing guarantees that our features contribute independently to genre membership\u2014rolling papers signify differently in front of a Bob Marley poster and in a gumshoe\u2019s breast pocket\u2014but I rarely think about independence. If you\u2019re designing Taguchi experiments, either you\u2019re deep in the realm of false precision or your inductive categories are already sophisticated.

I\u2019ve been leaning on films because they\u2019re popularly associated with genre, but the model and test are portable to other informal clustering activities. Do you cluster inbound support tickets to prioritize product improvements? Pinning down litmus tests for whether a new ticket belongs in a certain cluster means you can share the task with a person who would naturally cluster differently.

We learn duck tests for informal categories all the time; Subtraction Tests inch usefully towards formalism.

Caution: not everything that looks and quacks like a Subtraction Test is one. I\u2019m only writing any of this down because Gilman\u2019s The Theory of Gothic Architecture and the Effect of Shellfire at Rheims and Soissons6 falsely reminded me of Altman\u2019s Elvis example. Gilman, writing in 1920, discusses gothic cathedrals damaged in World War I. The War did the subtraction for him, so he needn\u2019t imagine structures without features: where artillery removevd a particular feature, Gilman considers whether the damage cascaded (e.g.\u00a0in a collapse), which would indicate that the feature was structurally necessary.

Structurally unnecessary features, he reasons, are expressions of a gothic architectural ideal unmuddled by the cathedral\u2019s physical pragmatics. The style is purest-expressed in the ornamentatal, and Gilman\u2019s contemporaries debated whether gothic cathedrals revealed their true structures, which might\u2019ve otherwise been hidden, or whether the cathedrals were ornamented with faux-structure\u2014piers that support nothing, decorative buttresses, and so on. Gilman concludes the cathedrals are sometimes honest\u2014

Closely connected with the principle of logic is that of revelation of structure for esthetic effect. Here again the ruins generally illustrate the theory and confirm it in its broad application. In the vaulting, the manner in which the breakage has occured, as well as its extent, does confirm strikingly the theory that the ribs carry the vault cells and are not only real functioning members, but are the most important part of the vault.7

\u2014and sometimes not:

Now actually we find that some of the cases often considered as revealed structure are not the real structure. At Soissons the single shaft on the lower story of the nave piers does not really support the vaulting shafts and, in turn, the vaults above [\u2026] It by no means follows that the shaft is not good design, only that it does not express the facts of this structure; it is in fact an eye-satisfying fiction, although perhaps, a proper one.8

Left: \u201cThe Church at Nettancourt.\u201d Right: \u201cShell Holes in Vaults: Rheims.\u201d Gilman\u2019s evidence for the structural necessity of vault ribs. He found vaults with just the ribs intact, but none where the cells outlasted the ribs.9

The first sign that Gilman\u2019s treatment of subtraction and genre differs from our Subtraction Tests is that subtractability leads Gilman to the opposite conclusion. In the damaged cathedral, that \u201cthe single shaft on the lower story of the nave piers does not really support the vaulting shafts\u201d\u2014that those single shafts are subtractible in Gilman\u2019s sense\u2014means we should consider them a purer reflection of a gothic architect\u2019s ideal. That\u2019s very different from concluding \u201cti [the lower story of the nave piers] is not necessary\u201d for a building to be gothic; that may be true (a feature can be characteristic without being necessary), but seeing a cathedral sans piers doesn\u2019t give us evidence to that effect. It doesn\u2019t even mean the feature is insignificant to the genre: that an architect chose a nonstructural pier rather than a nonstructural doric column makes the building more gothic, not less.

The key difference is that Gilman isn\u2019t applying a duck test. When he sees a partially destroyed vault, he asks not \u201cis this still gothic?,\u201d but \u201cis this still standing?\u201d He isn\u2019t comparing whether the building\u2019s genre status changes, and he gives no reason to believe it does (a building may go from \u201cgothic cathedral\u201d to \u201cgothic semi-ruins,\u201d but nobody thinks it stops being gothic).

Gilman\u2019s subtractions tell us about architectural intent\u2014choices to build structurally unnecessary elements, despite their difficulty and expense\u2014not about necessity and sufficiency to his architectural category. Be careful that your duck test is, in fact, a duck test.

Is \u201cgothic\u201d a genre? The category certainly developed a posteriori and inductively. Its builders were aware of their style, but I haven\u2019t read any indication that they were self-awarely participating in a break from the sacred architecture that came before them. To the contrary,

The architect who built the choir of St Denis must have spoken to Suger about the arcus in the vaults; William of Sens in speaking to the Abbot of Canterbury no doubt used the term fornices arcutae, and Villard de Honnecourt probably spoke to his apprentices of Ogives. However, no name for the style itself is known to have existed at this time; indeed, it is unlikely that any name did exist, for, in the regions where the Gothic style was born and developed, \u2018building in the Gothic style\u2019 was simply called building.10

\u201cGothic\u201d was only identified as a genre, with common architectural elements and decorative tendencies, in retrospect (typical: we have to find noncontemporary names for yesterday\u2019s \u201cmodern\u201d):

When in the course of the nineteenth century historical knowledge of the birth of the Gothic style, its development, and its spread increased, the Late Gothic style was still regarded as part of the Gothic style. Moreover, the growing study of the essential Gothic elements, beginning with Wetter\u2019s book, and the attempts to interpret the essence of the Gothic, which began with the work of Viollet-le-Duc, and have continued to our own day, have neutralized the effects of personal taste and have led to a more intensive analysis of the concept of Gothic style.

The scholarly consideration of the Gothic style began with descriptions of individual buildings and so came to the study of individual members, such as pointed arches, piers, rib-vaults, windows, doorways, roofs, and towers. At the same time a desire grew to understand the essence of the Gothic style which could permeate such divergent features as piers and windows. The Gothic style was said to possess a picturesque quality, a quality of infinity, a vegetal quality, a romantic quality. All these different concepts were first formulated in the eighteenth century and were then considered more closely in the nineteenth and systematically bound into one unified concept.11

Interestingly, the genre-awareness of 19th-century restorationism means \u201crestoration\u201d may be a misnomer for what Viollet-le-Duc went about doing. Viollet-le-Duc was widely criticized for intervening ahistorically in the buildings he restored, emboldened by his conclusions about the driving principles of Gothic style, including scrubbing them of historically authentic, if post-gothic, features. Perhaps the very notion of a unified principle is fraught: even the individual cathedrals in Voillet-le-Duc\u2019s canon were constructed over the courses of centuries. Should we believe these principles stayed fixed over several generations of builders?

Incidentally, Frankl also offers a better term than Gilman for style expressed in the structurally unnecessary, a \u201cmargin of freedom:\u201d

In other words, function always leaves a \u2018margin of freedom\u2019 which cannot be resolved by utilitarian considerations and which is not objectively pre-determined. [\u2026] Gothic vaults must be examined in the light of their function; [\u2026] even when these functions have been understood, the question as to why these mebers have so many different profiles in different churches still remains unanswered, since each profile fulfils the necessary static function, in so far as it exists equally well.12


  1. Olsen, Stein Haugom. \u201cThe concept of literary genre.\u201d Metodo. International Studies in Phenomenology and Philosophy 6.1 (2018).\u21a9\ufe0e

  2. Altman, Rick. \u201cA semantic/syntactic approach to film genre.\u201d Cinema Journal (1984): 6-18.\u21a9\ufe0e

  3. This formulation is inspired by Olsen. See especially page 56.

    I\u2019ve adjusted the notation, and I think Olsen\u2019s direction is a little misleading: theorizing about weighted features assumes we sort texts into genres in a really precise way. In practice, when you ask yourself \u201cis Fun in Acapulco a musical?,\u201d you\u2019re just applying a litmus test rather than weighting and summing.

    Extracting weights would be an interesting approach if we wanted to take a corpus and learn features that predicted membership, but that means starting with a corpus and working towards a genre definition, which isn\u2019t what Olsen\u2019s discussing.\u21a9\ufe0e

  4. Altman, Rick. \u201cA semantic/syntactic approach to film genre.\u201d Cinema Journal (1984): 6-18.\u21a9\ufe0e

  5. Olsen, Stein Haugom. \u201cThe concept of literary genre.\u201d Metodo. International Studies in Phenomenology and Philosophy 6.1 (2018).\u21a9\ufe0e

  6. Gilman, Roger. \u201cThe theory of gothic architecture and the effect of shellfire at Rheims and Soissons.\u201d American Journal of Archaeology 24, no. 1 (1920): 37-72.\u21a9\ufe0e

  7. Gilman, Roger. \u201cThe theory of gothic architecture and the effect of shellfire at Rheims and Soissons.\u201d American Journal of Archaeology 24, no. 1 (1920): 37-72.\u21a9\ufe0e

  8. Gilman, Roger. \u201cThe theory of gothic architecture and the effect of shellfire at Rheims and Soissons.\u201d American Journal of Archaeology 24, no. 1 (1920): 37-72.\u21a9\ufe0e

  9. Gilman, Roger. \u201cThe theory of gothic architecture and the effect of shellfire at Rheims and Soissons.\u201d American Journal of Archaeology 24, no. 1 (1920): 37-72.\u21a9\ufe0e

  10. Frankl, Paul, and Paul Crossley. Gothic architecture. Vol. 58. Yale University Press, 2000. 217-219.\u21a9\ufe0e

  11. Frankl, Paul, and Paul Crossley. Gothic architecture. Vol. 58. Yale University Press, 2000. 217-219.\u21a9\ufe0e

  12. Frankl, Paul, and Paul Crossley. Gothic architecture. Vol. 58. Yale University Press, 2000. 228.\u21a9\ufe0e

", "summary": "Discusses 'genre' as a generic model for informally learned categories, and a test for making them more formal by using a duck test to learn if a feature is necessary or sufficient for membership in a category.", "date_published": "2021-06-13T16:58:51+00:00" }, { "id": "./gen/blogging-with-tikz.html", "url": "https://lukasschwab.me/blog/./gen/blogging-with-tikz.html", "title": "Blogging with TikZ", "content_html": " Blogging with TikZ « blog

Blogging with TikZ

Programming Langauges and Compilers (CS 164) with Paul Hilfinger was the hardest class I took in college, bar none. I recommend it to anyone studying CS at Cal: the correspondence between formal grammars and state machines-as-parsers has been pretty personally impactful. State machine diagrams were an indispensible part of Hilfinger\u2019s notes on those subjects, and drawing out a state machine diagram to tap out the state transitions is how I recall those notes now.

State machine diagrams are indespensible to State Machines via Jorge Luis Borges because, without all my mnemonic drawing and tapping, I wouldn\u2019t have recognized the connection between text and theory. The diagrams are also an indespensable way to concretize the examples in that post. I\u2019m not prepared to explain (non)determinism without converting a specific graph, and I don\u2019t have the patience to squeeze all ambiguity out of mock-Borges prose.

Diagramming also presented an opportunity to go further than the CS 164 notes: rather than just presenting diagrams and counting on readers to understand the rules governing state transitions, I wanted the diagrams to demonstrate those rules by illustrating dynamic user input.

I did all of this with TikZ, a LaTeX extension for specifying diagrams (probably the same extension Hilfinger used). This might seem like an odd choice for interactive demos, but there\u2019s a logic behind the bodge!

Arriving at TikZ

pandoc-blog, the static site generator I use, is full of opinions. It uses Pandoc to convert Markdown and LaTeX source to HTML documents in order to

State Machines imposed particular requirements. All of my diagrams would have a great deal in common; they should represent states and transitions, with labels, in a consistent style. Given that similarity, I wanted a tool that\u2019d let me define the state machines as minimally as possible. Each diagram definition should be easy to read and version.

Finally, I needed something I could make interactive: node appearances need to change according to user input.

Avoiding big dependencies and preferring print-friendliness meant eschewing Javascript libraries for drawing graphs, even though a JS-controlled canvas element would probably serve well for interactivity.2

That interactivity requirement, along with the preferences for semantic markup and for concise and versionable definitions, rules out tools for drawing graphs that only output raster images.

HTML canvas and svg elements are natural fits for the technical requirements. They\u2019re lightweight, reasonably print-friendly, and semantic (which, in turn, means they\u2019re scriptable for interactivity). The trouble is SVG is just too flexible\u2014I want to be quickly defining labeled states and edges rather than individually arranging SVG elements.

That\u2019s where TikZ comes in: it\u2019s concise.3 The problem with TikZ is that it\u2019s a LaTeX extension meant for generating PostScript; I need it to generate SVG.

Implementation

Converting TikZ to SVG

Defining a state machine diagram in TikZ is fairly straightforward:

  1. Define a set of states, with their positions, labels, and IDs.
  2. Define a set of edges, with labels, between states.

These steps are just a matter of using styles and macros provided by the tikz-automata TikZ library; for an in-depth explanation, check out Drawing Finite State Machines in LaTeX using tikz. In practice, a TikZ state machine diagram is defined like this:

\\begin{tikzpicture}\n  % 1. Define the states, with their positions, labels, and IDs.\n  \\node[state, accepting, initial] (Rc) {$C$};\n  \\node[state, below=of Rc] (Rb) {$B$};\n  \\node[state, right=of Rc, yshift=-2cm] (Ra) {$A$};\n  % 2. Define the edges, with labels, between states.\n  \\draw\n    (Rc) edge[bend right, left] node{1} (Rb)\n    (Rb) edge[bend right, right] node{0} (Rc)\n    (Rb) edge[bend right, below] node{0} (Ra)\n    (Ra) edge[loop right, right] node{1} (Ra)\n    (Ra) edge[bend right, above] node{1} (Rc);\n\\end{tikzpicture}

This syntax isn\u2019t perfect\u2014in particular, I found positioning nodes pretty fiddly\u2014but it\u2019s concise (each node and edge on its own line) and readable (as readable as the labels and IDs one chooses).

Normally this tikzpicture would be embedded into a LaTeX document; how can it be turned into a standalone SVG? There\u2019s an answer on the TeX Stack Exchange:

  1. Use the standalone TeX package to generate a PDF of just the TikZ figure, without any other document cruft.

  2. Use pdf2svg to convert that PDF to an SVG file. Miraculously\u2014I have no idea how this works\u2014that SVG file is reasonably human-readable, if verbose: each element in the PDF is converted to an SVG path element.

Using standalone is as simple as wrapping our tikzpicture in a document with the standalone class. Doing so (and adding some TikZ style directives to the preamble) gives us our final definition of our figure:

% This preamble is reused for every diagram.\n\\documentclass{standalone}\n\\usepackage{xcolor}\n\\usepackage{tikz}\n\\usetikzlibrary{automata, positioning, arrows}\n\\tikzstyle{defaults}=[\n  ->,\n  >=stealth',\n  every state/.style={thick, fill=gray!10},\n]\n\n\\begin{document}\n  \\nopagecolor\n  % The same tikzpicture as before.\n  \\begin{tikzpicture}\n    [defaults, node distance = 3cm]\n    \\node[state, accepting, initial] (Rc) {$C$};\n    \\node[state, below=of Rc] (Rb) {$B$};\n    \\node[state, right=of Rc, yshift=-2cm] (Ra) {$A$};\n    \\draw\n      (Rc) edge[bend right, left] node{1} (Rb)\n      (Rb) edge[bend right, right] node{0} (Rc)\n      (Rb) edge[bend right, below] node{0} (Ra)\n      (Ra) edge[loop right, right] node{1} (Ra)\n      (Ra) edge[bend right, above] node{1} (Rc);\n  \\end{tikzpicture}\n\\end{document}

I wrote a short script, tikz2svg, which handles the TikZ \u2192 PDF \u2192 SVG conversion. Running tikz2svg nfa-conversion-pre.tex yields the SVG we\u2019re after:

nfa-conversion-pre.svg: the state machine diagram produced by tikz2svg nfa-conversion-pre.tex.

Building images in pandoc-blog

pandoc-blog\u2019s build processes are managed with make: if one modifies a post\u2019s Markdown source, the gen/%.html target rebuilds that post\u2019s generated HTML document.

Converting .tex precursors to .svg images manually is a hassle. This reconversion should happen automatically, the same way Markdown is reconverted to HTML: whenever a tex precursor is modified, rebuild the corresponding SVG.

Doing this for State Machines involved adding one Makefile and modifying another. The relevant directories are structured like this:

Most of this is typical of pandoc-blog:

What\u2019s atypical is the inclusion of our precursor (nfa-conversion-pre.tex) and its build product (nfa-conversion-pre.svg) in blog/img/borges-automata: generated content outside of blog/gen!

blog/img/borges-automata/Makefile builds all the diagrams in blog/img/borges-automata:

SOURCES = $(wildcard *.tex)\nTARGETS = $(patsubst %.tex,%.svg,$(SOURCES))\n\nall: $(TARGETS)\n\n%.svg: %.tex\n    tikz2svg $<\n\n.PHONY: clean\nclean:\n    rm *.svg

You can imagine any blog/img/*/Makefile exposing the same interface. The main blog/Makefile can delegate figure-generation to the img subdirectory for each blog post; this one happens to be TikZ-specific, but for future blog posts I can write image-generating Makefiles that encapsulate whatever other behavior I need.

We can update blog/Makefile to run make in each blog post\u2019s subdirectory:

POSTS=$(shell find posts/*)\n# OUT contains all names of static HTML targets corresponding to markdown files\n# in the posts directory.\nOUT=$(patsubst posts/%.md, gen/%.html, $(POSTS))\n# IMAGEDIRS is a list of img/ subdirectories containing makefiles.\nIMAGEDIRS=$(shell find img/*/Makefile | xargs -0 -n1 dirname)\n\nall: $(OUT) $(IMAGEDIRS) index.html\n\n# ... most targets unchanged from pandoc-blog/Makefile.\n\n# Runs `make` in the target directory.\nimg/%: img/%/*.tex img/%/Makefile\n    $(MAKE) -C "$@"\n    @touch "$@"

Now the make watch loop provided by blog/Makefile automatically rebuilds the diagrams whenever we save a change to the TikZ source!

This isn\u2019t a super novel way of composing Makefiles, but it\u2019s a fun complication of pandoc-blog. I didn\u2019t originally expect to be generating images, and I\u2019m pleased the root Makefile isn\u2019t sullied with blog-post-specific targets.

Adding interactivity

Diagrams are well and good, but you can\u2019t build an interactive state machine demo without, well, implementing a state machine. Generically, this means turning a topology (all the various states and transitions between them) and initial state into some object that exposes an interface for feeding it input.

I do this with an Automaton class, which I won\u2019t discuss in depth. The key to an Automaton\u2019s connection to an SVG diagram is buried in the nodes constructor argument: each state is defined with two callback functions, enter and exit, which are called when input causes the Automaton to enter and leave that state, respectively. When it enters state A, it calls A.enter(); when it receives any subsequent input, it calls A.exit() before proceeding.

To represent the Automaton\u2019s state in the diagram, A.enter() should turn the SVG element corresponding to A blue; A.exit() should reverse the change. All we need is a reference to that SVG element.

The problem: those SVG elements were defined by pdf2svg. Some of them have automatically-assigned IDs, but (for some reason) the circular paths demarcating states do not. I found the right paths with Chrome\u2019s dev tools and added ids manually to match the TikZ IDs:

<path\n  id="R0"\n  style="fill-rule:nonzero;fill:rgb(94.999695%,94.999695%,94.999695%);fill-opacity:1;stroke-width:0.79701;stroke-linecap:butt;stroke-linejoin:miter;stroke:rgb(0%,0%,0%);stroke-opacity:1;stroke-miterlimit:10;"\n  d="M 123.198937 42.518875 C 123.198937 49.397781 117.620812 54.972 110.745812 54.972 C 103.866906 54.972 98.292687 49.397781 98.292687 42.518875 C 98.292687 35.643875 103.866906 30.06575 110.745812 30.06575 C 117.620812 30.06575 123.198937 35.643875 123.198937 42.518875 Z M 123.198937 42.518875 "\n  transform="matrix(1,0,0,-1,53.137,83.722)"\n/>

Which leaves us to define enter and exit:

const enter = () => {\n  document.getElementById("R0").style.fill = "cyan";\n}\nconst exit = () => {\n  // The original fill is a light grey.\n  document.getElementById("R0").style.fill = "rgb(94.999695%,94.999695%,94.999695%)";\n}

\u2026and so on for each node in the graph; see the deterministic-binary-fsa.html source for a full example of a modified SVG and Automaton configuration bundled into an HTML document.

The final demo\u2014of a nondeterministic finite state machine handling input alongside its deterministic equivalent\u2014introduces a last complication. There\u2019s no structural issue with plopping two generated SVGs into an HTML document side-by-side, but there\u2019s a semantic one: ID collisions.

SVG\u2019s <use> element lets you duplicate another element by its ID. This is a neat little optimization: if you have a complex path you need to use twice, better to define it once and reuse it by reference! pdf2svg seizes on this optimization. The form of the digit 1 is only defined once; all the other 1s are defined, in source, by <use> elements referencing the original. The browser looks up the original element by its ID and re-renders it.

Remember how some of the pdf2svg-generated elements have IDs? Those are of the form glyph0-0, glyph0-1, and so on, which means that every pdf2svg-generated SVG has (and perhaps uses) a glyph0-0.

This wouldn\u2019t be an issue but for the way we\u2019re combining them: plopping the SVG source unceremoniously into an HTML document. When the browser looks up the DFA\u2019s glyph0-0 by reference\u2026 it finds the NFA\u2019s glyph0-0 instead. The first diagram is fine; the second one is a garbled mess of glyphs from the first.

The solution: prefix each SVG\u2019s IDs. Find-and-replace is robust enough:

  1. Replace all instances of id=\" with id=\"{prefix}.
  2. Replace all instances of \"# with \"#{prefix}.

I\u2019ve since updated tikz2svg to do this ID replacement automatically; now it prefixes generated SVG element IDs with the source filename. If you want to reuse filenames, insert a random component instead.

The Javascript animating this compound diagram is somewhat more complicated: it instantiates two Automatons, and feeds inputs to them simultaneously. See the nfa-conversion.html source for details.

Future work

Ideally, I\u2019d like to have a script to convert a TikZ state machine definition directly to an encapsulated demo.

  1. TikZ itself is an academic diagramming syntax; it offers a great deal of flexibility that must be excluded if the script is to extract a working state machine definition (that is, a topology).

  2. The multi-step process for producing an SVG\u2014renderinga PDF, then running that through pdf2svg\u2014means TikZ-specified data like state IDs are missing from the generated SVG. Remember: we re-added these IDs manually.

These issues are solved by avoiding TikZ, and instead building diagrams from some other domain-specific format. Javascript representations of the automaton topologies, like those configuring the interactive demos, would work. On the other hand, that means resigning TikZ\u2019s advantages. Honestly? I don\u2019t think I\u2019ll bother!

I\u2019d like to make some other incidental improvements (avoiding embedded iframes; referencing a single LaTeX header file between diagrams; things like that).

For now, I have what I need. Blogging with TikZ allowed me to avoid big dependencies, to keep my blog print-friendly, to keep its markup semantic, and to share interactive demos with the next generation terrorized by CS 164.

If my commitment to the ideological purity of my Pandoc-generated blog seems extreme, check out phiresky\u2019s \u201cHosting SQLite databases on GitHub Pages\u201d.

That blog\u2019s written as Pandoc Markdown; the Pandoc artifact is postprocessed with React to generate charts and runnable code blocks (including queries of a statically-hosted SQL database). That\u2019s commitment.


  1. Relevant: Maciej Ceg\u0142owski\u2019s evergreen slides on Chickenshit Minimalism. I\u2019m proud to keep this site hard-to-read without any help from npm.\u21a9\ufe0e

  2. Graph-drawing libraries are smaller than I expected, but not so small. The entirety of State Machines (complete with its diagrams and demos) is about the same size as minified d3.js: 94.5 and 83.6 kB, respectively.\u21a9\ufe0e

  3. There was another draw to TikZ which I didn\u2019t ever ultimately achieve: I figured that, for the static diagrams, I could define TikZ diagrams inline (in a LaTeX environment) in my post\u2019s Markdown source. I still think this is possible, but it\u2019d take writing a custom Pandoc extension to extract the TikZ picture, convert it to an SVG, and replace it with an HTML img referencing that SVG.

    I decided it\u2019d take too much fiddling to make that Pandoc extension reusable (especially outside of pandoc-blog), and that I might not actually have reason to reuse it.\u21a9\ufe0e

", "summary": "Notes on using TikZ, a LaTeX extension for defining vector graphics, to create static diagrams and interactive demos for *State Machines via Jorge Luis Borges.* These notes cover my motivations, my tools for generating SVGs as I wrote, and how I modified the script-generated diagrams for interactivity.", "date_published": "2021-05-21T02:44:41.018000+00:00" }, { "id": "./gen/borges-automata.html", "url": "https://lukasschwab.me/blog/./gen/borges-automata.html", "title": "State Machines via Jorge Luis Borges", "content_html": " State Machines via Jorge Luis Borges « blog

State Machines via Jorge Luis Borges

This post is, first and foremost, an introduction to state machines (also called \u201cautomata\u201d) via an extended Borgean allegory about an existence in a labyrinth, a sort of infinite gallery of mazes. It introduces several significant concepts\u2014equivalence, finitude, acceptance, determinism, and the powerset construction\u2014as a succession of realizations about that existence.

Why Borges? I picked up Labyrinths in early February, an anthology and a maze of furtive interreferences in its own right. Borges toys with combinatorics. He writes that \u201cLully\u2019s machine, Mill\u2019s fear and Lasswitz\u2019s chaotic library can be the subject of jokes, but they exaggerate a propension which is common: making metaphysics and the arts into a kind of play with combinations.\u201d He joins them, even reconstructing Lasswitz\u2019s library in The Library of Babel. The stories and essays are replete with villas subdivided into infinitely many rooms, repetitions collapsed into single events, points that contain all points, and so on.

I realize adding the allegory risks muddling the subject matter. My hope is that the metaphysical confusion of the labyrinth is relatable in a way that makes the mathematics feel more familiar. If I get away with that binding, it\u2019ll work in reverse: if the math resembles familiar philosophy, we can use the mathematical concepts as a grammar for their philosophical doubles.

I\u2019ve written another post about generating these diagrams and interactives with TikZ.


All the parts of the house are repeated many times, any place is another place. There is no one pool, courtyard, drinking trough, manger; the mangers, drinking troughs, courtyards, pools are fourteen (infinite) in number. The house is the same size as the world; or rather, it is the world.

The House of Asterion1

Imagine, if you will\u2014though you can relate to this freely if it sounds familiar\u2014that you live your life in a vast labyrinth of circular rooms, with marked doors that open into passageways which bring you from one room to the next. Each room is indistinguishable, except maybe by the number and labels of its exits. You\u2019re so disoriented that you can\u2019t know how you entered the room you\u2019re in. In fact, there may be no way to return to the room from whence you came\u2014the passages seem to close up behind you as you step out of them. There\u2019s no backtracking; you can only progress into the labyrinth, choosing among the doors each room affords you.

In your heart you know this is not a labyrinth one leaves. The labyrinth is the same size as the world; or rather, it is the world.

In one of the maze\u2019s innumerable connected rooms you\u2019re visited by the whisper of a blind sorcerer. The sorcerer explains he built the labyrinth\u2014designed its general schematic, its endless succession of identical rooms, the workings of its corridors, the labels on its innumerable doors, etc.\u2014but, blind, he can\u2019t read the signs, can\u2019t travel the monotonous expanse of corridors and galleries he devised.

This is the sorcerer\u2019s proposition: starting in some fixed room in the maze, you are to travel through a series of doors, following exactly his instructions for which door to select. If, at any point, he gives you an instruction you can\u2019t follow\u2014gives you some label that does not match any door out of the room you\u2019re in\u2014you can cry out, and (zap!) he will teleport you back to the starting room to receive a new set of instructions. When a set of instructions can be followed to the end, you are (zap!) also teleported back.

In this way, with you as a proxy, the sorcerer explores the great maze. In return, he promises, the many forms of the labyrinth and the meaning of your hermitude will be revealed.

Naturally curious, you accept.

This assemblage\u2014you in your maze, following instructions\u2014is essentially a state machine:

  1. Each room is a state; you can be in one state at a time.

  2. The fixed topology of the labyrinth determines which state transitions are allowed. A given room does not necessarily allow passage into every other room in the labyrinth; you can only move to a successive state available from your present state.

  3. Your task is to discover whether the sorcerer\u2019s inputs can be followed, whether they\u2019re acceptable to the state machine.2

Equivalence

The universe (which others call the Library) is composed of an indefinite and perhaps infinite number of hexagonal galleries, with vast air shafts between, surrounded by very low railings. From any of the hexagons one can see, interminably, the upper and lower floors. The distribution of the galleries is invariable.

The Library of Babel

The first room has two doors: one marked left, the other marked right. The sorcerer\u2019s first instruction, left, takes you through that left door and its narrow passageway, into another room with two doors: one marked left, the other marked right.

In fact, every door\u2014left or right\u2014opens on another indistinguishable room with identical doors; there is always a left door and a right door to pass through. Each foray ends either with some unfollowable instruction (e.g.\u00a0to take a door marked south, or salad bar, or de Broglie: none of these doors is ever available) or with the exhaustion of the sorcerer\u2019s sequence, at which point (zap!) you begin, with a new sequence, from the start. A lifetime of lefts and rights passes. The sorcerer\u2019s instructions are inexhaustible; some sequences take eons to follow.

In fact, if it\u2019s constructed how it appears\u2014each room leading to two more rooms\u2014this labyrinth must fork infinitely towards the horizon, right? And, since only one set of left-right instruction sequences leads you to any one of the infinite rooms, the sorcerer has a literally inexhaustible list of routes to explore: one route for each room.3 We can diagram the start of this maze as follows:

A Garden of Forking Paths. Each room has two doors\u2014one marked left, one marked right\u2014each of which leads to a room with two doors, and so on ad infinitum. The ellipses at the right of this diagram indicate continuation. In fact, at each ellipsis begins a series of forks exactly like the one pictured.

While it explains what you observe of the maze, this seems a bleak existential outlook\u2026 but you\u2019re rightfully suspicious of existential certitudes! In fact, out of the mire of indistinguishable rooms and doors and passageways surfaces a suspicion about the maze itself.

The sorcerer\u2019s instructions seem farsical. Predictably, choosing the left door or the right door makes no difference: they lead to indistinguishable rooms. What if the left and right doors mark, in truth, two passageways into the one room?

A Braid. Maybe this is our maze: each pair of doors leads to the same room, which has its own pair of doors, and so on and so forth.

Here, it seems, there are fewer rooms to see. Whereas before there was one room for every acceptable sequence of instructions, now there is only one room for every length of acceptable instructions. Any sequence of three instructions\u2014be it left-left-right, right-right-right, or some other\u2014leads you to the third room. There are precisely 2n paths to the nth room.

This does little to relax the absurdity of your agreement with the sorcerer. First of all, no sequence of sorcerers\u2019 instructions will ever disambiguate an infinite Braid from an infinite Garden of Forking Paths, even though their diagrams look different. An instruction valid in one is always valid in the other; an instruction invalid in one is correspondingly invalid.

Second, and more importantly, an infinite Braid is still infinite. Though there may be more than one path to a given room in the sequence, there is still an infinite number of rooms; the sorcerer will never visit them all. If you were to draw out a map of this labyrinth which admits an infinite number of paths, the map would extend infinitely to one side (this map would, you realize, expand to cover the labyrinth itself, no matter how small its lines and letters).

In one of your travels, after an eternity, something occurs to you about the indistinguishability of the rooms: just as two doors may be identical aside from their labels, as in the Braid diagrammed above, perhaps two indistinguishable rooms\u2026

You stop the blind sorcerer between instructions. You explain to him that he has, in visiting this room, visited the labyrinth. In fact, for all the vertigo of his infinite acceptable sequences, this room is equivalently the labyrinth.

The sorcerer, with an air of didactic satisfaction, asks for an explanation.

It\u2019s a little roundabout to start an explanation of finite state machines with an allegory involving infinite rooms\u2014infinitely many states, a sort of \u201cinfinite state machine.\u201d These don\u2019t have any practical application except to illustrate the relationship between the topology of the maze and the set of instructions it accepts. First we have a maze that accepts an infinite number of instruction sequences, and where no distinct sequences lead to the same room; then we have another maze that accepts an infinite number of instruction sequences (the very same!), but where distinct sequences can lead us to the same place.

These two labyrinths, the Garden and the Braid, also introduce the idea of equivalence. If two state machines accept the same inputs\u2014if there\u2019s no series of instructions from the sorcerer that reveals, via its acceptability in one and unacceptability in the other, in which maze we\u2019re trapped\u2014then the two state machines are equivalent.

From here on we\u2019ll deal with finite state machines (labyrinths with a finite number of rooms and doors), beginning with your answer to the sorcerer. Finite state machines are more useful: we can actually write programs that implement them, we can draw them out in finite diagrams, and there\u2019s nothing we can express in an infinite maze that we can\u2019t equivalently express in a finite one.

Finitude

Today, one of the churches of Tl\u00f6n Platonically maintains that a certain [room with two doors is] the only reality. All men, in the vertiginous moment of coitus, are the same man. All men who repeat a line from Shakespeare are William Shakespeare.

Tl\u00f6n, Uqbar, Orbis Tertius

This is your explanation: these rooms are indistinguishable to your explorations; every state permits the same two state transitions as the start state, so every state transition is essentially a return to the start state. Framing the state machine this way simplifies it: we can think of a single room instead of an infinite number of rooms!

Our first finite state machine: all the seemingly infinite rooms are a single room.

Importantly, this simplification doesn\u2019t change what instructions the maze allows: any among the infinite number of instruction sequences accepted by the infinite state machines diagrammed above is also accepted by the one-room simplified state machine diagrammed here. It\u2019s a finite maze with an infinite number of paths.

In fact, any finite maze with a cycle\u2014any way to leave a room and eventually return to it\u2014accepts an infinite number of paths, because you can validly move through the cycle once, twice, or any number of times. This one is a one-room cycle: you can transition from the one state to itself.

This dynamic is what makes finite state machines so useful: they\u2019re concise.

Sometimes they offer minimal notations for sequences that are otherwise hard to describe.4 It\u2019s easy to describe \u201cany sequence of lefts and rights,\u201d but sometimes we want to describe more complicated families of instructions. Compare the following diagrams to the verbal descriptions of the instructions they allow:

Description Diagram
Any sequence of lefts and rights where odd-indexed instructions are right.
Any sequence of lefts and rights, but where every third instruction can also be up.
Any number of up instructions, followed by any number of repetitions of the order right-up-left.

Each of these mazes describes a distinct infinite set of instruction sequences. The diagrams are clearer than the verbal descriptions.

The set of instructions the maze allows is locked into the maze\u2019s topology; the maze is a grammar for its own instructions. We can use the maze to generate instructions, not just test them: if the sorcerer were to ask you to produce a valid instruction sequence for a maze (some \u201csyntagm\u201d in the maze\u2019s grammar), you can take an arbitrary series of doors to build one.

Note, these state machines\u2019 properties are stable independent of their labels. We\u2019ve been working with directional labels (left and right) because they suit the labyrinthine theme, but nothing changes if we replace left with 0 and right with 1.

If we replace the lefts and rights, with 1s and 0s respectively, we have a state machine that accepts any finite binary string.

In fact, there are more equivalent finite labyrinths\u2014labyrinths with a finite number of rooms, that admit any sequence of binary instructions\u2014than those covered here. Their construction is left as an exercise for the reader.5

The blind sorcerer listens quietly to your explanation of the infinite maze with one room, which perfectly describes every instruction you could ever accept from him\u2014every path in the Garden of Forking Paths. He smiles, apparently satisfied with the explanation; \u201cif all its infinite rooms are one room, this is a crummy corner of my labyrinth. Let\u2019s find someplace with variety.\u201d

As he says these words (zap!) you are transported to a wholly different maze.

Acceptance

When it was proclaimed that the Library contained all books, the first impression was one of extravagant happiness. All men felt themselves to be the masters of an intact and secret treasure. There was no personal or world problem whose eloquent solution did not exist in some [room].

The Library of Babel

In the interest of difference, the sorcerer introduces a new rule. At first you were to accept any set of instructions you could follow; now you are to reject unfollowable instructions, but also to object if the last instruction in a followable sequence doesn\u2019t lead you to a room containing a MacGuffin.

Two rooms with the same doors\u2014up to this point, indistinguishable\u2014might now be different: perhaps one contains a MacGuffin and the other one does not. Moreover, two rooms which are indistinguishable to their occupant at a point in time (two rooms with the same doors, neither of which contains a MacGuffin) may be provably distinct because they have different positions relative to MacGuffin rooms in the maze\u2019s topology.

As you explore these mazes, you\u2019re struck by their insistence on their own size. If an infinite braid of rooms has a single MacGuffin in the hundredth rooms, the rooms before the MacGuffin can\u2019t be summarized as a single room: one is just before the MacGuffin room; the room before that is two rooms before the MacGuffin; and so on. Even though they have the same doors, these rooms are clearly distinguishable.

Past the MacGuffin, though, you have no such information. You know there is a room after the MacGuffin, but since you will never encounter another MacGuffin you can never know whether there are rooms after that one (e.g.\u00a0in an infinite braid) or whether it\u2019s a single trick room with exits that lead you back into it.

Before this introduction of MacGuffins, in the language of state machines, every node was an acceptor\u2014it was as if every room contained a MacGuffin. Now only certain nodes are acceptor states. If, after exhausting input, a state machine is an acceptor state, it \u201caccepts\u201d the input sequence.

State machine diagrams denote acceptors with double-walled circles (which is why the diagrams before this section have all had double-walled rooms: every room was an acceptor). Single-walled circles are not acceptors\u2014these are the rooms we can pass through but don\u2019t want to end in.

If this dynamic isn\u2019t clear, take a moment to pretend to be the sorcerer. The room you\u2019re in is colored cyan; you can use the 0 and 1 buttons to provide input (to select doors to take. The checkbox at the bottom indicates whether the state machine accepts the input: it\u2019s checked when you\u2019re in the acceptor state R1, but not when you\u2019re in any other state.

Significantly, these mazes can more finely restrict what instruction sequences they accept. Without these special MacGuffin rooms, any prefix of an acceptable instruction was itself an acceptable instruction; if less than every room contains a MacGuffin, that\u2019s not necessarily the case! Before, every room was an acceptor, every state an acceptable terminal state. Now there are nonterminal states: valid instructions can take us through them, but can\u2019t end there. Now we can design mazes where a prefix of a valid instruction sequence isn\u2019t guaranteed to be acceptable:

Description Diagram
Any nonempty binary string that begins and ends with the same digit. This accepts 1-0-0-0-1, but not 1-0-0-0.
Any binary string ending in 0-0. This accepts 1-1-1-0-0, but not 1-1-1 or 1-1-1-0.

As your understanding of the labyrinths deepens, your exploration with the sorcerer seems to accelerate, as if freed from ponderous existential questions. Your juvenile dread that the great labyrinth might be meaningless is supplanted by a vertigo of interpretations: a din of instructions, a nauseating race through passageway after passageway, MacGuffin and non-MacGuffin rooms, accepted and rejected sequences, and the blind sorcerer\u2019s increasingly frenetic jumps (zap! zap! zap!) from one maze to the next.

Determinism

In all fictional works, each time a man is confronted with several alternatives, he chooses one and eliminates the others; in the fiction of the almost inextricable Ts\u2019ui P\u00ean, he chooses\u2014simultaneously\u2014all of them. He creates, in this way, diverse futures, diverse times which themselves also proliferate and fork.

The Garden of Forking Paths

That nausea is interrupted, suddenly, by a dilemma. In a new maze, you\u2019re confronted with choice: one door is marked 0, and two doors are marked 1. When the sorcerer instructs you to pass through the door marked 1, through which door should you pass?

In this two-room maze we must end up in the acceptor room R1; any instructions that leave us in R0 are invalid. Depending on how we interpret the last 1, the instruction set 0-0-0-0-1 could either be valid (if we take the door leading from R0 to the acceptor room R1) or invalid (if we take the door leading from R0 back into R0).

\u201cYou will take both doors,\u201d explains the blind sorcerer. Your task is to find out whether the instructions lead you to a MacGuffin through either door. This puzzles you, but the sorcerer explains that there are a couple ways to take it literally:

  1. Whenever you encounter several identical door, the sorcerer can conjure an avatar of you to pass through each door in parallel. If two avatars would meet in some room down the line, they are consolidated into one.

    This prospect deeply unsettles your sense of self (are you the avatar? or the original?), and the sorcerer admits it gives him migraines: conjuring avatars takes time and effort, and delivering simultaneous instructions to innumerable (and maybe ever-multiplying!) avatars is, he says, \u201ca pain in the ass.\u201d

  2. You can pass through both doors by backtracking. When you choose one door over another identical one, note the decision; then, if the first door doesn\u2019t lead you to accept the instructions, you can return to take the second door.

    This constant retracing, of course, takes time: you have to return to a room once for each identical door, and subsequent rooms can themselves have identical doors. Before, you would pass through at most n rooms to (in)validate a sequence of n instructions. Now you may need to pass through many more

You commit to backtracking (since, if you have to spend your existence trying to make sense of an infinite labyrinth for a mad sorcerer, you might as well feel existentially unique while you do it). The vertigo returns, but you have the kernel of an idea to eliminate the problem of choice.

These mazes\u2014with several indistinguishable exits from a given room\u2014are nondeterministic finite state machines (NFAs). They\u2019re \u201cnondeterministic\u201d because your state at a given time isn\u2019t totally determined by a set of instructions; you could be in several states at a given time, depending on which doors you chose. The two strategies proposed by the sorcerer correspond to two strategies available to a program implementing an NFA: they can evaluate possible states sequentially, or they can track multiple states in parallel. Both of these strategies are undesirable: you can have avatars in up to every room of a maze at once, which makes applying inputs burdensome. Regular expression implementations use NFAs; these dynamics are what make certain regular expressions inefficient.

You think, in all the time you now spend backtracking, about the avatars. Some number of avatars, in certain rooms in a finite maze, receive a certain instruction and at once spring into certain other rooms. In this process, some avatars will split, some will consolidate, and those for whom the instructions are unfollowable will blink out of existence.

It\u2019s difficult to imagine the possible configurations of avatars in rooms because there are so many6, but a marvelous fact occurs to you: from the sorcerer\u2019s perspective, a configuration of avatars behaves deterministically. If avatars are in some set of rooms, then after receiving a certain instruction they will be in a certain determinable second set of rooms regardless of how they traveled to that prior arrangement.

Is this passage from configuration to configuration really so different from you passing, deterministically, from one room to the next?

You interrupt the sorcerer a second time, and you make him an offer: for any finite maze confused with choice, you say, any maze burdened with identical doors, you can design for him a finite equivalent maze devoid of choice.

This is what the avatars reveal: every nondeterministic finite state machine is equivalent to some deterministic one!

Your proposal:

  1. For every configuration of avatars in rooms in the nondeterministic maze, build a room in our new deterministic maze. If a configuration has an avatar in a room with a MacGuffin, then put a MacGuffin in the new room corresponding to that configuration.

  2. For each configuration of avatars, consider each instruction that at least one avatar could validly follow (in other words, consider the set of door-labels the avatars see at once). For each instruction I:

    1. Figure out what every avatar would do if given this instruction: some move simply into another room, some split and move into several. This gives you a successor configuration.

    2. In the new deterministic maze, build a door out of the room corresponding to the initial configuration that leads to the successor configuration. Label it with the instruction I.

  3. Some rooms in our new maze may not be reachable from the maze\u2019s start. Demolish these; they\u2019re superfluous.

Since we never repeatedly assess the same instruction applied to the same configuration of avatars, we will never build two doors in the same room of our new maze that have the same label. Thus, the new maze can be explored deterministically.

Since any configuration in the nondeterministic maze accepts the same instructions as its corresponding room in the deterministic one, we haven\u2019t changed the followability of any instructions. Because every room with a MacGuffin in the new maze corresponds to a configuration in the nondeterministic maze where some avatar is in a room with a MacGuffin, the mazes make the same terminal judgements of followable sequences. They\u2019re equivalent!

Of course, since there are so many possible configurations of avatars in mazes, the new deterministic maze may be considerably larger than the nondeterministic maze it models.

This algorithm\u2014the powerset construction, adapted for our allegory\u2014is best explained with an example. Consider the following nondeterministic finite maze, with rooms labeled A, B, and C:

A nondeterministic maze. There are two doors out of room B marked 0 and two doors out of room A marked 1. Any instruction sequence that leaves an avatar into room C is accepted.

Our first step is to plan out the rooms in the new maze, a finite state machine with one state for each possible configuration of avatars in the nondeterministic maze. We can use set notation to describe the arrangements of avatars:

and so on. Any state with an avatar in room C is an acceptor state, and at the start there is only one avatar (in room C). We can list out the seven states corresponding to the seven possible arrangements of avatars in the three-room maze:

State in new maze Acceptor? A B C
{A} No \u2714\ufe0f
{B} No \u2714\ufe0f
{C} (start state) Yes \u2714\ufe0f
{A,\u2006B} No \u2714\ufe0f \u2714\ufe0f
{A,\u2006C} Yes \u2714\ufe0f \u2714\ufe0f
{B,\u2006C} Yes \u2714\ufe0f \u2714\ufe0f
{A,\u2006B,\u2006C} Yes \u2714\ufe0f \u2714\ufe0f \u2714\ufe0f

Next, we need to consider the possible transitions out of each of these states. Thankfully this is a pretty simple maze: we only need to consider what happens to each of these states if the sorcerer delivers the instruction 0 or the instruction 1:

State +0 +1
{A} {A,\u2006C} Invalid
{B} Invalid {A,\u2006C}
{C} {B} Invalid
{A,\u2006B} {A,\u2006C} {A,\u2006C}
{A,\u2006C} {A,\u2006B,\u2006C} Invalid
{B,\u2006C} {B} {A,\u2006C}
{A,\u2006B,\u2006C} {A,\u2006B,\u2006C} {A,\u2006C}

Finally, we prune the unreachable states and diagram our deterministic finite state machine. It\u2019s worth taking a moment to compare it to the tables above: confirm that each of the acceptor states includes an avatar in room C; check that the state transitions correspond to those described in the instruction table.

Our powerset-constructed deterministic equivalent!

This process lets us do away with all the complexity of avatars and backtracking, without changing which instructions are accepted and rejected, for any finite state machine, no matter how complex. You and the sorcerer can explore these deterministic mazes without any insufferable nondeterminisms; though this deterministic state machine looks nothing like the equivalent it models, exploring it is essentially exploring the collective experience of the avatars in their nondeterministic maze. For any arrangement the avatars would realize collectively, you reach a corresponding state.

If you need to convince yourself of this correspondence, you can use the interactive demo below to provide simultaneous inputs to the nondeterministic and deterministic equivalent state machines. Cyan nodes indicate the presence of an avatar in a room; note that the state of the deterministic machine always identifies the occupied rooms in the nondeterministic one.

Allegory

Augustine had written that Jesus is the straight path that saves us from the circular labyrinth followed by the impious; these Aurelian, laboriously trivial, compared with Ixion, with the liver of Prometheus, with Sisyphus, with the king of thebes who saw two suns, with stuttering, with parrots, with mirrors, with echoes, with the mules of a noria and with two-horned syllogisms. [\u2026] Like all those possessing a library, Aurelian was aware that he was guilty of not knowing his in its entirety; this controversy enabled him to fulfill his obligations to many books which seemed to reproach him for his neglect.

The Theologians

The sorcerer (Borges) and his mazes are an indulgent conceit for an explanation of state machines. For a reader really interested in the mathematical concepts, the comparison might confuse more than it makes clear: it may be harder to imagine yourself splitting into multiple indistinguishable avatars than it is to imagine a multi-node state, and so my explanations in this essay become less allegorical as the concepts become more disorienting.

I\u2019m satisfied with my exploration of the correspondence between state machines and sequences, but the allegory says nothing about the \u201cmeaning of your hermitude\u201d in the maze. To do so requires stepping back out of the diegetic \u201cyou\u201d in the allegory, the little traveler in the labyrinth, and considering the you reading the allegory.

First, I can offer you a shallow meaning: state machines are useful tools for turning seemingly complex grammars or agents into simple, applicable programs. That\u2019s how they were covered in my undergraduate computer science curriculum. It\u2019s not a wrong justification\u2014these applications are worthwhile\u2014but it\u2019s partial.

The other justification is to recognize that this essay is a reversal of work Borges has already done. His stories don\u2019t discuss mathematics explicitly (they\u2019d be worse\u2014more gimmicky\u2014if they did) but they rework topology and combinatorics into phenomenological and existential puzzles. I can explain topology via a phenomenological puzzle about a maze because Borges establishes, in various narrative guises, the structure of the allegory. Even if Borges hadn\u2019t, at certain touchpoints the allegory is obvious. Topology connects intuitively to the everyday way one moves between physical spaces; surprising topological twists, like the equivalence of an infinite maze and a minimal one, correspond to uncanny views of the everyday.

Borges skillfully extends these allegories beyond their natural correspondences. The magic of A New Refutation of Time is that it imagines a kind of spatialized time, wherein two indistinguishable moments are ontologically one:

That pure representation of homogeneous objects\u2014the night in serenity, a limpid little wall, the provincial scent of the honeysuckle, the elemental earth\u2014is not merely identical to the one present on that corner so many years ago; it is, without resemblances or repetitions, the very same. Time, if we can intuitively grasp such an identity, is a delusion: the difference and inseparability of one moment belonging to its apparent past from another belonging to its apparent present is sufficient to disintegrate it.

This is precisely the same radical idealism that lets us say an infinite maze is a single room. If you prefer a philosophical connection, Borges treats the phenomenology of time the way Aug\u00e9 treats place.

Where does Borges\u2019s labyrinth lead? To a dizzying metaphysics, as compelling as it is disconcerting. The Borgean maze is a trap, an entanglement of symbols and experiences, reflections and repetitions. I hope that following his allegories in the opposite direction\u2014beginning with the existential allegory, but taking it as a point of departure into the mathematics Borges adapts but elides\u2014has the opposite effect: mazes can be games, too. The trick is to enjoy them.


  1. Jorge Luis Borges. \u201cThe House of Asterion.\u201d In Labyrinths: Selected Writings & Other Stories. New York: New Directions Books, 1962. Note: all subsequent quotations from Borges short stories are from this translation.\u21a9\ufe0e

  2. Using a state machine to accept or reject inputs might be foreign to readers who use state machines to, for example, determine the actions of an agent in a game. In the interest of keeping the Borgean allegory as simple as possible\u2014it can be hard to follow as it is\u2014this story concerns itself with state machines as they\u2019re used in pattern-matching: we give the sorcerer a boolean output.\u21a9\ufe0e

  3. Note, there is one acceptable route to each room (that is, a sequence of lefts and rights). There are also innumerable unfollowable instructions\u2014which say to pick a door marked axaxaxas ml\u00f6, for example\u2014but we won\u2019t sweat that here.\u21a9\ufe0e

  4. Minimal notation is really a feature of formal grammars. Diagrams of state machines corresponding to those grammars are handy visualizations (I find them much easier to skim).\u21a9\ufe0e

  5. This is a cheeky suggestion. There\u2019s an infinite number of finite state machines that admit any sequence of binary instructions; they just need to end in some cycle (either a single-node cycle, where one constantly reenters the room they\u2019re in, or a multi-node cycle like a looped braid).

    Those terminal cycles can be preceeded by any non-cyclical labyrinth (e.g.\u00a0the tree in figure 1, the braid in figure 2, or some other directed acyclic graph) as long as every room has a door labeled 0 and a door labeled 1.

    And, finally, there are unexplored dimensions: a finite state machine that accepts a superset of the set of finite binary inputs satisfies our requirements.\u21a9\ufe0e

  6. After a certain set of instructions, each room can either contain or not contain an avatar. Thus, for a maze with n rooms, you have 2n possible configurations of avatars in the maze.

    Note, though, that this is an upper bound. If the only way into room Rleft is a door marked left and the only way into room Rright is a door marked right, then no set of instructions will simultaneously put avatars in Rleft and Rright (because no sequence can end with both left and right).\u21a9\ufe0e

", "summary": "An introduction to state machines (also called \"automata\") via an extended Borgean allegory about an existence in a labyrinth, a sort of infinite gallery of mazes. It introduces several significant concepts---equivalence, finitude, acceptance, determinism, and the powerset construction---as a succession of realizations about that existence.", "date_published": "2021-02-19T02:15:24.753000+00:00" }, { "id": "./gen/worse-on-purpose.html", "url": "https://lukasschwab.me/blog/./gen/worse-on-purpose.html", "title": "Making Things Worse on Purpose", "content_html": " Making Things Worse on Purpose « blog

Making Things Worse on Purpose

Lately I\u2019ve enjoyed breaking bad habits and resolving little frictions in my life by tweaking apps that cause and enable them.

news-feed-eradicator specifically eliminates Facebook\u2019s news feed, that erstwhile attention-trap. More generically, you can customize an ad blocker to block arbitrary content on a website; I use this to hide comment sections which would otherwise waste my time. These social features make products \u201cbetter\u201d inasmuch as they make them more profitable (via user engagement), but the psychology of a bottomless social feed is such that my engagement doesn\u2019t necessarily correspond to it providing me any value at all.

YouTube\u2019s default video quality throttling mismanages my computer\u2019s resources rather than my attention: their playback quality throttling considers bandwidth, but not the client\u2019s graphics hardware or playback speed. The fan on my 2013 laptop is screaming after a minute of streaming 1440p video at 60 frames per second and 2.5x normal speed.

Because YouTube\u2019s player controls store a manually-set quality limit in Chrome\u2019s Local Storage (as yt-player-quality), a Chrome extension can simulate manually correcting the defaults. I don\u2019t need to abide by YouTube\u2019s expectation that I prefer higher-quality video.

Unfortunately, we don\u2019t have this degree of flexibility in mobile browsers (which lack the extensibility of desktop browsers) or closed-source native apps (which don\u2019t share a single customizable client). This is the same product-user dynamic that makes syndication feeds important: allowing end-users to tinker with an interface alleviates some pressure to predict what they need.

Tinkering is a gratifyingly active kind of usership, one that \u2014 at its extreme \u2014 cuts against applications\u2019 commercial drive to be desirable, to induce dependency.

", "summary": "Customizing one's browser is a neat way to cut out app \"features\" that squeeze value out of users by being more irritating than useful.", "date_published": "2021-02-13T21:46:38.335000+00:00" }, { "id": "./gen/hearst-hall.html", "url": "https://lukasschwab.me/blog/./gen/hearst-hall.html", "title": "Hearst Hall", "content_html": " Hearst Hall « blog

Hearst Hall

An urban college campus is an interesting place for architectural archaeology. UC Berkeley is dense, and in constant redevelopment; for a couple hundred years, layers of Berkeley have been plowed under to make way for new buildings, programs, and generations of students.

That plowing-under is so complete that, for five years, I lived a short walk from what has become one of my favorite buildings in California: a hall built and rebuilt by Bernard Maybeck, a star of the California Arts & Crafts movement, to house a women\u2019s social center-slash-theater-slash-concert hall. Hearst Hall is full of gothic drama and prefigurations of modernism. For five years I didn\u2019t realize it because Hearst hall no longer exists.

I only discovered Hearst Hall after walking past the Swedenborgian Church in San Francisco\u2014also a Maybeck building\u2014enough times to remember to look it up. That church led me to Maybeck, who designed a great number of notable buildings in Berkeley and San Francisco in his idiosyncratic medieval-revivalist mashup style.

The interior of Bernard Maybeck\u2019s Hearst Hall, built in 1899. This photo shows the interior configured as a sort of sitting room or theater space or something; the hall served a myriad of purposes

Phoebe Anderson Hearst turned to philanthropy, including a long patronage of the University of California, after her husband\u2019s death in 1891. In 1896 she funded a $100,000 competition to design an expanded general plan for the University (the third such general design commissioned, after an original design by F.L. Olmstead and an 1870 David Farquarson expansion Olmstead bitterly resented). As the competition\u2019s sponsor, she not only moved to Berkeley but commissioned from Bernard Maybeck\u2014then a professor\u2014a new building to host competitors and judges: Hearst Hall.1

The Hall\u2019s main chamber is striking: the 16-meter-tall vault of a gothic arch, extruded like an A-frame but closed at the ends, strung with round arches of incandescent lights. It\u2019s as if Maybeck sliced the top off of an enormous redwood gothic basilica.

The decorators\u2019 clever use of suspension (both in the decor for this sitting-room arrangement and the gymnastic equipment pictured below) gives the impression of matryoshka volumes, one inside the other. A fabric hung unevenly by its four corners looks like it\u2019s caught mid-fall. The furniture is eclectic and sparse. Planters flank the alcoves. The sense of space in these photos\u2014the nested volumes drawing attention to the voids between them\u2014reminds me of this Paris Review line about Berlin living rooms:

In my romance, the apartment is large enough for a given room to have whatever purpose the occupant chooses for it. The point of a Berlin apartment of this kind is to enjoy the emptiness.2

Which, to my surprise, fits Maybeck\u2019s architectural philosophy:

The house after all is only the shell and the real interest must come from those who are to live in it. If this is done carefully and with earnestness it will give the inmates a sense of satisfaction and rest and will have the same power over the mind as music or poetry or any healthy activity in any kind of human experience.3

Hearst, like Maybeck, wanted the hall to serve a succession of different virtuous purposes. Phoebe Hearst brought Berkeley a world-class education in classical music\u2014an event series complete with individual invitations to students. When she left Berkeley for Pleasanton, Hearst had the hall disassembled, moved, and reassembled on the university\u2019s campus to serve as a women\u2019s social center (for \u201cSaturday afternoon receptions, musicals on Sundays, \u2018At Home\u2019 Wednesdays, dinners three evenings a week\u201d4). Hearst and Maybeck may have had that move in mind from the start: the hall is a construction of \u201claminated arches and diaphragms\u2026 sections of the building [were] independent, movable, and easily re-assembled units.\u201d5 Realizing that \u201cthe women\u2019s physical education program was almost non-ecistent since the facilities of Harmon Gymnasium were available to them only during those few hours a week when wthe men were engaged in drill,\u201d Hearst paid to repurpose and expand the building as a women\u2019s gymnasium.6

Hearst Hall reconfigured as a Gym after its relocation to campus in 1901.

The Hall was destroyed by a fire in 1922, three years after Phoebe Hearst\u2019s death. Its adjoining swimming pools\u2014built when it became a gym\u2014were used for swimming and hydrological experiments until they were cleared in 1955 to make way for Wurster Hall.7

William Randolph Hearst\u2014Phoebe Hearst\u2019s son\u2014commissioned a new women\u2019s gym in his mother\u2019s memory from Maybeck and Julia Morgan;8 that\u2019s the Hearst Memorial Gymnsium on Cal\u2019s campus today. Hearst Gym has none of former Hearst Hall\u2019s bohemian charm: it trades in the quirky medieval revivalism for a much more fireproof stone-and-cinderblock Beaux-Arts. George Hearst\u2019s namesake builing, the Hearst Mining building in the northeast corner of campus, is much prettier.

Wurster is an interesting point of dialog for its predecessor. It houses Berkeley\u2019s College of Environmental Design (CED), including Cal\u2019s architecture library; people without a taste for concrete joke about it driving architecture students crazy. Personally I think Wurster Hall\u2019s ugliness is way overrated (Evans is a far more visually offensive, and neighboring Law School commits the double-sin of being ugly and boring), but it certainly can\u2019t be mistaken for a Maybeck building: there\u2019s nothing in the least bit handcrafted or medieval about it, maybe save its general external menace. Wurster Hall doesn\u2019t exactly project whimsy.

Beneath this concrete surface, though, Wurster Hall shares Hearst Hall\u2019s philosophical bones.

The modern hulk is named for William Wurster, founding president of the CED. Wurster, a student of Maybeck\u2019s, graduated from Berkeley in 1919. Like Maybeck, Wurster\u2019s professional style tended quirky, low and modern but quintessentially Californian. Why would he commission such a stark building for his department?

Wurster Hall nearing completion.9

Bill Wurster wanted the designers to design what he called a ruin, a building that \u201cachieved timelessness through freedom from stylistic quirks.\u201d [\u2026] Historian Sally Woodbridge neatly sums up both William Wurster\u2019s ideals and the building that was named after him: \u201cAs Wurster Hall weathered without mellowing, it reflected Wurster\u2019s opinion that a school should be a rough place with many cracks in it. Perpetually unfinished, Wurster Hall was an open-ended and provocative environment for teaching and questioning.\u201d10

Woodbridge quotes Wurster\u2019s Oral History:

\u201cI wanted it to look like a ruin that no regent would like\u2026 It\u2019s absolutely unfinished, uncouth, and brilliantly strong.\u201d11

\u201cA ruin.\u201d Maybeck\u2019s Palace of Fine Arts in San Francisco was famously designed as a \u201cforgotten and overgrown Roman ruin\u201d in 1915.12 Though they\u2019re ruins in very different styles, Wurster Hall and the PFA reference ruination to signify a similar openness. Wurster talks about achieving timelessness through quirklessness, but \u201cruin\u201d evokes more than just bareness: ruins point back at a pre-ruined history. Maybeck\u2019s point back is nostalgic for medieval handcrafts and classical arts, a rebellion against industrial production. Wurster Hall feels altoghether bleaker; it looks skeletonized, burned clean. Whereas the PFA points hauntologically at an unrealized enlightened progression out of classicism, Wurster Hall points back at biblical warnings: \u201cMENE, MENE, TEKEL, UPHARSIN.\u201d

The buildings share some minor external similarities, though I have no evidence they\u2019re intentional. Hearst Hall\u2019s facade, with its towers, is hulking and carceral. The balcony on Wurster\u2019s southern face is like a vestigial remnant of Hearst\u2019s weird turret-obelisk. I\u2019ve waxed poetic about Hearst Hall\u2019s interior, but I\u2019m cherry-picking: nothing about this exterior appeals to me.

\u201cPerpetually unfinished.\u201d Wurster\u2019s vision is consistent with Maybeck\u2019s philosophy, in which \u201cthe house after all is only the shell.\u201d Wurster\u2019s bareness leaves it ready for contstant reconfiguration\u2014as an event space, as galleries of removable walls for end-of-semester showcases, for students working in all kinds of groups on all manner of projects. While the building itself can\u2019t be disassembled and moved like Hearst Hall was, its interior is constantly disassambled and rebuilt.

This isn\u2019t surprising for a modern design college, but I\u2019m surprised to spot echoes of the 19th century building once built on the same ground. I\u2019d go so far as to say that Wurster is the only building at Cal channeling Hearst Hall\u2019s ethos: this openness to rearrangement, the deference to the building\u2019s inhabitants to give it purpose and interest. Moreover, it\u2019s one of very few buildings (along with the Hearst Mining building, donated by Phoebe Hearst in memory of her mining magnate husband), that reveals its own structure: whereas Hearst Hall bared its ribcage of laminated arches, Wurster Hall flaunts slab concrete.

The comparison doesn\u2019t only spice up Wurster Hall; it reveals Hearst Hall to be, in some respcts, modern. Modularism, repurposing spaces, and the unity of structural and aesthetic elements seem like modernism\u2019s innovations, but they\u2019re all present in this 1899 building by an architect whose architect was in rebellion against industrialist modernization.

Buildings are weird texts; they don\u2019t lend themselvves to comparative analysis. So much of architecture\u2014Hearst Hall, for example\u2014is unique: they aren\u2019t reproduced like books from an ur-text, which means that when Hearst Hall burns down I\u2019ll never really be able to \u201cread\u201d it as I can read an extant building. Even extant buildings are complicated. As a building\u2019s purpose shifts, so does the way its occupants experience it. I like the idea, though, that buildings point into the past like the Archaeological Research Facility points, tongue in cheek, at its frat origins: \u201clook what used to be where I am now; consider the purpose I was given then; consider the ruin I will eventually become.\u201d Like Hearst Hall\u2019s nested arches, spaces inside spaces.

After some continued reading on Bernard Maybeck, I\u2019ve written a follow-up to this essay. It uses stronger evidence to cover much the same ground, and extends my discussions of Maybeck\u2019s architectural philosophy and the design of Wurster Hall.


  1. Corbett, Michael. \u201cHearst Memorial Gymnasium Historic Structure Report.\u201d 2005: 12-13. Note: the most useful detailed, but brief, chronology of Phoebe Hearst\u2019s involvement with UC Berkeley I could find.\u21a9\ufe0e

  2. Nabokov, Dominique & Darryl Pinckney. \u201cBerlin Living Rooms.\u201d Paris Review, 2017.

    Never mind that I\u2019m using this quote in a wildly different architectural context. It\u2019s funny to compare International Style sparseness with Arts & Crafts sparseness. Two departures from 19th century maximalism, but departures towards very different metaphysics: medievalism and handcraftsmanship on one hand, modernism and industrial production on the other.\u21a9\ufe0e

  3. Cardwell, Kenneth H. Bernard Maybeck: artisan, architect, artist. Santa Barbara and Salt Lake City: Peregrine Smith, 1977. (157)

    Also quoted in Le Guin, Ursula K. \u201cWords Are My Matter: Writings About Life and Books, 2000-2016.\u201d (52)

    Le Guin grew up in a Berkeley house designed by Maybeck. Apparently quite a quirky one: \u201cour house, for example, originally had no stairs to the basement. \u2018Maybeck was moody about stairs.\u2019\u201d (53)

    Le Guin calls Maybeck\u2019s architectural philosophy premodern. I read The Left Hand of Darkness recently; maybe there\u2019s an inspiration here for Karhide\u2019s distinctly premodern cities, with their unreachable doors?

    For more on Maybeck\u2019s medievalism, see also Mitchell, James. Medieval San Francisco: Neo-Gothic Architecture in Northern California, 1900-1940 (2016), 89-96.\u21a9\ufe0e

  4. Kantor, J.R.K. \u201cCora, Jane, & Phoebe: Fin-de-Si\u00e8cle Philanthropy.\u201d Chronicle of the University of California 1, no. 2 (1998): 6.\u21a9\ufe0e

  5. Miller, Leta E. \u201cPractical Idealism: The Musical Patronage of Phoebe Apperson Hearst.\u201d Journal of the Society for American Music 10, no. 4 (2016): 397.\u21a9\ufe0e

  6. Kantor, J.R.K. \u201cCora, Jane, & Phoebe: Fin-de-Si\u00e8cle Philanthropy.\u201d Chronicle of the University of California 1, no. 2 (1998): 6.

    For a gushing contemporary account of the events and activities to which Hearst Hall was home, see the 1904 Blue and Gold excerpt on page 9. For a detailed discussion of its impact on women\u2019s athletics at Cal, see \u201cA Gym of Their Own\u201d (Park) in the same volume, especially pages 25-26.\u21a9\ufe0e

  7. Dougherty, Carolyn. The Loafer\u2019s Guide to UC Berkeley.\u21a9\ufe0e

  8. Maybeck and Morgan were surprising choices: Phoebe Hearst vocally supported John Galen Howard, the Supervising Architect hired in the campus expansion competition she sponsored.

    Howard turned out to be something of a tyrant. From The Loafer\u2019s Guide to UC Berkeley, with my emphasis added:

    [Howard] had always insisted on complete control of campus design, as well as the sole right to design campus buildings; this the Regents were no longer willing to grant him. In 1922 they awarded the design of the new Hearst Gym to Bernard Maybeck and Julia Morgan while Howard was in Europe. This building, and the attached buildings and grounds originally planned, were the seed of a \u201crevolt\u201d organized by Bernard Maybeck, Julia Morgan and William Randolph Hearst, whose memorial gym for his mother had originally been intended as the nucleus of a \u201ccountercampus\u201d in the style of San Simeon.

    There\u2019s probably a much more dramatic way to tell this story: three generations of Hearsts (mining, publishing, and bank-robbing); fires; rival architects with their respective patrons\u2026\u21a9\ufe0e

  9. UC Berkeley College of Environmental Design. College History.\u21a9\ufe0e

  10. UC Berkeley College of Environmental Design. College History.\u21a9\ufe0e

  11. Woodbridge, Sally. \u201cReflections on the Founding: Wurster Hall and The College of Environmental Design [Two Place Tales].\u201d Places 1, no. 4 (1984).\u21a9\ufe0e

  12. National register of Historic Places. Registration Form: Palace of Fine Arts.\u21a9\ufe0e

", "summary": "Hearst Hall is a stunning example of California Arts & Crafts architecture. What does a gothic hall—built, moved, and built again by a mining baroness, burned down in 1922, and now largely forgotten—say about the spaces that replace it?", "date_published": "2020-11-22T00:14:10.653000+00:00" }, { "id": "./gen/json-feed-tools.html", "url": "https://lukasschwab.me/blog/./gen/json-feed-tools.html", "title": "Web Syndication with JSON Feeds", "content_html": " Web Syndication with JSON Feeds « blog

Web Syndication with JSON Feeds

Sometime in the last ten years, while you were mourning the loss of Google Reader, we entered the golden age of content syndication. Our social media overlords hit the syndication Comstock Lode. For all their dystopic visioneering, I doubt the feed-accelerationists at Microsoft and Netscape in the mid 90\u2019s foresaw these particular macroeconomics.

Arguably, though, we\u2019re also in a golden age of nondystopic author-managed syndication. Free and nearly-free tools for hosting static sites are only outnumbered by static site generators; new ones are released every week. These tools, like blogosphere-era blogging platforms, can generate feeds as side effects of the routine publishing activity of their users; many do so by default. Even if it\u2019s only a feed of content previews (to draw users onto the publisher\u2019s site), each feed is a contribution to the digital commons.1

Syndicated feeds \u2014 for which RSS, Atom, and JSON Feed are specifications \u2014 are essentially different from the feeds turning social media users into blue-app-anxiety foie gras. Rather than an algorithmically ranked and collated series of texts from a variety of sources, syndicated feeds just list items as a single source; the categorizing, collating, and display of those items is left up to feeds\u2019 consumers. This has accessibility upsides, makes feeds easy to process programmatically, and provides a neat interface for users waiting on sparse updates (e.g.\u00a0a blog that only updates once in a blue moon).

Providing a feed might mean content loses ad impressions to feed readers, but feeds generally align the interests of author-publishers who want their work read with the folks doing the reading.

The social challenge: maintaining and checking a feed reader will reward users only if their favorite sources of content provide feeds to be followed. For those sources of content, maintaining a feed (trickiest during site migrations!) is only worthwhile if readers would not otherwise follow them.

An introductory note on feed formats \u2014 RSS has the longest history and is the most widely-known, but its XML specification is pretty deeply janky. I would not recommend writing code for working with RSS feeds. Atom, RSS\u2019s successor in the XML feed tradition, is a strict improvement. Most feed readers support both.

JSON Feed is a relative newcomer, introduced by the authors of NetNewsWire and Micro.blog. It has less client support than Atom/RSS, but it\u2019s a sweet format to tinker with. I find JSON easier to read than XML, and my languages of choice these days (Go, Python, Typescript) have much nicer support for parsing and writing JSON objects than for XML (even with Python\u2019s feedparser).

JSON feeds make syndication so simple that I\u2019ve written a cluster of interrelated tools for working with them. Here\u2019s a narrative breakdown of how they came to be and how I use them together.


Habitually collecting feeds makes one very aware of how many sites don\u2019t (but should!) have them; how many have feeds but don\u2019t prominently list links to them; and how many publications offer central aggregate feeds but not feeds broken down by category or author. I\u2019ve built myself a few tools to help with this.

feedscan is a bash utility for discovering feeds by checking the routes that commonly host them: /feed, /atom.xml, and so on. If I find a sweet blog at lukasschwab.me/blog, I try feedscan https://lukasschwab.me/blog before digging for a link on the site itself. It\u2019s totally disconnected from the other projects discussed here, but it has saved me a lot of frantic searches.

jsonfeed is a JSON feed parser and constructor package written in Python, the backbone to most of my other JSON feed tools. I wrote a Go equivalent, go-jsonfeed, but haven\u2019t used it much. This very blog is generated with a fork of pandoc-blog, which generates a JSON feed using the jsonfeed package I authored.

I discovered a little running project for myself in building and hosting public feeds for sites that don\u2019t offer them. I got my start with arxiv-feeds, which converts Atom to JSON using jsonfeed, but it\u2019s a relatively boring wrapper. Blogs and news sites are more fun because they involve scraping feed items from the sites on demand. I wrote separate Python scraper/generator apps for a couple of sites, then realized those generators shared a certain procedural structure:

  1. Fetch some target page.
  2. Parse the target page HTML (with Beautiful Soup).
  3. Extract feed items from the parsed HTML, an operation unique to each site.
  4. Return the constructed feed.

Steps 1, 2, and 4 were essentially shared, so I factored them out into jsonfeed-wrapper, which takes the site-specific HTML-to-feed transform and wraps it with the standard fetching and feed-serving logic. I originally designed it for use with Google App Engine, but last weekend I rewrote it to expose a Google Cloud Function target.2 Cloud Functions save me a couple bucks a month.

I generate and host feeds for It\u2019s Nice That, Bandcamp artists, The Baffler, and Atlas of Places. Generating feeds from scraped HTML is somewhat brittle, but these have been reliable enough for the last few months. Adding a new site to the list takes about an hour of filling out a jsonfeed-wrapper template; shortening that time is the jsonfeed-wrapper project\u2019s north star. Everything deserves a feed.

The next frontier: feed filters with CEL. cel-go works neatly with raw JSON, but the JSON Feed schema is well defined \u2014 why not create a CEL environment with types and macros for filtering feeds?

I have a Cloud Function that does nothing but parse Bruce Schneier\u2019s RSS feed, filter out the feed items involing squid (Bruce\u2019s hobby outside of security), and re-host the feed. There\u2019s no reason this filtered feed should be re-hosted on its own when it could instead compile a CEL expression it receives from a client:

!item.tags.exists_one(t, t == "squid")

\u2026and just return items where that expression returns true. User-defined CEL expressions are non-Turing-complete and safe to execute, so I can use them in lieu of parsing and documenting some feed-specific filter API. Different requests, passing differenct CEL expressions, can fetch differently filtered feeds from the same endpoint.

I will probably never convince anyone to host a feed that behaves this way, but that\u2019s the neat thing about syndication: I can mirror or aggregate other feeds in a feed of my own that provides the interface I want. No need to ask anyone else to implement anything.


  1. Podcasts are a good example of syndication working as expected: a system of RSS/Atom feeds means your podcast is available in whatever listening environment you like, including as a raw audio file. Platform-exclusive podcasts are trendy but somewhat toxic.\u21a9\ufe0e

  2. Rewriting jsonfeed-wrapper from something App Engine specific to something that could equivalently provide a Cloud Function was super simple because of how each of those Cloud Services register the code\u2019s entry points.

    Google App Engine\u2019s Python runtime requires a WSGI-compatible object (in this case, a bottle application). Google\u2019s Cloud Function runtime, on the other hand, asks you to define a function for handling requests.

    The platform-ambivalent rewrite of jsonfeed-wrapper exports interfaces for constructing both, and everything but the request/response handling code is shared. Moreover, since instantiating a bottle app doesn\u2019t do the hard work of actually running it, leaving the app definition in the Cloud Function code has a minimal impact on performance.

    The result: each of my jsonfeed-wrapper applications can be deployed either as a Cloud Function or as an App Engine app without changing a single line of code.\u21a9\ufe0e

", "summary": "Sometime in the last ten years, while you were mourning the loss of Google Reader, we entered the golden age of the feed. A brief pitch for web syndication and an overview of the tools I've built for generating and working with JSON Feeds.", "date_published": "2020-11-02T02:50:26.425000+00:00" }, { "id": "./gen/cloud-function-pdf-processing.html", "url": "https://lukasschwab.me/blog/./gen/cloud-function-pdf-processing.html", "title": "Processing PDFs with Cloud Functions", "content_html": " Processing PDFs with Cloud Functions « blog

Processing PDFs with Cloud Functions

Two months ago I found the trove of papers on Nancy Leveson\u2019s MIT homepage1, which planted the seed of a thought. Academic homepages are full of interesting reading (unpublished rambles; administrivia; pre-prints, the published versions of which are inaccessible behind paywalls). Academic homepages are also woefully ill-maintained. Why not scrape them?

They\u2019re good candidates. Most are hand-written static sites with simple DOMs. Most are reachable from central departmental indices.

In the days after I found Dr.\u00a0Leveson\u2019s homepage, in the breaks between her papers, I threw together a scrapy spider for pulling PDF URLs a department at a time, and for jotting those URLs down into text files\u2026 but that\u2019s where the project stalled out.2

https://arxiv.org/pdf/1711.02226.pdf\nhttp://proceedings.mlr.press/v70/carmon17a/carmon17a.pdf\nhttp://proceedings.mlr.press/v70/namkoong17a/namkoong17a-supp.pdf\nhttps://web.stanford.edu/~jduchi/projects/SinhaDu16.pdf\n...

My scraper was fast, but I had no interest in actually downloading tens of gigabytes of PDFs\u2013\u2013tens of thousands of files!\u2013\u2013that I would, realistically, never read. The code would be annoying to write and molassify my home network. I moved on to other work.

But what is cloud computing for, if not the senseless accumulation of online stuff? Google Cloud Platform (or the competing cloud platform of your choice) has everything we need to store and index the papers my scraper discovers:

I haven\u2019t used Cloud Functions much in the past (I often reach for App Engine out of habit, even when Cloud Functions will suffice). My impressions from this project:

Advantages Disadvantages
+ Concurrency without code
+ Easy IAM permissioning
+ Built-in event triggers
+ Multi-language pipelines
- Awkward local debugging
- Awkward remote debugging
- Multi-minute deployments

Instead of focusing on their shortcomings, we\u2019ll walk through what they handled well: I\u2019ve indexed more than 20,000 PDFs\u2013\u2013just over 20 gigabytes\u2013\u2013and extracted 500 megabytes of plain text.

$ gsutil ls "gs://documents-repository/**" | grep ".pdf$" | wc -l\n  20243\n\n$ gsutil du -e "*.txt" -sh "gs://documents-repository/"\n20.07 GiB    gs://documents-repository\n\n$ gsutil du -e "*.pdf" -sh "gs://documents-repository/"\n498.76 MiB   gs://documents-repository

All this without using ssh to connect to a server, configuring a network, sharing a client among concurrent routines, or spending more than $1.00.

Uploading documents

Our first challenge: process the old files of jotted-down URLs and persist the PDFs. Several properties of this problem make Cloud Functions attractive.

  1. I can pipe the PDFs straight from the network into Cloud Storage without holding an entire file in memory.

  2. Since there are so many PDFs, I\u2019d like to download as many of them concurrently as possible. A highly concurrent program on a single device might end up I/O-bottlenecked, but each Cloud Function has its own network resources! Need more concurrent uploads? GCP will automatically provision more Cloud Function instances.

  3. Since invoking a Function is lightweight\u2013\u2013a POST request with a URL\u2013\u2013we can eventually invoke it directly from my scraper instead of writing PDF URLs to an intermediate file. If the scraper code had to download and upload files itself, it\u2019d be unworkably slow.

  4. The scraper doesn\u2019t validate the URLs; they end with .pdf, but they could point at anything. Some may lead to sites that no longer exist. Some may point at redirects. Since we have few guarantees about the quality of our input, we don\u2019t have to bother managing failures in individual uploads. Let them fail!

We can split uploading documents into two scripts: a Cloud Function which takes a single URL and pipes the PDF to Cloud Storage, and a script that turns my scraped files into invocations of that Cloud Function.

Piping PDF data from a response body into GCS reminded me of Chris Roche\u2019s post on splitting io.Readers Since this Cloud Function is self-contained and provides a neatly defined interface ((url) => pdf), we don\u2019t have to worry about our upload language being suitable for analyzing/indexing the PDFs. We\u2019re free to choose whatever language fits our upload style. In Go:

// Package nmt contains a Cloud Function for streaming documents to GCS.\npackage nmt\n\nimport (\n    "context"\n    "encoding/json"\n    "fmt"\n    "io"\n    "net/http"\n    "net/url"\n\n    "cloud.google.com/go/storage"\n)\n\n// message is the data to parse from an invocation body. At this stage, there\n// is no metadata besides a PDF URL.\ntype message struct {\n    URL string `json:"url"`\n}\n\n// Name for the bucket to which documents should be written. This Cloud\n// Function assumes the bucket is in the same GCP project.\nconst bucketName = "documents-repository"\n\n// Ingest streams the file at a URL to Cloud Storage.\nfunc Ingest(w http.ResponseWriter, r *http.Request) {\n    // Parse invocation body for PDF URL.\n    var decoded message\n    if err := json.NewDecoder(r.Body).Decode(&decoded); err != nil {\n        http.Error(w, "Failed to decode message", 400)\n        return\n    }\n    filename, url, err := parseURL(decoded.URL)\n    if err != nil {\n        http.Error(w, "Failed to parse URL", 400)\n        return\n    }\n\n    resp, err := http.Get(url.String())\n    if err != nil {\n        http.Error(w, "Failed to request URL", 404)\n        return\n    }\n    defer resp.Body.Close()\n    if resp.Header.Get("Content-Type") != "application/pdf" {\n        http.Error(w, "Not a PDF", 404)\n        return\n    }\n\n    // Create a storage client.\n    ctx := context.Background()\n    storageClient, err := storage.NewClient(ctx)\n    if err != nil {\n        http.Error(w, "Failed to create storage client.", 500)\n        return\n    }\n    defer storageClient.Close()\n    // Initialize a storage object by creating a writer for it.\n    writer := storageClient.Bucket(bucketName).Object(filename).NewWriter(ctx)\n    defer writer.Close()\n\n    // Return a 200 status.\n    w.WriteHeader(http.StatusAccepted)\n    // Pipe PDF body to Cloud Storage.\n    writer.ContentType = resp.Header.Get("Content-Type")\n    written, err := io.Copy(writer, resp.Body)\n    if err != nil {\n        fmt.Println("Error encountered when piping to Cloud Storage", err)\n    }\n    fmt.Printf("Wrote %d to Cloud Storage\\n", written)\n}\n\n// parseURL parses a URL. The returned filename is the URL host and URL path,\n// concatenated. GCS treats the `/` tokens as folder separators.\nfunc parseURL(input string) (filename string, url *url.URL, err error) {\n    url, err = url.Parse(input)\n    filename = fmt.Sprintf("%s%s", url.Host, url.Path)\n    return filename, url, err\n}

Less than a hundred lines, and not a buffer in sight! That io.Copy() call\u2013\u2013from the HTTP response to the newly initialized GCS object\u2013\u2013is doing the bulk of the work. Almost everything else is error handling.3 There\u2019s pleasantly little tool-specific boilerplate, so this could be refactored into an HTTP handler function on a webserver if we saw fit.

Once this Cloud Function\u2019s deployed, we can invoke it in a fairly tight loop: additional Cloud Function instances are automatically provisioned to manage the load. I wrote a short aiohttp Python script for this. Uploading 20 GB of PDFs\u2013\u2013hundreds of PDFs at a time\u2013\u2013takes just minutes. Conveniently, the object names are their URLs.

Extracting text

Extracting text from PDFs is hard. PDFs are essentially visual documents; they\u2019re meant to be read visually rather than parsed programmatically. There are, broadly, two ways of turning them into plain text:

OCR is overkill for our PDF-indexing use case; extracting a bag of words will do just fine, and there\u2019s such a wealth of PDFs on academic homepages that we can satisfy ourselves with indexing most of them. That\u2019s not to say this text extraction is a simple thing to build yourself, or even easy to solve with libraries in a variety of languages: I struggled for hours with a Go module before giving up and switching to Python.

This is a sweet feature of a radically modular infrastructure: for a given stage of our data pipeline, we\u2019re free to pick the language with the best support. When we need to pull text from PDFs, we can pick the language with the most effective published tools (like we picked a language that suited our ingestion strategy). Python has plenty; pdfminer.six is decent. In a cloud function:

from google.cloud import storage\nimport tempfile\nfrom pdfminer.high_level import extract_text\nfrom flask import abort\n\n\ndef main(request):\n    """main is theCloud Function entry point. It downloads the specified\n    existing PDF from GCS to a temporary location, extracts the text from that\n    PDF, then uploads that text to a new GCS object.\n\n    request -- the Flask request that invoked this Cloud Function.\n    """\n    # Parse object name from request.\n    request_json = request.get_json()\n    objectName = request_json['object']\n    assert objectName.endswith(".pdf")\n    print("Got request to handle", objectName)\n    # Connect to GCS bucket.\n    client = storage.Client()\n    bucket = client.bucket("documents-repository")\n    # Initialize temporary file for downloaded PDF.\n    pdf = tempfile.NamedTemporaryFile()\n    try:\n        # Download blob into temporary file, extract, and uplaod.\n        bucket.blob(objectName).download_to_filename(pdf.name)\n        return extract(bucket, objectName, pdf.name)\n    except Exception as err:\n        return abort(500, "Exception while extracting text", err)\n\n\ndef extract(bucket: storage.Bucket, objectName: str, pdf: str):\n    """extract pulls text out of a downloaded PDF and uploads the result to a\n    new GCS object in the same bucket.\n\n    bucket -- the GCS bucket to which the resulting text will be uploaded.\n    objectName -- the prefix for the uploaded text object. Usually this is the\n        object name for the processed PDF.\n    pdf -- the filename for a downloaded PDF from which to extract text.\n    """\n    # TODO: silence pdfminer noisy logging.\n    text = extract_text(pdf)\n    # Upload extracted text to new GCS object.\n    dest_blob = bucket.blob(objectName + ".txt")\n    dest_blob.upload_from_string(text)\n    return text

Downloading a PDF to a temporary file is clumsy, but the published tools expect filenames (and, as I said, they\u2019re complex enough to dictate my implementation). Like the upload Cloud Function before it, we can invoke this one in a tight loop using the scraped URLs from before\u2026 et voil\u00e0! After a few minutes, the pdfminer output for each stored PDF is tucked next to its corresponding PDF in the bucket.

Listening for storage events

This extraction strategy works, but it involves manually triggerig both stages: first, we trigger PDF ingestion; second, after the upload is finished, we separately trigger our text extraction function. Instead, we can just trigger the ingestion, and have the \u201cafter the upload is completed\u201d event trigger our Function automatically/immediately.

Google Cloud Storage Triggers are a neat Pub/Sub interface for invoking Cloud Functions. Instead of manually announcing \u201chey, I uploaded this object, it\u2019s ready for processing,\u201d the Cloud Function can consume the GCS-published finalize event marking the Storage object creations and updates. Refactoring our code from before:

def on_finalized(event, _):\n    """on_finalized is the Cloud Function entry point for handling GCS object\n    finalized events. It downloads the specified PDF from GCS into a temporary\n    file, extracts the text from that PDF, then uploads that text to a new GCS\n    object.\n\n    event -- the received GCS event. Includes the bucket name and the name of\n        the finalized object.\n    """\n    bucket = event['bucket']\n    objectName = event['name']\n    # Skip non-PDF files: this function writes to the bucket it watches.\n    if not objectName.endswith(".pdf"):\n        print("Skipping request to handle", objectName)\n        return\n\n    print("Extracting text from", objectName)\n    # Connect to GCS bucket.\n    client = storage.Client()\n    bucket = client.bucket(bucket)\n    # Initialize temporary file for downloaded PDF.\n    pdf = tempfile.NamedTemporaryFile()\n    try:\n        # Download blob into temporary file, extract, and uplaod.\n        bucket.blob(objectName).download_to_filename(pdf.name)\n        extracted = extract(bucket, objectName, pdf.name)\n        print("Success: extracted {} characters".format(len(extracted)))\n    except Exception as err:\n        print("Exception while extracting text", err)

The extract function\u2013\u2013and, indeed, everything but the code to pull the objectName from the Pub/Sub message\u2013\u2013is unchanged. With this update, there\u2019s no need to invoke anything but the ingestion function: once a PDF has been streamed into GCS object, the text extraction will automatically kick off; we don\u2019t need any second pass over scraped URLs.

We can daisy-chain this further by writing Cloud Functions invoked by finalize events on .txt objects, or publish events to Pub/Sub topics linking successive pipeline stages. TFIDF? Elasticsearch? Write the next stage in Typescript? The sky\u2019s the limit.


  1. I skimmed most of the PDFs listed on Dr.\u00a0Leveson\u2019s site. If you\u2019re interested in software-system safety analysis, my personal favorites were

    • \u201cSystems Theoretic Process Analysis (STPA) of an Offshore Supply Vessel Dynamic Positioning System.\u201d I skipped a lot of the topic-specific detail, but its STPA overview is one of the best I read and its example (maintaining the relative positions of two ships without running either aground) is memorable.

    • \u201cSoftware Deviation Analyis: A \u2018Safeware\u2019 Technique.\u201d Discusses modeling a software system and, well, kind of Chaos Monkeying it: see what happens when different combinations of the software\u2019s controls are violated.

    • \u201cInside Risks: An Integrated Approach to Safety and Security Based on Systems Theory.\u201d A strong case for cross-applying principles from system safety (Leveson\u2019s primary focus) into information security (my primary focus these days).

    Leveson doesn\u2019t index them by title, but they\u2019re in there!\u21a9\ufe0e

  2. The scraper that produces the lists of PDF URLs is pretty janky. It requires some tweaking for each academic department. The abridged code:

    import scrapy\nfrom scrapy.linkextractors import LinkExtractor\n\nclass CustomLinkExtractor(LinkExtractor):\n  def __init__(self, *args, **kwargs):\n      super(CustomLinkExtractor, self).__init__(*args, **kwargs)\n      # Keep the default values in "deny_extensions" *except* for PDFs.\n      self.deny_extensions = [ext for ext in self.deny_extensions if ext != ".pdf"]\n\nclass Spider(scrapy.Spider):\n  name = "nmt-spider"\n  # allowed_domains limit the domains to be scraped rather than the PDF links\n  # to be extracted.\n  allowed_domains = ['department.domain.edu', 'personal.sites.domain.edu']\n\n  def start_requests(self):\n    # An iniial entry point; usually a faculty index.\n    yield scrapy.Request(url="department.domain.edu/faculty")\n\n  def parse(self, response):\n    pdf_extractor = CustomLinkExtractor(allow=r'.*\\.pdf$')\n    with open("jots.txt", "a") as f:\n      for pdf_link in pdf_extractor.extract_links(response):\n        f.write(url + "\\n")\n    for link in LinkExtractor.extract_links(response):\n      # ...filter out problematic links here before yielding.\n      yield response.follow(link, callback=self.parse)

    Running scrapy runspider scrape.py yields a file jots.txt full of PDF URLs.\u21a9\ufe0e

  3. I made a conscious effort to stick to \u201cThe Kyoto School of Go Nihilism\u201d\u2013\u2013every err checked, every deferred function executed.\u21a9\ufe0e

", "summary": "A personal project of mine requires storing and processing tens of gigabytes of scraped files. I discuss using polyglot Cloud Functions as a rudimentary data pipeline: first, a function for highly parallel file uploads; second, one for PDF text extraction.", "date_published": "2020-10-19T05:24:26.091000+00:00" }, { "id": "./gen/product-ontology.html", "url": "https://lukasschwab.me/blog/./gen/product-ontology.html", "title": "Web App Product Ontologies", "content_html": " Web App Product Ontologies « blog

Web App Product Ontologies

Web apps with user interfaces lend themselves to object-orientation. It\u2019s in the middle of the acronym: Document Object Model. This isn\u2019t to say you a \u201cfunctional\u201d product model is impossible, but visualizing a function involves expressing it with an object; to shoehorn a product model into a UI is to congeal it into an assemblage of objects.

Articulating a good product model requires picking the most expressive classes of objects and describing how they\u2019re conceptually related: developing a product ontology.1

Formal XML-derived syntaxes have been developed for physical product ontology, but the primacy of physical components makes them awkward fits for describing the product models behind web applications. Whereas physical product ontologies describe concrete units\u2014an air purifier with one fan assembly and one filter assembly, working together\u2014a web product model is essentially nonconcrete until runtime.

Lee and Suh\u2019s meta-product ontology divides entities into physical and abstract entities.2 Web app product models rarely have equivalents to physical Objects (global and static), but deal heavily in SetOrClass and Datatype entities that are instantiated by users.

Rather than describing a collection of identifiable entities and their relations, then, a description of a web product specifies classes of entities and the relations between their classes. A user who udnerstands those classes and relations can use the product to express and control the concrete, identifiable stuff.

In some respects, this is an easier task than developing a formal language for describing air purifiers! Web product planning can borrow the notion of an underlying, describable ontology without falling into the mire of rigidity/nonrigidity, subassemblies, and so on. Because the product\u2019s implementation in code is an unambiguous and analyzable expression of the product\u2019s structure, a web product-modeler doesn\u2019t have to be as exhaustive.

What are your product\u2019s primary concepts, its objects under manipulation? When a user uses your product, what do they control?

I think of this as nailing down a product model\u2019s noumena, those stubborn concepts that persist independent of their representation. Whether or not they\u2019re expressed directly\u2014whether or not the user even conceives of the product in their terms\u2014these are the product\u2019s underlying objects. An app for scheduling events, no matter what interface exposes, works with some internal notion of what constitutes an \u201cevent;\u201d this is its primary noumenal class.3

Describing noumena and interdependencies early could kick off data model design before, or concurrently with, interface design. A well-expressed product model can be the joint precursor to both, rather than building one to fit the other.

That said, sketching a product ontology shouldn\u2019t be mistaken for data model design. The engineering task is to produce a data model that can unambiguously represent the space of permitted product model states. There may be multiple data models that achieve this. In my experience, a single product model will be implemented and reimplemented with a succession of data models before it\u2019s changed.

As a result, it\u2019s easy to think through other products\u2019 ontologies without thinking about the details of their implementation.

An example: Trello is essentially a tool for organizing cards representing records. Their product involves other nouns\u2014lists, boards, etc.\u2014these are derivative concepts for organizing and displaying cards. Regardless of whether you click on a card on a board, in a calendar view, or in the archived query view (when a card has been disassociated from its board/list), the card is readable and editable in exactly the same popover format. Easy to recall, easy to teach.

An unpleasant example: I can\u2019t think up a tight product ontology for Notion. Is it for organizing records (in table or gallery views)? Is it for organizing lightly interactive pages, like a wiki with embedded forms? Or am I organizing the sections of content on those pages, which are drag-and-drop rearrangebale? There must be some underlying consistent data model, but I can\u2019t find it in the fog of concepts and interactions.

A good product teaches its users to perceive its abstract product model, and to intuit based on their understanding thereof. While the noumena in that model may not be represented directly or consistently, users will learn them more effectively when they are.

Controlling noumena

We can turn sketched classes of product entities into a fistful of representational litmus tests. The product we build around our ontology will necessarily involve mechanisms for users to view, organize, and modify our noumena. Representing the entities in the product model directly and comprehensively should keep the user\u2019s model of the system state close to the system\u2019s internal state; keeping a user\u2019s understanding accurate should minimize corrective error.4

Consider your product as a control interface for each class of entities: how ergonomically can a user assess the state of an entity and cause precisely the represented change they intend? To that end\u2026

The Cognitive Dimensions of Notations framework6 is a list of digital product usability anti-patterns (with some neat properties, e.g.\u00a0pairwise independence). It may seem a stretch to think of web apps as \u201cnotations,\u201d but these antipatterns can appear anywhere users do composition (whether they\u2019re composing a vacation, their account settings, or a program). Users work to get a product to express their intent. Blandford and Green turned the Cognitive Dimensions of Notation into a set of usability litmus tests like the ones I propose above.7

Besides this usability-forecasting, a well-specified product model offers a nice starting point for breaking a product into reusable components. It might be sensible to build a single component to manage representing a noumenon in a variety of contexts; \u201ccards\u201d (neatly arranged into arrays; vertically more forgiving than table rows) are a common pattern for this. If certain contexts require the card to be too compact to be a good control surface, clicking on it can pop a more articulate modal:8 picking noumena that exist independent of outside context should mitigate the context-downside of hiding the rest of the page.

Comparative advantage

Instead of beginning with ontology, we could begin with functionalism: \u201cwhat do users want to do with our product?\u201d This doesn\u2019t avoid creating a product ontology, but it defers to an imaginary user\u2019s incumbent ontology. If a product would simplify a complex workflow, this approach cements into its heart an model just as unwieldy as the one it should replace.9

Focusing on user workflows too early also runs the risk of tailoring our data model too closely to a particular representation that achieves a very specific function. Since representations are immediately user-facing, they\u2019re more likely than the data models to be subject to early user feedback and iteration. Better to build a flexible data model and leave our UI easy to invalidate.

Directness and comprehensiveness are useful heuristics for UI options. Conversational interfaces are hip for their humanism, but a conversation with intense control requirements is frustrating: it\u2019s untransparent, which makes it error prone.

Winograd (1979)10 gets at something similar, which he terms \u201cthe subject domain.\u201d One difference: he treats it as a puzzle for programmers rather than a dedicated \u201cproduct\u201d function, so he places special emphasis on the difference between a product ontology and a data model:

This system [for organizing room assignments at a university], like every practical system, is about some subject. There is a world of rooms and classes, times and schedules, that exists completely apart from the computer system that is understood as referring to them. [\u2026] One of the primary tasks in programming is to develop a set of descriptions that are adequate for talking about the objects being represented. There are descriptions for things we think of as objects (e.g.\u00a0buildings, rooms, courses, departments) and also for processes (e.g.\u00a0the scheduling of events). These descriptions are relative to the goals of the system as a whole and embody a basic view of the problem. For example, what it takes to represent a room would be different for this system and for a system used by contractors in building construction.

All too often the development of descriptions in this domain is confused with the specification of data structures (which are in the domain of implementation). In deciding whether we want a course to be associated with a single teacher, or to leave open the potential representation of team-teaching, we are not making a data structure decision. The association of a teacher (or teachers) with a course may be represented in many different data structures in many different components of the system. One of the most common problems in integrating systems is that the components are based on different decisions in the subject domain, and therefore there is no effective way to translate the data structures.

Winograd argues

A programming system needs to provide a powerful set of mechanisms for building up and maintaining \u2018world views\u2019\u2014coherent sets of description structures in the subject domain that are independent of any implementation. Each component can then implement part or all of this in a way that will be consistent with both the structure of that component and the assumptions made in other components.

Those nineties product ontology projects might be seen as languages for this kind of specification, but with the big ambition of describing any system, any \u201cworld-view\u201d or set of relations. Teams seem to do this kind of work in documentation tools, but they rely on the total flexibility of linked text; Winograd argues docstrings are too local, too prone to inconsistency, to count.

I might just be reinventing domain-driven design. My \u201cnoumena\u201d captured by are classes and individuals, ontology components.

I feel strongly that \u201cnoumena\u201d was a poor choice, not least because it suggests a difference from \u201cphenomena\u201d for which there isn\u2019t a product analogy.


  1. This goal is complicated by the reflexive relationship between a product\u2019s ontology and its functionality. Some ontology might achieve a product\u2019s functional intent better than others\u2026 but the functions users desire are themselves formed through their understanding of the product model, and they can more articulately communicate desires in terms of the product concepts that are named for them. An ontology-axiology bind!\u21a9\ufe0e

  2. Lee, Jae-Hyun, and Hyo-Won Suh. \u201cOWL-based product ontology architecture and representation for sharing product knowledge on a web.\u201d In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, vol.\u00a048035, pp.\u00a0853-861. 2007.\u21a9\ufe0e

  3. I\u2019m self-conscious about \u201cnoumenon\u201d being a pretentious term, but there\u2019s a utility for me in setting up an unfamiliar category. Terms like \u201centity\u201d or \u201cobject\u201d are overloaded (esp.\u00a0\u201cobject,\u201d in an essay that\u2019s essentially about preferring object-oriented representations for noumena).

    It should be clear that we\u2019re not talking about the instance a user perceives and interacts with. A user\u2019s perception of the product model is wont to be inaccurate; whether there\u2019s really a correct perception of a product model feels like a question for metaphysics.

    A user\u2019s perception of the product model may be more or less close to the designed product model, though, and the control systems reading I reference supports my opinion that closeness here is good. The term \u201cnoumenon\u201d emphasizes the fallability of the UI phenomena you build.\u21a9\ufe0e

  4. I\u2019ve been poking thorugh \u00c5str\u00f6m, Karl Johan, and Richard M. Murray. Feedback systems: an introduction for scientists and engineers. Princeton university press, 2010.\u21a9\ufe0e

  5. \u201cPosition control\u201d and \u201cdirect control\u201d are terms taken from Abrecht, Blake, and N. G. Leveson. \u201cSystems theoretic process analysis (STPA) of an offshore supply vessel dynamic positioning system.\u201d Massachusetts Institute of Technology, Cambridge, MA (2016).\u21a9\ufe0e

  6. The Cognitive Dimensions are decently summarized on Wikipedia, but more clearly in Blackwell, Alan, and Thomas Green. \u201cNotational systems - the cognitive dimensions of notations framework.\u201d HCI models, theories, and frameworks: toward an interdisciplinary science. Morgan Kaufmann (2003).\u21a9\ufe0e

  7. Blandford, Ann, and Thomas Green. \u201cOSM: an ontology-based approach to usability evaluation.\u201d Note: the notion of user-private, device-private, and shared concepts butts up againts my idea of a \u201cnoumenon\u201d here, but more value-neutrally. In my mind, the noumena form a starting set of shared concepts.\u21a9\ufe0e

  8. Trello does this, but Airtable\u2019s modal view for a record is a great example from a more complex product. Sometimes the record is a row in a table; sometimes it\u2019s a card in a gallery; no matter what, clicking on it will reveal all the data for the record and allow you to directly modify any of it. No matter the representation, the user is one modal-expanding click away from a consistent control experience.\u21a9\ufe0e

  9. Additionally, users are nonhomogeneous; overfitting to your early users\u2019 mental models might be a mistake. This is especially true for teams with the ambition to shift up/down-market or stretch their product into new applications, or strong selection biases in their initial user base (e.g.\u00a0a bias towards users from a certain accelerator cohort).\u21a9\ufe0e

  10. Winograd, Terry. \u201cBeyond programming languages.\u201d Communications of the ACM 22, no. 7 (1979): 395.\u21a9\ufe0e

", "summary": "What are a product's central concepts? I consider planning products around a product model's \"noumena.\" Such an object-oriented organizing principle should mitigate corrective error and, hopefully, set you up for interpretable and reusable componentry.", "date_published": "2020-08-22T23:49:12.795000+00:00" } ] }