0010: dida, live repls, query planning for streaming, rust allocators, more zig goto, database resources, guix on mac, criticising people's work, pay what you want

Published 2021-05-18

The main thing I've been working on is a port of differential dataflow. I have most of the basics sketched out with naive implementations and I'm currently working on implementing progress tracking correctly, at which point simple examples like graph reachability should start working.

Github doesn't have any way of saying "give all sponsors read-only access to this private repo" short of making a brand new github organization and manually spamming everyone with invites. I will figure something out eventually. But for now here are the contents of the readme:

Dida is a (WIP) library for streaming, incremental, iterative, internally-consistent computation on time-varying collections.

The jargon-free version: You write code that manipulates collections using familiar operations like map, join and loop. You run the code on some input and get some output. Then when the input changes, you get changes to the output, much faster than recomputing the whole thing from scratch. (And the outputs will be correct!)

Dida is heavily based on differential dataflow. Why start a new implementation? The goals are (in roughly priority order):


On a related note, I wrote up a draft of some recent experiments on making live repls behave. This is something I need to figure out in order to have interactive dida examples and visualizations on a web page.

Both the async version and the threaded version work in native code, but both have issues in wasm:

For small examples in wasm I can always fall back to copying all the data to a webworker every time the interpreter resets, but it would be nice to figure out something that works for real datasets and uses the same code for both native and wasm.


I wrote up some notes on why query planning for streaming systems is hard.

I don't plan to work on this any time soon - this post is more of a "note to self" so that when I do get around to working on this I remember all the problems I've already thought of.


In my assorted thoughts on zig and rust post I mentioned the Rust Allocators Working Group which is working on adding custom allocators to the rust standard library.

After reading this recent discussion on the rust internals mailing list it seems like the kind of tricks I'm using for the live repls will still not be possible in rust, because they require passing around not just custom allocators but non-global allocators.


The zig goto saga continues. Someone had great success writing an interpreter using tailcalls instead of using the traditional goto pattern, and zig already supports tail calls.

I have no expertise on this subject at all, but I'm fascinated.


I occasionally get asked for good resources on how databases work. Andy Pavlo's course is my favourite, and the red book is also pretty good.


I had assumed that guix' lack of support for mac was for ideological reasons, but it seems that there might be legal obstacles too.


Some of my work (eg) is strongly critical of other peoples work. I think this kind of work is important - it helps provide the gradient that software quality climbs. But it's also easy to do a lot of damage.

A trick I've found useful lately to find the right balance is to imagine reading the article to the author of the work I'm criticising. I'll still point out things that I think are wrong, but it strips out most of the negative adjectives and interpretation and leaves the evidence to stand by itself.


Github launched pay-what-you-want for sponsors, so I've enabled that with a default value of $20 and removed all the fixed tiers.


They are designed to focus attention but don't train you to overcome the obstacles to being focused. They are fun but don't tend to make a person more interesting.

Talking about video games, but also applies to much of modern consumer tech.

From You by Austin Grossman