February was really broken up by Systems Distributed so not as much coding or papers to talk about as I'd like. The talks were all recorded but I don't know when they'll be published yet.
- tigerbeetle
- table sizing
- bench harder
- wasm first steps
- sycl vancouver
- reading
- breathing for warriors
- move your dna
- the molecule of more
- how to decide
- slouching towards utopia
table sizing
LSM trees have in-memory buffers that get periodically compacted into on-disk tables. In tigerbeetle, the on-disk tables are 64mb and the buffers vary in size depending on the tree but the largest was 4kb.
Way back in November I was worrying about the relative sizes. The size of the buffer dictates how often we need to do compaction. The size of the tables dictates how expensive compaction can be. For every 4kb of incoming data, we might have to write up to 8 * 64mb of data to disk during compaction. That's a terrible amount of write amplification.
I finally got around to fixing this, making the table size match the buffer size and then increasing the buffer size to match what we expect to eventually use in production.
This had no effect on the performance in our benchmark, which was puzzling. DJ realized that the way that data is generated in the benchmark makes every index append-only, and so it's not actually stressing compaction at all. I tweaked the data generation a little to force just one index to have to compact. Throughput dropped from 129k transfers per second to 14k, and then applying the table size patch brought it back up to 83k. So the table sizing was a big problem but we actually weren't measuring it before.
bench harder
With that in mind I spent some time fleshing out the benchmarks to try to avoid having similar blind spots in future. I also extended them to measure end-to-end latency per transfer in addition to the processing latency per batch, and to allow varying the offered load so that we can draw throughput-latency curves rather than just getting a single point.
The new workload stresses compaction and caching a lot more so the overall throughput is lower, but King has been working through the compaction todo list this month and clawing some of the throughput back. At present I'm seeing 88k transfers per second with the default benchmark settings, with a p90 batch latency of 47ms and a p100 batch latency of >4s (the latter being from checkpointing the superblock and client tables, which is not yet incremental).
I also tried running a much longer benchmark - ingesting a billion transfers as fast as possible. At some point there was a very sharp phase-change in performance, with throughput dropping to 8k t/s and p100 batch latency though the roof. It also bounces between short periods of high write bandwidth (presumably compaction) and long periods of high read bandwidth. I haven't had a chance to explore this properly yet, but I speculate that what is happening is that once the database is large enough the bloom filters alone don't fit into the cache, and then the transfer id uniqueness checks end up needing to hit the disk per transfer instead of mostly being negative bloom filter lookups. I know that this can a problem for rocksdb and they offer partitioned bloom filters to address this, but that seems like a lot of complexity to add to tigerbeetle. Hopefully we just need to better tune cache sizes based on the maximium database size.
wasm first steps
I've meaning to learn more about webassembly for a while. For all the standard reasons - targetting the web with sane languages, portability across many platforms, language-agnostic sandboxing - but also because it's becoming an interesting middleware target. It's much easier to analyze and transform than x86, and much more stable than llvm ir. Projects like tangle and asyncify are just the start.
I hand-ported zig's wasm2c to zig (hand-ported because I thought I'd understand the code after translating it line by line) and hooked it up the wasm spec testsuite. For a codebase that was stripped down and specialized to a single input it does pretty well on the tests, passing the tests for 796/1125 modules (ignoring the simd tests, which are definitely not supported).
More to come on this front when I have time.
sycl vancouver
Matt Knight is running a Vancouver version of the software you can love conference.
I'll probably be there, since it's a short walk from my apartment. Just waiting on other plans to shape up before committing to a ticket.
I recommend attending, because if people come here then I don't have to go to them :)
breathing for warriors
Possibly the worst book I've ever read. Incredibly repetitive, barely coherent, often self-contradictory.
I'm looking for recommendations on a book about breathing techniques that is not awful.
move your dna
Not terribly well supported by its own citations, but has a couple of interesting suggestions.
the molecule of more
Did you know that everything is dopamine? Pop trash, avoid.
there are places in the world where rules are less important than kindness
A collection of newspaper columns written by a famous physicist. Fun easy reading. Was welcome on the flight to cape town where my brain was too foggy for anything serious.
how to decide
The second book by Annie Duke after thinking in bets.
It's well written. Doesn't contain anything surprising if you've had any prior exposure to the subject but would make a good introduction if you haven't.
slouching towards utopia
I nearly dropped it because the intro was so meandering, but it shapes up once it gets into the meat of the subject. It doesn't seem to have a central message or theme but each chapter was interesting in its own right.
I'm not sure how much of the non-economic history to put weight in. Eg after Japan invaded China, the US was under pressure to limit their expansion. They froze all Japanese assets in the US, which was supposed to be a moderate slap on the wrist. But all the other oil-producing countries would only accept payment in US dollars so the asset freeze amounted to a total oil embargo. That's how the author presents it. But when I looked this up I found that the US also stopped exporting oil to Japan and denied them access to the Dutch East Indies, which sounds like a very deliberate oil embargo.
The economic parts are presumably more accurate. Probaby most interesting was the authors explanation for the rise of neoliberalism, which he lived through and was somewhat mystified by at the time.