Jamie Brandon

jamie@scattered-thoughts.net

I've worked on query engines, storage engines, incremental view maintenance, streaming queries, declarative languages, end-user programming tools, IDEs, interpreters and compilers, and general performance engineering.

I'm happiest being aimed in the general direction of a hard problem and trusted to figure it out.

I'm based in Vancouver BC. I'm not interested in relocating but I'm happy to travel. I have Canadian and British citizenship.

employment

TigerBeetle 2022-2023. High-performance database for financial transactions, able to detect and recover from most hardware failures.

I was the main person working on storage engine performance and correctness. When I joined we were averaging 2,734 transfers/second with a peak (closed loop) latency of 14,749ms. Within three months we had it up to 120,397 transfers/second with a peak latency of 3,817ms. I also identified many bottlenecks that led to later performance improvements, and fixed some gnarly bugs (eg #390 which combined a race, a use-after-free, a hash collision, and a zig compiler bug).

Jamie is a lion you let loose on some of the toughest technical problems in your code base. Jamie is fiercely independent and will deliver on skunkworks projects. He cares deeply. You can trust Jamie to do it right, and see it through.

Joran Dirk Greef, CEO

 

Materialize 2019-2020. Incrementally maintained sql views over streaming data.

I wrote much of the sql frontend and ported >6m tests from sqlite/postgres/cockroachdb. I wrote the decorrelation pass, which I believe at the time was the best in the world (hyper was comparable, but had some nasty edge cases that I accidentally avoided). I also redesigned the in-memory storage format, halving our peak memory usage in benchmarks.

Fun fact: I was hired on a three month trial contract because Materialize were reluctant to have a remote employee. The trial was converted to full-time on my third day after I landed support for union, distinct, aggregates, and also found and fixed a bug that silently dropped duplicate rows.

Jamie was fundamental to building the foundation of Materialize. Not only was he a phenomenally productive engineer, he was able to synthesize lots of academic research to solve complex problems from first principles.

Arjun Narayan, CEO

 

RelationalAI 2017-2019. Declarative programming languages for building complex business models.

I developed the staged compilation approach used for the Rel language, allowing us to mostly write an interpreter while selectively jit-compiling hot operations. I wrote an early prototype of Rel where I figured out how to give sound denotational semantics to their higher-order relations and designed a simple type analysis that prevents logical contradictions. I also wrote several of their open-source libraries: blobs.jl for memory-safe zero-cost zero-copy deserialization, and rematch.jl adding pattern-matching to the Julia language.

Jamie is a first principles thinker with a deep understanding of systems and algorithms. He is fearless in the way he approaches problems. Jamie's research on staged compilation saved our team many person years of effort, and his training and support was crucial to our adoption of Julia. I hope to have the opportunity to work with him again soon.

Molham Aref, CEO

 

Eve 2013-2016. A sane end-user programming environment.

We built a declarative language with incremental re-evaluation, a live notebook interface, a gui library, an inspector and a debugger. The team started with just two engineers - mostly I worked on the language implementation while Chris worked on the IDE.

Jamie brought a degree of rigor and theoretical knowledge that took the company to places we never could have gone otherwise. From exploring the implications of language semantics to building high performance database engines, he didn't just help us get things done, he helped us get them done right. As a result, we were able to explore in weeks what could've easily taken years.

Chris Granger, CEO

consulting

Runway 2025. Timeseries spreadsheet for financial planning.

I was brought in to help with the performance of their timeseries interpreter as they were scaling up to bigger customers. I reduced the p50 recalculation time by 4x and killed several superlinear edge cases that were plaguing specific customers. I also helped set up continuous benchmarking infrastructure and expanded their existing continuous profiling to break costs down by customer and function, to ensure that they were able to catch future regressions.

 

Gel 2025. Database with a query language for returning hierarchical data.

I wrote a detailed review of their query language looking for sharp edges, inconsistencies, opportunities to simplify constructs, pointers to existing solutions in other query languages etc. I also discovered several serious performance bugs.

 

SuperDB 2024. Query engine for heterogeneous data.

I prototyped the architecture for their new vectorized query engine, which was rolled out incrementally by opportunistically replacing nodes in the query plan with vectorized operators and falling back to the scalar query engine for the remaining operators. Implementing the basic sargable operators gave them an immediate performance boost by reducing both the amount of data downloaded from the data lake and the amount of data flowing into the scalar query engine, while also leaving the door open to incrementally vectorize the rest of the language later. I added a fuzzer to compare the scalar and vectorized implementations, which also surfaced a number of crashes and infinite loops in the original scalar query engine.

 

XTDB 2022. Bitemporal database.

I was brought in as an advisor on their sql implementation to steer them clear of common pitfalls, stress-test weird sql edge cases, and point out which research papers actually work and are worth reading.

 

LogicBlox 2016-2017. Declarative programming language for business intelligence.

I wrote an experimental compiler that allowed writing gui programs on top of the Logiql language, reusing the existing incremental view maintenance to derive precise DOM updates.

selected projects

While I've never published a paper, my blog has managed to be cited by the GoogleSQL team in VLDB and by Martin Kleppmann in an ACM keynote.

 

Zest 2023 - ongoing.

A WIP language aiming to be both malleable and legible. Sound gradual typing. Borrow-checking without type-checking. Designed to have well-defined semantics for hot code reloading, without resorting mutable environments. Staged evaluation. Interpreted or compiled to wasm.

 

HYTRADBOI 2022 / 2025.

I single-handedly organized an online conference, with a total of 72 talks and 1010 attendees between the two events.

HYTRADBOI was the best bang-for-my-buck (both in time and money) event that I attended last year. This year looks equally great! If you are interested in PL or DB, it's worth it; if you like both it's a must see!

Sidenote: I think it's a great example of how to run an efficient remote conference!

Max Willsey, UC Berkeley

 

Internal consistency in streaming systems 2021

I demonstrated that popular streaming systems (kafka streams, flink) can produce large, cascading, and chaotic errors even on simple queries, leading to a situation where the vast majority of outputs are not only incorrect but are not even possibly correct for any set of inputs (eg outputting a negative stddev). I also explained the additional consistency requirement that would guarantee that outputs are sane.

Jamie Brandon's (@sc13ts) work on stream processors is like @aphyr's work on databases: empirically testing consistency. Very interesting.

Martin Kleppmann, author of Designing Data-Intensive Applications

 

From 0 to sqllogictest 2022.

As a challenge, I tried to write a sql query engine from scratch in a week that could pass all 4259065 tests in the public sqlite test suite. I managed 95.26%.

 

Imp 2019 - 2022.

Various experiments in creating a purely relational language. Supports higher-order queries and correlated subqueries, with a type system that ensures that every well-typed program can be lowered to a first-order relational dag. Gradually typed for interacting with unstructured data. Algebraic types and first-class modules fall out of the same type system.