Thoughts on benchmarking streaming systems

Published 2021-09-06

I spent some time thinking about how to benchmark streaming systems. Though I didn't go through with the project, the notes might still be useful for others.

What is the point of benchmarking?

Output of a good benchmark:

The above output is per hardware configuration, per query set, per data set. Far too many variables to explore exhaustively. We need a model of performance.

Existing public benchmarks do none of this. They usually contain one (simple) query, one data set, one hardware config. They're only useful for marketing.

Benchmarking practices:

Focus on benchmarking vertical scaling first:

Micro benchmarks:

Macro benchmarks: