Replacing Your Database with Beam Pipelines

(Aug-25 20:20 UTC)
-----

We use streaming Beam pipelines to index data into an object store (S3) and compute summaries of new data with periodic batch pipelines. These summaries are analogous to indices in a database and allow for efficient lookup along a fixed set of axes. The result is a scalable and inexpensive data storage solution for applications that serve data with limited query patterns.

Vinay Mayar
Senior Software Engineer at Expanse