Overview
DDIA is aimed at experienced programmers who want to make informed choices regarding performance, scalability and integrity of data persistence at scale, or are wanting to more effectively use their existing stacks. I recommend the book for those audiences.
The book is long and information dense (which I appreciate) but is more of a survey. The reader will not be an expert in any one tool by the end, but you will cover the fundamental concepts, concerns and limitations to set your expectations of what the persistence layer is providing vs what the application software remains responsible for.
As someone who has been using relational databases for decades, writing SQL queries and diagnosing and improving query performance, I still learned things from the sections applicable to RDBMS, such as around isolation levels. Sprinkled throughout the book are mentions of which products employ certain strategies, but the book is focused on identifying the tradeoffs and intended usage, and does not promote one product or type of solution over the others. Every chapter has copious footnotes for any topic you want to dig deeper in to.
One footnote link on database performance for developers: https://use-the-index-luke.com/sql/preface
Impressions
If your needs donāt require scaling out your persistence layer, keep it simple. (Just use Postgres). Measure, donāt assume where your bottlenecks are. Scaling read-only copies is straight-forward (as evidenced by cloud managed databases offering this with a button click). After adding multiple writer instances the data layer can no longer be treated as a black box; understanding the capabilities and limitations becomes a concern of the code base. Modularize so that only high volume data is distributed, leaving the lower volume parts of datasets simpler.
Application software often considers it acceptable to abort or rebuild in the face of unexpected faults. Reliable data storage systems, on the other hand, must pay close attention to unlikely edge and error conditions such as concurrent calls with transactions that roll back, or reaching consensus in the face of network glitches, server crashes, and misaligned clocks. A one in a million occurrence is no longer infrequent running at high scale. (Jepsen tests distributed system vendors against their safety claims).
By the end of the book we come full circle comparing similarities between responsibilities of databases vs distributing data across federated systems. Indices start to look similar to derived data conveyed via asynchronous event logs with idempotent writes.
While federation addresses read-only querying across several systems, it does not have a good answer to synchronizing writes across those systems. We said that within a single database, creating a consistent index is a built-in feature. When we compose several storage systems, we similarly need to ensure that all data changes end up in all the right places, even in the face of faults. Making it easier to reliably plug together storage systems (e.g., through CDC and event logs) is like unbundling a databaseās index maintenance features in a way that can synchronize writes across disparate technologies
When copies of the same data need to be maintained in several storage systems in order to satisfy multiple access patterns, you need to be very clear about the inputs and outputs. Where is data written first, and which representations are derived from which sources?
If it is possible for you to funnel all user input through a single system that decides on an ordering for all writes, it becomes much easier to derive other representations of the data by processing the writes in the same order. This is an application of the state machine replication approach that we saw in āConsensus in Practiceā. Whether you use CDC or an event sourcing log is less important than the principle of deciding on a total order.
In principle, derived data systems could be maintained synchronously, just as a relational database updates secondary indexes synchronously within the same transaction as writes to the table being indexed. However, asynchrony is what makes systems based on event logs robust. It allows a fault in one part of the system to be contained locally, whereas distributed transactions abort if any one participant fails, so they tend to amplify failures by spreading them to the rest of the system.
I also appreciated the inclusion of the last chapter, Doing the Right Thing, on ethical concerns with data, which our industry often gives little consideration to.
most users have little knowledge of what data they are feeding into our databases or how it is retained and processedāand most privacy policies do more to obscure than to illuminate. Without understanding what happens to their data, users cannot give meaningful consent. Often, data from one user also says things about other people who are not users of the service and who have not agreed to any terms. The derived datasets that we discussed in the last few chaptersāin which data from the entire user base may have been combined with behavioral tracking and external data sourcesāare precisely the kinds of data that users cannot meaningfully understand.
Moreover, data is extracted from users through a one-way process, not a relationship with true reciprocity or a fair value exchange. There is no dialogue, no option for users to negotiate how much data they provide and what service they receive in return. The relationship between the service and the user is asymmetric and one-sided; the terms are set by the service, not by the user