Decoupling two workloads on an event platform

Context

An industrial data platform in aerospace, ingesting millions of events per day. Each event carries a progress state that a human operator moves through a fixed lifecycle:

to review → open → scheduled → under watch → closed

So two very different workloads travelled the same path: on one side a high-volume machine ingestion, continuous and indifferent to the odd second; on the other a low-volume human interaction, where every millisecond shows because an operator is sitting in front of a screen waiting.

Constraint

Both workloads shared the same code path and the same data model. Three consequences compounded.

First, state transitions were processed inline with ingestion. Every ingestion carried transition work it did not need, and every state update queued behind the ingestion backlog.

Second, calls to the operator-facing API competed with ingestion traffic for the same resources. A spike in ingestion, that is, a perfectly ordinary moment, directly degraded the user experience.

Third, the DynamoDB key design matched neither access pattern. The reads operators needed were not served by the keys, and therefore fell back to scans.

The diagnosis that mattered is none of those three symptoms taken alone. It was not “DynamoDB is slow”, nor “we need more Lambdas”: it was that two workloads with opposite characteristics had been modelled as one. For as long as that conflation held, every local optimisation merely moved the problem, because speeding up a shared path also speeds up what should never have been on it.

Decisions

Decouple ingestion from transition processing. When ingestion finishes processing an event, it publishes a notification to SNS instead of doing the transition work itself. SNS fans out to SQS, and SQS triggers a dedicated transition Lambda. Ingestion returns as soon as its own job is done, carrying nothing that isn’t its concern. The trade-off is explicit: progress state becomes eventually consistent with ingestion rather than immediately consistent. We accepted it because the consumer of that state is a human whose working loop is measured in minutes, not another machine about to make an automated decision within the second. The same decoupling would have been the wrong call had a downstream system depended on that immediate consistency.

Give concurrency to the queue rather than to code. Several events concerning the same entity could otherwise be processed simultaneously and produce an inconsistent final state. The reflex answer is a distributed lock: a lock table, leases, expiry timeouts, and the whole class of bugs that comes with them. We preferred to use the queue’s own guarantees. A MessageGroupId per entity means messages in the same group are processed strictly in order, one at a time, while different groups continue to be processed in parallel: ordering is guaranteed exactly where it matters and nowhere else, so without sacrificing overall throughput. A MessageDeduplicationId protects against duplicate publishes. This is the decision I am happiest with in the whole overhaul, not because it is clever but because it is boring: the guarantee is provided by the infrastructure, it is tested by AWS, and there is no home-grown coordination layer to maintain or to debug at three in the morning.

Reshape the DynamoDB table around the real access patterns. The partition key became the event’s unique identifier, the sort key its arrival timestamp. Two consequences are worth spelling out. First, each state change is written as a distinct item instead of mutating a single row: there is no read-modify-write cycle any more, and therefore no lost update when two processes cross. Second, state history comes for free: operators needed the complete history of an event’s states, and a query on the partition key returns it already ordered by date. No separate audit table, no second write path to keep consistent with the first. A feature we would otherwise have had to build became a property of the data model.

Isolate the human-facing API in its own Lambda. API Gateway routes to a dedicated function, separate from the ingestion path. Ingestion spikes no longer degrade the operator experience, and more importantly the two can be sized and reasoned about independently: the reserved concurrency, timeouts and memory profile of one no longer have to be a compromise with those of the other.

Replace scans with secondary indexes. Some operator queries did not align with the primary key. Rather than scanning the table and optimising at the margins, we added global secondary indexes shaped for those specific queries. The underlying principle is worth stating, because it governs everything else: in DynamoDB you model the table around the queries you know you will run. A scan on a hot path is not a query to optimise, it is the signal that the data model is wrong.

Outcome

The two workloads no longer compete for the same resources: an ingestion spike, that is, an ordinary day, no longer degrades the operator experience, and the two paths can now be sized and reasoned about separately. Per-entity processing order is guaranteed without a single line of coordination code, and therefore without the class of bugs that comes with distributed locks. As for state history, which would otherwise have required dedicated development with its own write path, it became a plain property of the data model.

What this overhaul confirmed goes beyond the particular case: when two workloads with opposing constraints share a path, no optimisation of that path can satisfy both. You have to separate them first, after which the decisions become simple, because each half now has only one goal to serve.