A cache that fought itself

This follows on from A live screen is mostly not live. That post is about why the cache exists. This one is about what it did once it existed.

The cache worked. Every page served the right data, the warehouse behind it was no longer in the request path of anything that redraws every ten seconds, and the compute bill on the warehouse side had fallen the way the design said it would.

Then I looked at the Redis dashboard and it said 600 million commands a month.

Nothing was broken. Every one of those commands returned the right answer. That is what made it hard to see: a cache with a correctness bug announces itself, and a cache with an efficiency bug just quietly works, at a price nobody is looking at yet.

Why serverless makes this worse

The application runs on serverless functions. That means an instance can appear, answer a few requests, and disappear, and the next request may land on a brand-new one with empty memory. Whatever a normal server would keep warm in process for hours, this platform throws away constantly.

So the shared cache is not an optimisation layered on top of a warm process. It is the warm process, externalised. Every instance that starts cold goes to Redis for everything it needs, and there can be hundreds of starts a day. That multiplies whatever waste is in the access pattern by the number of instances, and the number of instances is not something you control.

This is the lens for everything below. None of the four problems would have mattered much on one long-lived server. All of them mattered on a fleet of short-lived ones.

Four kinds of waste, not one

It would be tidier if there were a single cause. There were four, and they were independent, which is why fixing any one of them on its own looked like it had not worked.

Chatter

A shop-floor tracking screen polls every ten seconds, and each poll enriches about sixty production orders from the cache. The enrichment was written the obvious way: one GET per order, per key family, and there were three key families.

Worse, every GET was two commands. The cache layer supports values too large for a single Redis record by splitting them into chunks with a small manifest key that records the count. The read path probed for that manifest first, every time, and for the small per-order entries it never existed. Half of all commands were probes that always missed.

Call it 360 commands per poll. Six polls a minute, ten screens, a working month: that alone reproduces most of the 600 million.

The previous post already made the point that an MGET collapses this and that on a fixed-price tier it saves no money, which is exactly why it survives. I wrote that paragraph and then did not do it for three weeks. One MGET for the batch, a second MGET of manifests for only the keys that missed, and 360 commands became about six.

A cache that fought itself

This is the one worth the post.

The heaviest cached queries read from a planning schedule that a scheduling engine rebuilds several times a day. When it rebuilds, everything derived from it is stale. So each cached result carries a tag identifying which engine run it came from, and a read whose tag does not match the current run is treated as a miss.

The tag itself was derived by each instance from a cheap query, and then remembered in that instance's memory for twelve minutes. The comment next to the code argued, correctly, that noticing an engine rerun twelve minutes late is harmless. Nobody needs a plan board to refresh within the minute.

What the comment missed is that the tag is not only a freshness signal. It is the thing that decides whether a shared entry is usable. And "shared" is the problem.

For up to twelve minutes after every engine run, instance A with the new tag writes an entry, instance B with the old tag rejects it, re-runs the query against the warehouse and overwrites the entry with the old tag, and A then rejects that. The cache for the application's most expensive queries was not merely cold once after each run. It was actively invalidating itself, and every rejection was another full scan of a large table.

The fix was to publish the tag through Redis with a sixty-second window, so that every instance reads the same value and the fleet agrees. The general lesson is the part I want to keep:

Any cache whose validity depends on a token derived per instance will thrash on a shared tier.

It does not matter how reasonable the per-instance derivation is. If two instances can hold different opinions about what "current" means, and both can write, they will take turns being wrong about each other.

A second change made the same class of bug structurally impossible for the four heaviest queries: the engine-run stamp went into the key instead of alongside the value. A new run is simply a new key. There is nothing to compare, nothing to reject, and no way to serve a stale result, because a stale result lives under a name nobody asks for any more.

Setup replayed per instance

Twenty-eight routes ran table-creation and schema-ensure statements on first use. On a long-lived server that is once per deploy. On serverless, "first use" is every cold start, so the fleet was replaying the same DDL hundreds of times a day, each one a round trip to the warehouse and several to Redis.

A shared marker per deployment, set the first time any instance finishes the batch, means each batch now runs about once fleet-wide. The marker fails open: if Redis is unreachable, the instance runs the DDL itself, which is the pre-fix behaviour and is safe.

Silent gaps

Eleven queries had no entry in the table that assigns cache lifetimes, and a missing entry meant a lifetime of zero. Nothing errored. They just always went to the warehouse. A check script now counts the entries and reports the number, so the next query added without one shows up as a number going the wrong way rather than as a slow page six weeks later.

What moved

	August	September, projected
Commands	~600M	~75M
Per tracking poll	~360	~6
Warehouse requests, bad day	911K	~307K

Reads fell by about 91 percent. Writes fell by about 63 percent. That gap is the one loose end, and it is the subject of the next section.

Then the price of a command changed

For all of the above, the Redis store was on a fixed-price tier. Commands were free and unlimited. The thing that tier capped was bandwidth, and we hit 456 of its 500 gigabytes, so every optimisation was aimed at bytes on the wire.

Then the store moved to pay-as-you-go. On that plan bandwidth is nearly free, at three cents a gigabyte after a generous allowance, and commands cost twenty cents per hundred thousand.

Run August's volume through September's prices and it comes to about $1,200 a month in commands alone. The actual September bill for commands is on track for about $150.

The code did not change between those two numbers. The cost model did. A cache that was cheap under one pricing scheme was an order of magnitude too expensive under another, and nothing in the code could have told you which one you were on.

This is the second thing worth keeping:

The wasteful axis of a cache is whichever one your plan does not meter.

On the fixed tier, chattiness was invisible because it was not billed, so it grew until it hit a wall that was billed, bandwidth. On pay-as-you-go, chattiness is the bill, and bandwidth is the thing nobody will notice growing. Whatever the dashboard shows as unlimited is where the next problem is accumulating.

The loose end

Writes now outnumber reads, 28.7 million to 23.6 million in the first three weeks of September. For a cache that is backwards. A cache is supposed to write once and read many times.

The cause is the same tracking poll. When a batch of sixty orders misses, the write path stores an entry for every order in the batch, including a negative entry for orders that had no data, so the next poll does not re-ask. That is the right behaviour. But the batched write is a pipeline, and the provider bills each command inside a pipeline separately. Sixty entries is sixty billable writes, whether they travel in one HTTP request or sixty.

Pipelining bought latency. It did not buy money. That is the same lesson as the MGET one, arriving from the other direction: the thing that saves round trips and the thing that saves commands are not always the same thing, and you need to know which one your plan charges for before you can tell whether you have fixed anything.

That is next.

The rule

A cache that returns the right answer can still be wrong. Count what it costs to be right.

Correctness is table stakes and it is what every test checks. What the tests do not check is how many times the cache had to be asked, how many of those asks were the cache arguing with itself, and which axis of all that the bill is actually attached to this month.