What the medallion is actually for

Every explanation of medallion architecture is the same diagram: three boxes, left to right, labelled bronze, silver and gold, with an arrow between each. Raw, then cleaned, then business-ready. It is accurate and it is nearly useless, because it describes the shape without saying why the shape helps.

I built one from nothing: an ERP and a Dataverse-backed application unified across finance, inventory, sales, product development and production, ending up at thirteen lakehouses, ten warehouses and twenty-six PySpark notebooks. What the diagram never told me is that the layers are a division of responsibility for being wrong.

Bronze's only job is to be faithful

The temptation in bronze is to fix things on the way in. A date arrives malformed, a column has the wrong type, a code needs trimming, and you are already writing Spark, so why not.

Because the moment bronze transforms, you have lost your only copy of what the source actually said. When a number is disputed three months later, the question is always "what did the system send us", and bronze is the only layer that can answer it. If bronze has opinions, nothing can.

So bronze takes the hit for source weirdness and passes it on. One exception earns its place: values that cannot be represented at all, as opposed to values that are merely wrong. We had dates arriving as 0001-01-01, which land as a day number no downstream reader can parse. Those get clamped to null in the writer, because the alternative is a file nobody can open. That is a rule about physics, not about business meaning.

Silver conforms; gold serves a question

Silver is where the same idea from two systems becomes one idea: types settled, keys conformed, duplicates resolved, one grain per table. It is allowed to be opinionated because bronze is still there to check against.

Gold is where I got the mental model wrong for a while. I treated it as "silver, but nicer", a general-purpose clean warehouse. Gold is better understood as a set of answers to specific questions. Shaped for one consumer, denormalised without apology, and cheap to throw away and rebuild when the question changes. If a gold table is trying to serve everybody, it is a second silver and it will accumulate the compromises of both.

The failure that taught me the most

None of that is what actually bit me. This is.

Bronze-to-silver notebooks ran in parallel, because they are independent and parallel is faster. Silver-to-gold notebooks also ran on their own schedule. Both were correct in isolation. Together they raced: gold began reading a silver table while a bronze-to-silver writer was still part-way through rewriting it, and produced numbers that were not wrong so much as impossible: half of one load and half of another.

It is a horrible class of bug. It does not throw. It does not reproduce, because it depends on which job happened to be mid-write. It just occasionally emits a figure nobody can reconcile.

The lesson is that the arrows in the diagram are not decoration. Medallion ordering is a real constraint, and "these jobs are independent" is only true within a layer. Across layers it is a lie, and the orchestrator has to enforce it.

Two more that cost me a day each

Append-mode mirroring needs a deduplication key. Mirroring a source in append mode is attractive because it is simple and never destroys history. It also happily writes the same row again on every overlapping run. If you append, you must dedupe on read or on merge. Pick one deliberately, because the default is silent duplication that inflates every count downstream.

Incremental means a watermark and a merge on a stable identity. Ours keys on the source system's own row identity with a high-water mark per table. Both halves matter: the watermark decides what to fetch, and the stable key decides what to overwrite versus insert. Get the second one wrong and re-running a load quietly doubles a table rather than being a no-op, and re-runnability is the whole point.

The part I would tell myself at the start

The layers are not a cleanliness gradient, and moving data through three copies is not the benefit. The benefit is that each layer has exactly one job, so when a number is wrong you know which layer to interrogate: bronze if the source lied, silver if we conformed it badly, gold if we asked the wrong question.

That is what makes the extra copies worth their storage. Not tidiness. Attributable error.