Article

Silent schema drift in ETL

Silent schema drift is when a pipeline run succeeds but the contract changed underneath: a rename, nullability flip, or type widen that never fails the job. Fix it with explicit detect, block-on-break, and accept-new-column gates — not hope.

What “silent” actually means

Job success is not contract success. Loaders coerce strings into numbers, drop unknown fields, or write nulls into formerly required keys. The warehouse table still grows. The dashboard that joined on the old key goes sparse. Ops sees green; finance sees a wrong number on Monday.

The three drift classes that matter

Additive: new optional columns. Usually safe if consumers ignore unknowns, but still need an explicit accept so models do not invent meaning. Type change: int to string, timestamp without zone, decimal precision. These pass many loaders and poison aggregates. Breaking drop or rename: the field consumers still read disappears or changes name. Block these by default.

Detection that operators will use

Compare expected schema (last accepted) to observed schema on every run. Surface a visual diff: add / type / drop. Page or ticket on break; never auto-merge a drop into production tables. Keep the last-good schema pinned until a human accepts.

How ETL Pack approaches it

ETL Pack detects add, remove, and type-change; blocks breaking drops; requires explicit accept for new columns; alerts on drift. Details and waitlist: https://91crew.com/etl-pack.html

Related: ETL Pack · Schema evolution · Failure recovery