Article

Backfill without breaking incremental

A safe backfill reloads a bounded date or ID range with the same upsert/merge key as incremental sync, isolated from the live watermark so catch-up jobs do not race the clock.

The failure mode

Someone runs a full reload while hourly incremental keeps writing. You get duplicates, overwritten watermarks, or a table that looks complete until the next day-partition gap shows up.

Rules that hold

Bound the window. Use the same primary key merge as incremental. Do not advance the live watermark from a historical job. Prefer a dedicated backfill run type in the UI so operators do not improvise SQL.

Idempotency first

If merge is correct, re-running the same window is cheap insurance. If append-only is all you have, quarantine the window and rebuild the mart — do not pretend append is merge.

Pack direction

ETL Pack treats backfill as a first-class action: date/ID range, reload table, initial historical load. https://91crew.com/etl-pack.html

Related: ETL Pack · Schema evolution · Failure recovery