Data Pipeline Checkpointing and Failure Recovery: A Practical Guide
Checkpointing looks like a performance feature: a way to avoid reprocessing records you've already handled. It's actually a correctness contract between two separate writes, and the order you perform them in decides whether a restart costs you a few duplicate rows or a silent, undetected gap in your data.
TL;DR
- A pipeline's progress has to live in durable storage, not in the memory of the process running it.
- The checkpoint advances only after the corresponding output is confirmed durable, never before.
- Pipeline output is written idempotently, so reprocessing after a restart never creates duplicate rows.
- A stage writes to a staging location and promotes atomically; nothing downstream ever sees a stage half-finished.
- Every stage resumes from its own checkpoint and reads a bounded range, not the whole pipeline from scratch.
- A single malformed record goes to quarantine with a reason attached; it never takes down the run.
- Pipelines can each get checkpointing right in isolation and the fleet still drifts, because schema, partitioning, and lineage conventions are decided pipeline by pipeline.
What Data Pipeline Checkpointing Actually Protects
Ask an engineer why a pipeline lost or duplicated data and most will describe a rare edge case - a spot instance reclaimed at the wrong moment, a network blip during a write. That's usually not wrong, but it's incomplete. The failure isn't that the crash happened; crashes are guaranteed to happen eventually. The failure is that the pipeline's checkpoint and its actual output can disagree about what got written, and nothing in the code prevents that disagreement from becoming permanent.
None of the seven principles below are about handling failure better. They're about making sure a pipeline's recorded position and its actual output can never drift apart, so a restart is a boring, predictable event instead of the moment a real incident begins.
1. Checkpoints Outlive the Process That Wrote Them
A pipeline restarts - a deploy, an out-of-memory kill, a reclaimed spot instance - and either reprocesses everything from the start because there was no durable record of progress, or resumes from a stale position that lived in a local variable and evaporated with the process. The rule: the last successfully processed offset or watermark gets written to a durable store the moment it's confirmed processed, never held only in memory.
last_offset = 0
for batch in kafka_consumer.poll_batches():
process(batch)
last_offset = batch.max_offsetlast_offset = checkpoint_store.get("orders-topic-partition-3")
for batch in kafka_consumer.poll_batches(start=last_offset):
process(batch)
checkpoint_store.put("orders-topic-partition-3", batch.max_offset)The honest tradeoff: durable checkpoint writes add I/O on every batch, and checkpointing too aggressively throttles throughput. Flink's checkpointing model batches on a configurable interval instead of every record — a wider replay window traded for lower overhead.
2. Checkpoints Advance Only After the Write Lands
A pipeline that advances its checkpoint as soon as it believes a batch is processed, rather than after confirming the write landed, creates a specific failure: a crash between "processed" and "written" leaves the checkpoint saying the work is done for records that were never written anywhere. The order is fixed: write the output, confirm it's durable, then advance the checkpoint - never the reverse.
await checkpointStore.set(partitionKey, batch.maxOffset);
await s3.putObject({ Bucket: "orders-curated", Key: outputKey, Body: rows }).promise();await s3.putObject({ Bucket: "orders-curated", Key: outputKey, Body: rows }).promise();
await checkpointStore.set(partitionKey, batch.maxOffset);This is the mechanism Apache Flink's two-phase commit approach is built around: don't finalize a checkpoint until the sink confirms the write is committed. Reversing the order doesn't just risk a bug — it silently downgrades exactly-once delivery to something weaker, without telling anyone.
3. Replayed Records Never Become Second Rows
Principle two guarantees that after a restart, a pipeline reprocesses a small tail of records it may have already written once. If the output write isn't idempotent, that reprocessing duplicates data on every restart. Output writes need a deterministic key - the source partition, the record's natural key, the processing window - written as an upsert, a partition overwrite, or a deterministic file path.
INSERT INTO daily_revenue (date, amount) VALUES ($1, $2) ON CONFLICT (date) DO UPDATE SET amount = EXCLUDED.amount;
Stripe's approach to idempotent requests and Amazon's writeup on making retries safe with idempotent APIs both converge on the same design: a caller-supplied or data-derived key that lets the same operation run twice with one effect.
Tradeoff worth naming: upserts cost more than blind appends, since they require a lookup or conflict check on every write. For very high-throughput streams, a partition-level overwrite is often cheaper than a row-level upsert and delivers the same guarantee.
4. Half-Written Stages Never Reach a Consumer
A downstream dashboard flickers between correct and wrong numbers throughout a pipeline run, then settles once the run finishes - usually because a stage writes records directly into its destination as it produces them. The fix is a staging location nothing downstream reads from, with a promotion step that happens only once the stage is fully finished.
var staging = fileSystemClient.GetDirectoryClient($"_staging/{runId}");
await staging.UploadAsync(rows);
await staging.RenameAsync($"curated/{partitionDate}");This is the write-audit-publish pattern, popularized by Netflix's data engineering team for exactly this reason: check the data after it's produced but before anyone else can see it.
The tradeoff is latency to visibility - data isn't queryable until the swap happens, a deliberate trade of freshness for correctness, and the right one for almost anything a business decision gets made on top of.
5. Every Stage Resumes Itself, Not the Pipeline
A five-minute network blip or one flaky worker forces a full multi-hour pipeline re-run from the beginning, because only the top-level orchestrator tracks position. Every stage should begin by reading its own persisted checkpoint and starting from there, and read a bounded range on every run, so resuming and running normally are the same code path.
List<Row> rows = sourceTable.readAll(); writeTransformed(rows);
Instant watermark = checkpointStore.get("backfill_stage");
List<Row> rows = sourceTable.readRange(watermark, watermark.plus(Duration.ofHours(1)));
writeTransformed(rows);
checkpointStore.put("backfill_stage", watermark.plus(Duration.ofHours(1)));Diagnostic worth running: kill a stage mid-run and restart it. If the time to catch up is proportional to how long it was down, not to the size of the whole pipeline, the stage is resumable. If not, it isn't.
6. One Bad Record Costs You One Record
A single malformed payload throws an unhandled exception and takes the entire batch down with it, or gets caught by a broad try/except and disappears with no trace it ever arrived. A record that fails parsing or validation goes to a dedicated quarantine destination with the failure reason attached, and the run continues past it.
Diagnostic worth running: send one deliberately malformed record through the pipeline. If the run halts, or the record vanishes with no trace, the pipeline doesn't have a quarantine path yet.
AWS's dead-letter queue pattern is the message-queue version of the same idea. The same discipline applies to external calls inside a transform: bound the retries with backoff, and route anything that exhausts its retries to the same quarantine destination rather than stalling every record behind it.
7. Correct Pipelines Still Drift Apart at Scale
None of the six principles above are hard to get right inside a single pipeline. The problem is that "right" isn't the same decision twice across an organization - one team partitions by event time, the next by processing time; one team writes lineage columns into every output row, another never does, and an incident review that should take ten minutes takes three days.
Rule of thumb: if tracing a bad record back to the run that produced it requires asking three different teams, the pipeline's lineage metadata isn't the problem - the fleet's conventions are.
Every one of those choices looked locally reasonable to the team that made it. This is the point where individual engineering discipline stops being sufficient and an enforcement layer becomes the only thing that scales.
How Pandorian Enforces This Guide
Every principle above survives a design review. Few survive six months of independent team decisions, because the drift isn't visible from inside any one pull request - it's visible only in the aggregate. Pandorian enforces this guide across three surfaces:
Caught at PR level
Every pull request is checked before merge: checkpoint ordering, upserts instead of appends, lineage columns on new schemas. Violations surface with the line, the rule, and a fix.
Full scans
An on-demand scan across the whole codebase establishes the day-one baseline: which pipelines still append instead of upsert, which still abort the run on a single bad record, which have never written a run identifier into their output.
Scheduled scans
The same full scan, run on a cadence, catches drift introduced outside any pull request - a shared library change that silently strips lineage columns from a dozen pipelines' output, or a new pipeline added by a different team that never went through review against these standards.
A Taste of the Guideline Package
Four of the checks that ship in this package - each runs at the PR level, on full or scheduled scans, or several at once.
FAQ
What is checkpointing in a data pipeline?
+Checkpointing is the practice of durably recording how far a pipeline has progressed, so that a failure resumes from the last known-good position instead of restarting from the beginning or, worse, from an unknown position. A checkpoint stores the offset or watermark plus whatever state the processing depends on, committed atomically with the output it corresponds to.
The word doing the work is atomically. A checkpoint written separately from the output it describes will eventually record progress for data that was never written, or fail to record data that was.
What is idempotency, and why does a data pipeline need it?
+An idempotent operation produces the same result whether it runs once or many times. Pipelines need it because recovery always replays: a job that failed after writing half its output will, on restart, reprocess records it already handled. Without idempotency, that replay doubles rows, double-counts metrics, or fires the same downstream event twice.
In practice it means writing with a deterministic key (upsert rather than insert), making side effects deduplicable by an event ID, and treating "this record was already processed" as a normal outcome rather than an error.
What is the difference between at-least-once, at-most-once and exactly-once delivery?
+At-least-once guarantees no record is lost but allows duplicates, at-most-once guarantees no duplicates but allows loss, and exactly-once guarantees each record is reflected in the output once. Most systems give you at-least-once by default, because acknowledging after processing is the only ordering that avoids loss.
Exactly-once is achievable at the level of end state, not at the level of network delivery. It is built by combining at-least-once transport with idempotent writes and transactional checkpointing, which is why the practical goal is effectively-once processing rather than exactly-once delivery.
Is it ever fine to just re-run the whole pipeline instead of checkpointing?
+Yes, and for a meaningful class of jobs it is the better engineering decision. A full recompute is simpler, has fewer failure modes, and is trivially correct if the job is deterministic, the input is immutable, the output is a full overwrite, and the runtime fits comfortably inside the recovery window.
It stops being fine at a predictable point: when the runtime grows past the time available to recover, when the source is a stream with retention shorter than the rerun, when the job has external side effects that cannot be replayed, or when reprocessing cost becomes a budget conversation. Teams usually cross that line without noticing, because nothing breaks on the day it happens.
How do I tell if our pipeline can actually recover from a mid-run failure?
+Kill it and find out. Terminate a job at roughly the halfway mark in a staging environment with production-shaped volume, restart it, then compare the output against a clean full run. Row counts, aggregate sums and a sample of records should match exactly.
Two things this surfaces that a design review will not: whether the checkpoint and the output are actually committed together, and whether any side effect (an email, a webhook, a downstream event) fired twice. Add a second test where the failure lands immediately after a write but before the checkpoint, because that is the window where correctness is genuinely at risk.
Can't the orchestrator handle recovery?
+An orchestrator like Airflow, Dagster or Step Functions handles retries, scheduling and task-level state. It knows a task failed and can run it again. What it cannot know is whether running it again is safe, because that depends on whether your task's writes are idempotent and whether it recorded its own progress. Retrying a non-idempotent task is how a duplicate-rows incident starts.
The properties that make retry safe live in the pipeline code itself, which is where they have to be checked: on every pull request that adds or modifies a job, and via scans that inventory the existing jobs writing without a deterministic key.
Get the Guideline Package
You'll get this as a guideline MD file - hand it straight to your own coding agent or CI setup. Or skip the DIY step: give these same guidelines real enforcement superpowers, with PR checks and scans live on your repos, through Pandorian.
From Principles to Enforcement
Every rule in this guide is easy to agree with in a design review and easy to violate six sprints later, one locally-correct decision at a time. Pandorian turns boundary discipline like this into CI guardrails - scoped to the repos, teams or services you choose.
