CDC is a production contract, not an ingestion shortcut

A nightly extract feels safe until the first delete disappears.

The source application removes a customer address. The warehouse still has yesterday's row. A dashboard joins to a dimension that should no longer be valid. A compliance report asks why the record was still visible after the operational system changed. The pipeline did not fail. It just never had a reliable contract for change.

That is the real reason Change Data Capture matters in data engineering.

CDC is not only a faster way to move rows. It is a design pattern for preserving the sequence, meaning, and recoverability of change from one system into another. Done well, it lets a data platform keep tables, marts, search indexes, semantic models, and operational replicas close to the source without repeatedly copying the whole database. Done poorly, it delivers wrong data with lower latency.

Decision rule

Treat CDC as a production contract. The contract must define initial state, ordering, retention, schema evolution, delete handling, replay, and ownership. Without that contract, CDC is just a faster path to inconsistent data.

Evidence note: vendor-specific behavior in this article comes from current official documentation and public research linked in the references. The operating-model guidance is engineering judgment from running analytical and lakehouse platforms where downstream correctness matters as much as ingestion speed.

What CDC changes

Traditional batch ingestion asks, "What does the source look like now?"

CDC asks a harder question: "What changed, in what order, and how should the target apply it?"

That difference changes the architecture.

A full extract can rebuild a table from the latest source state. It is easy to reason about when the source is small, the freshness requirement is loose, and deletes are handled explicitly. But it becomes expensive and fragile when tables grow, source systems cannot tolerate large repeated scans, or consumers need to understand the difference between a late update and a new event.

A CDC pipeline reads changes from a source-system mechanism such as a transaction log, write-ahead log, binary log, change table, stream, or table change feed. It then carries enough metadata downstream for consumers to apply those changes safely: keys, operation type, sequence position, commit time, before or after values, and sometimes schema-change information.

The value is not just lower latency. The value is that the platform can process smaller units of work while preserving the source's mutation history.

That is why CDC sits close to many serious data engineering problems:

• incremental lakehouse ingestion,
• SCD Type 1 and Type 2 dimensions,
• operational replicas,
• search-index synchronization,
• data product freshness contracts,
• audit and recovery workflows,
• migration from one platform to another,
• near-real-time BI surfaces.

The same mechanism that feeds a lakehouse table can also expose whether the downstream system is lagging, whether a replay is safe, and whether a deleted record really disappeared from the target.

The first trap is the initial state

The hardest part of CDC is often not the stream. It is the first copy.

A target table needs a complete starting point before it can apply future changes. If the platform starts consuming log events at noon but the target has no correct state as of noon, the stream alone is not enough. Transaction logs usually do not retain the full history of the database forever. A pipeline needs both a baseline snapshot and a change stream boundary.

That boundary is where many CDC designs fail.

If the initial snapshot takes hours, the source continues accepting writes during the scan. A row read early in the snapshot may be updated before the snapshot finishes. Another row may be deleted after the scan passes its key range. If the pipeline does not interleave the snapshot with log positions correctly, the target can miss changes or apply them twice.

This is why modern CDC systems care about watermarks, offsets, log sequence numbers, replication slots, chunked snapshots, and resumable backfills. These are not implementation details. They define whether a downstream table is equivalent to a real source state.

The practical design rule is simple: never say "CDC is enabled" until you can explain how initial state and ongoing changes meet.

Ordering is part of the data

Change events are not independent records. Their order matters.

A customer row can be inserted, updated, deleted, and reinserted. A product can change category before a transaction arrives downstream. A source transaction can update several tables together. If the target applies those events out of order, the final table may look valid while carrying the wrong meaning.

CDC pipelines need a sequencing contract. In PostgreSQL that might involve logical replication and write-ahead log positions. In SQL Server CDC, change rows include commit LSN and sequence metadata. In Delta and Databricks patterns, consumers may use table versions, commit timestamps, or explicit sequence columns. In warehouse stream patterns, the stream's offset and retention behavior become part of the contract.

A useful CDC target design asks these questions before implementation:

If these answers are not explicit, the pipeline might still run. It just cannot be trusted during incidents.

CDC is not a magic real-time SLA

CDC is often sold internally as "real time." That phrase is too loose for production.

A source system must expose changes. A capture process must read them. A broker or pipeline must transport them. A target must apply them. Each step has throughput limits, retries, credentials, retention, schema rules, and operational failure modes.

AWS DMS documentation is explicit that CDC does not provide a real-time replication guarantee; latency depends on source workload, network, replication resources, target capacity, and data shape. SQL Server CDC has capture and cleanup jobs, and retention determines how far back consumers can query change tables. PostgreSQL logical replication uses replication slots, and retained WAL can become an operational concern when consumers lag. These details are the difference between a platform and a demo.

A better freshness contract says something like this:

• Source changes are normally visible in the bronze table within five minutes.
• Alerts fire when capture lag exceeds fifteen minutes.
• The source log retention window supports at least twenty-four hours of replay.
• Consumers can replay from the last committed checkpoint without duplicate side effects.
• If the retention window is breached, recovery requires a controlled resnapshot.

That wording is less exciting than "real time." It is also something an operations team can support.

Lakehouse CDC is a second contract

Database CDC and lakehouse CDC are related, but they are not the same thing.

A database log tells you what changed in the source. A lakehouse change feed tells you what changed in a table after data has already entered the analytical platform. Both are useful. They operate at different boundaries.

Delta Lake Change Data Feed records row-level changes for Delta tables after CDF is enabled. Databricks documents Change Data Feed for querying table changes and AUTO CDC APIs for applying CDC feeds or ordered snapshots into SCD Type 1 and Type 2 targets. As of the current Databricks documentation, AUTO CDC replaces the older APPLY CHANGES API recommendation while keeping compatible syntax. Snowflake streams expose table changes that can be consumed by downstream tasks, with stream retention and consumption behavior becoming part of the design.

The engineering point is this: CDC can exist at multiple layers.

• Source CDC captures application database changes.
• Bronze ingestion preserves raw change events.
• Silver tables apply merge rules, deduplication, and delete semantics.
• Gold tables expose current-state, history, or semantic-ready views.
• Downstream table feeds expose changes from curated data products.

When teams blur these layers, they lose observability. A data product consumer should know whether they are reading raw source mutations, cleaned current state, or governed business history.

The target model decides the meaning

The same CDC event can create different analytical truth depending on the target model.

For a current-state customer table, an update overwrites the previous values. That is SCD Type 1 behavior. For a customer history table, the update closes one version and opens another. That is SCD Type 2 behavior. For audit-heavy systems, business time and system time may both matter: when the event happened versus when the platform learned about it.

This is why CDC should not stop at ingestion. The downstream model needs its own contract:

• Type 1 for current operational state.
• Type 2 for dimensional history.
• Event table for immutable mutation history.
• Bitemporal model when business effective time and system ingestion time both matter.
• Snapshot table when only periodic reporting state is required.

CDC gives you the raw material. It does not decide the model for you.

A common mistake is to land CDC events and call the job done. The business does not consume "operation equals update." It consumes customer status, product availability, account balance, policy entitlement, inventory position, or order lifecycle. The platform still has to translate source mutations into durable analytical semantics.

What a production CDC design needs

A useful CDC architecture has a few non-negotiable parts.

First, it needs a capture boundary. Which tables are captured, which columns are included, how are schema changes handled, and what permissions does the capture process require? Source owners should understand that CDC is not free. Logs, slots, capture jobs, and supplemental logging can affect source operations when neglected.

Second, it needs a checkpoint boundary. The pipeline must know the last source position it has safely applied. This position should be durable, observable, and recoverable. A restart should not depend on someone remembering which file or timestamp looked correct.

Third, it needs an apply boundary. The target should define keys, sequence rules, deduplication, delete behavior, and late-event handling. A merge statement without these rules is only half a design.

Fourth, it needs a retention boundary. Every CDC mechanism has a recovery window. If the source log, change table, table stream, or change feed expires before the consumer catches up, the system cannot pretend replay is still safe.

Fifth, it needs an ownership boundary. Someone owns capture health. Someone owns target correctness. Someone owns schema-change coordination. Someone owns replay and resnapshot procedures. CDC fails slowly when ownership is vague.

Common CDC failure modes

CDC failures rarely look like a red pipeline on day one. They often look like quiet drift.

A consumer falls behind but no one notices until the source log retention window is gone. A schema change adds a column, but the target ignores it for two weeks. A delete event arrives, but the merge only handles inserts and updates. A primary key changes in the source, and downstream logic treats it as two entities. A replay duplicates records because the merge key is not stable. An initial snapshot races with live writes and creates an impossible target state.

These are engineering failures, not CDC feature failures.

Watch for these warning signs:

• The team cannot name the current source offset or table version.
• Retention is shorter than the expected incident response time.
• Delete and truncate behavior are not tested.
• Source DDL changes are discovered from broken jobs instead of planned contracts.
• Backfills use a different code path from daily ingestion.
• Current-state and history tables are built from the same stream without separate acceptance tests.
• Monitoring tracks job success but not lag, skipped events, duplicate keys, or freshness.

If a CDC pipeline only proves that data moved, it has not proven that change was applied correctly.

When snapshots are enough

CDC is not always the right answer.

Snapshots are often enough when the source is small, the business accepts daily freshness, deletes are rare or explicitly represented, and the target does not need mutation history. A product reference table with a few thousand rows may not need CDC. A monthly planning extract may not need CDC. A dataset rebuilt from an authoritative API every night may be simpler and safer as a snapshot.

The decision is not "CDC versus batch." The decision is which contract the data product needs.

Use CDC when:

• the table is too large for repeated full extracts,
• source load from snapshots is unacceptable,
• deletes and updates must be captured quickly,
• consumers need incremental freshness,
• replay from a known point is important,
• SCD history depends on row-level changes,
• migration or dual-running requires close synchronization.

Prefer snapshots when:

• the table is small,
• freshness is loose,
• full rebuilds are cheap and reliable,
• source APIs do not expose stable change positions,
• the target only needs periodic state,
• the team cannot yet operate CDC safely.

Simplicity is a valid architecture choice. But it should be chosen deliberately, not because the team avoided defining change.

Decision checklist

Before approving CDC for a production pipeline, I want clear answers to these questions:

1. What source mechanism exposes changes?
2. How is the initial snapshot aligned with the change stream?
3. What position, version, LSN, timestamp, or sequence column defines order?
4. What is the maximum supported consumer outage before data loss risk?
5. How are inserts, updates, deletes, truncates, and schema changes represented?
6. What target model is being built: current state, history, event log, or bitemporal table?
7. How does replay avoid duplicate side effects?
8. What metrics prove the stream is healthy?
9. Who owns source configuration, downstream apply logic, and incident recovery?
10. What is the resnapshot plan when the contract is breached?

If the team cannot answer those questions, implementing CDC may still be technically possible. It is not yet production-ready.

Why CDC matters now

Data platforms are being asked to serve operational analytics, AI retrieval, governed data products, near-real-time BI, and cross-platform migrations at the same time. Full-table movement is too blunt for that world.

CDC gives data engineering a sharper tool. It lets teams move what changed instead of everything that exists. It gives downstream systems a chance to preserve order, handle deletes, recover from checkpoints, and build history from source mutation. It also forces teams to face the contracts that batch pipelines often hide.

That is why CDC matters for our data engineering world: not because every dataset needs streaming, but because modern platforms need honest change semantics.

The best CDC pipelines are not the flashiest. They are boring in production. They know where they started. They know what they have applied. They know how far behind they are. They know when replay is safe. They know who owns the answer when the source changes shape.

CDC is not the goal. Trustworthy change is the goal.

Related tools

Use the Pipeline Recovery Planner to pressure-test replay, checkpoint, and consumer-impact decisions before a CDC incident turns into a manual reconstruction exercise.

Use the Data Product Contract Builder to define freshness, grain, ownership, access, lifecycle, and consumer expectations around a CDC-backed table.

Use the Lakehouse Table Layout Advisor when CDC-driven merges start creating file pressure, clustering drift, or inefficient update patterns in lakehouse tables.

Key takeaways

• CDC matters because it preserves the meaning and order of change, not only because it reduces batch load.
• Initial snapshot alignment is part of the CDC contract.
• Retention windows define how long recovery remains safe.
• Database CDC, table change feeds, and warehouse streams operate at different architectural layers.
• The target model decides whether a change overwrites state, preserves history, or supports audit timelines.
• A CDC pipeline without replay, monitoring, delete handling, and ownership is not production-ready.

References

Official documentation

• Debezium Tutorial - Debezium. URL. Why it matters: describes Debezium as a platform that converts database changes into event streams and shows create, update, delete, and restart behavior.
• Debezium connector for PostgreSQL - Debezium. URL. Why it matters: documents PostgreSQL CDC capture behavior, snapshots, logical decoding, replication slots, and schema-change handling.
• Logical Decoding - PostgreSQL. URL. Why it matters: explains how PostgreSQL exposes changes from tables into a consumable stream.
• Logical Replication - PostgreSQL. URL. Why it matters: documents initial synchronization, ordered application, and common logical replication use cases.
• What is change data capture (CDC)? - Microsoft SQL Server. URL. Why it matters: explains SQL Server CDC change tables, LSN ordering, retention intervals, schema-change behavior, and capture/cleanup jobs.
• Creating tasks for ongoing replication using AWS DMS - AWS Database Migration Service. URL. Why it matters: documents ongoing replication from source logs and explicitly cautions that CDC latency is not a real-time SLA.
• Change data feed - Delta Lake. URL. Why it matters: explains table-level row change feeds for Delta tables after change data feed is enabled.
• Use change data feed on Databricks - Databricks. URL. Why it matters: documents querying row-level change information from Delta tables on Databricks.
• The AUTO CDC APIs: Simplify change data capture with pipelines - Databricks. URL. Why it matters: current June 2026 Databricks documentation for applying CDC and ordered snapshots into SCD Type 1, Type 2, and bitemporal targets.
• Introduction to streams - Snowflake. URL. Why it matters: documents Snowflake streams as a table-change tracking construct for downstream consumption.

Research

• DBLog: A Watermark Based Change-Data-Capture Framework - Andreas Andreakis and Ioannis Papapanagiotou. URL. Why it matters: explains a watermark-based CDC approach for combining initial backfills with ongoing transaction-log events.
• A Theoretical Study of DBLog: Certified Virtual Cuts for a Snapshot-Equivalent Replay of Live Databases - Andreas Andreakis. URL. Why it matters: provides fresh 2026 formal context for snapshot-equivalent replay while source databases continue accepting writes.

Disclosure

This article was co-written with an AI agent and reviewed by Rujikorn Ngoensaard.