Open table format debates usually sound like tech theater. Teams compare features on a slide, then discover the hard part is the contract they just signed by choosing a format.

Formats matter, but they also encode assumptions that your platform team must operate every day: metadata behavior, maintenance ownership, evolution constraints, and support boundaries.

Why format choice is an operating model

If the table is a shared platform object, the format determines:

  • who can safely reason about schema and partition changes,
  • how quickly teams can repair corruption,
  • which team owns table health,
  • what cross-engine support actually works in practice,
  • and which incidents count as infrastructure versus data mistakes.

When we say open table format, we are usually choosing a pattern for shared operating behavior before choosing a SQL syntax.

Metadata and control plane: more than a pointer file

Delta Lake stores transactional history in a JSON-based log. The protocol and feature matrix are not just implementation details:

  • protocol versions affect which features are legal in your runtime,
  • table features can silently split compatibility between engines,
  • and upgrade paths depend on which readers your cluster team still has to support.

Delta UniForm is a concrete example. It improves interoperability, but only when your table feature set and runtime versions stay aligned.

Apache Iceberg builds a control structure from snapshots and manifests. Good for visibility, but operationally it adds a set of state objects you must care for with intent.

Apache Hudi uses table services and timelines, so your maintenance model includes explicit lifecycle steps rather than only append/compaction decisions.

Table maintenance is now part of your service model

In all three formats, maintenance is not optional:

  • Delta relies on compacting and vacuuming as part of normal operations.
  • Iceberg requires periodic rewrite/maintenance behavior to prevent metadata and file bloat.
  • Hudi expects compaction, cleaning, and clustering planning to preserve read cost and correctness.

A platform team that treats this as a one-time setup task quickly inherits backlog from every ingestion owner.

Schema, partition, and protocol evolution

Most outages are not from wrong SQL; they are from unmanaged evolution. With shared tables, you need governance around:

  • partitioning policy (including evolution strategy),
  • schema compatibility windows,
  • protocol changes in Delta,
  • and rollback/repair behavior your job scheduler can explain under pressure.

Delta has protocol and features, Iceberg has snapshot and metadata evolution, and Hudi has timeline-driven evolution patterns. Same surface problem, different control knobs.

Interoperability without illusions

No engine supports all corner cases equally. Query engine drift is a real thing: the same table can behave differently across engines because of parser, planning, or feature support variance.

You do not need to avoid open formats for this reason. You need explicit compatibility contracts:

  • what read/write features are required,
  • what fallback options are allowed,
  • what happens when one engine misses a feature set.

Performance failure modes and lock-in beyond Parquet

Everyone expects file-level performance. Real incidents happen in metadata paths:

  • snapshot/log growth causing planning delay,
  • manifest drift causing unnecessary scan overhead,
  • compaction lag creating too many small files,
  • and cleanup debt causing query instability.

These are still lock-in problems beyond simply staying on a format like Parquet. They may not be vendor lock-in; they can be runbook lock-in if you do not define clear ownership and cost model.

Governance, security, and support boundaries

Treat open table formats as product decisions with clear ownership:

  • platform team owns feature/reader policy,
  • domain teams own data quality and evolution requests,
  • platform SRE owns operational cadence and alerting,
  • leaders own the explicit cost of incident ownership.

When everyone agrees on this, table format debates get easier and incidents cheaper.

References

Disclosure

This article was co-written with an AI agent and reviewed by Rujikorn Ngoensaard.