Open table formats remove one class of storage cost argument, but they do not remove operational cost. They relocate it into metadata health, file layout, maintenance jobs, and ownership.

That is a good move when teams intentionally budget for it. It is expensive when teams assume the format choice itself is the full cost model.

The new recurring cost buckets

Small file handling

Small file pressure is the most common tax. Ingesting too fast or too fragmented creates thousands of tiny objects. Without compaction or clustering discipline, query performance degrades and your job runtimes become noisy and expensive.

Snapshot, log, and manifest growth

Every table format tracks history differently:

  • Delta keeps transaction logs and checkpoints.
  • Iceberg has snapshots and manifests.
  • Hudi tracks timeline actions.

Growth in these state layers is expected. Unmanaged growth is not.

Retention, vacuum, expire, and orphan cleanup

You need to define retention windows and enforce them.

  • Delta: VACUUM for stale file cleanup, with retention policy aligned to replay requirements.
  • Iceberg: snapshot expiration and orphan file removal, plus periodic manifest rewrites for consistent planning.
  • Hudi: compaction, cleaning, and clustering as part of the table service budget.

Query engine drift and rewrite overhead

Multiple engines on the same table means your optimization settings are never purely a storage problem. You must monitor:

  • read amplification,
  • metadata cache misses,
  • manifest/log bloat,
  • and write/read latency shifts after schema or partition changes.

When one engine writes to one table and another reads, the team should assume ongoing compatibility work, not one-time setup.

What to monitor

Track at least:

  • file sizes and compaction age,
  • stale snapshots/log checkpoints,
  • small file ratio,
  • orphan and undeleted file growth,
  • queue lag in maintenance jobs,
  • and ownership response times for table incidents.

Who owns table health

If no one owns maintenance, no one owns cost. If no one owns cost, nobody owns incidents.

Create a small matrix per domain:

  • domain owner: schema and business contract,
  • platform owner: maintenance and feature policy,
  • incident owner: on-call escalation.

Small checklist

  1. Set table-level compaction and retention policy before onboarding the first producer.
  2. Decide retention and snapshot/orphan cleanup jobs by default, not per team.
  3. Validate read/write feature compatibility across engines before production launch.
  4. Add maintenance debt metrics to every platform health dashboard.
  5. Assign a clear owner when metadata or maintenance deviates from expected behavior.

References

Disclosure

This article was co-written with an AI agent and reviewed by Rujikorn Ngoensaard.