The format war is over. Your next lock-in moved upstairs. 🧊

Every Iceberg pitch lands the same way: your data, your object storage, an open format, queryable from anywhere. The demo genuinely works. The trouble starts about two weeks into production, when someone asks who is allowed to see column seven — and the answer turns out to live somewhere that isn't open at all.

In this edition of datapro.news, we cover the fight nobody announced. A directory of Parquet files in a bucket is inert; it is storage, not a database. Something has to tell an engine which metadata pointer is the current valid state of a table, and whether the querying principal may read it. That something is the catalog. By commoditising the file format, the industry did not eliminate the vendor lock-in it spent a decade complaining about — it relocated it, from the storage layer where it was visible and much-discussed to the governance layer where it is neither. Your Parquet files stay portable. The 5,000 grants, masking rules and policy tags you authored do not.

Here are the 3 catalogs the decision actually comes down to:

🧭 Apache Polaris: Learn why graduating to Apache Top-Level Project in March 2026 is worth more to you than any feature comparison, and why the neutrality you get with it is also an on-call rota — self-hosting means owning HA for the one service every pipeline and dashboard in the company stops without.
🌿 Project Nessie: Discover the most elegant idea in the field — Git branching, tagging and merging across your entire lakehouse state — plus the due-diligence detail that should stop you adopting it as a destination, because its own sponsor has stated it will fold those capabilities into Polaris and retire the project.
🔗 Unity Catalog: See why the criticism you have probably repeated in an architecture review is now out of date, and where the real asymmetry sits instead: outbound it is a well-behaved Iceberg REST server, inbound a reluctant client, and that is a design choice rather than a missing feature.

We also get specific about what the REST spec standardises (more than sceptics allow, and far less than portability requires), and why the two-catalog hedge every nervous architect proposes is a split-brain waiting for a Tuesday — the kind that goes wrong silently, because nothing throws an error. The issue closes with a scorecard: what each layer does, and what to check before you commit.

Storage is free. Execution is commoditising. The catalog is the last place in the stack where a vendor can still charge rent, which is precisely why the marketing there is loudest and the specifications quietest. An open format proves your files are portable. Only a governance plane you can actually move proves your platform is. Those are not the same claim, and the difference shows up three years later, in the size of the rewrite.

Check out the video edition below 👇

7:15

1 comment

The format war is over. Your next lock-in moved upstairs. 🧊

Data Innovators Exchange

skool.com/data-innovators-exchange

Your source for Data Management Professionals in the age of AI and Big Data. Comprehensive Data Engineering reviews, resources, frameworks & news.

Leaderboard (30-day)