21d • In the News
The multimodal lakehouse is real. The diagram everyone is drawing is not. 🎞️
Every RAG project starts the same way: a few hundred PDFs, a chunking strategy copied from a blog post, an off-the-shelf vector store. The demo genuinely works and the budget clears. The trouble starts a quarter later, when the business stops asking about documents and wants the model to review the security footage.
In this edition of datapro.news, we cover the rewrite nobody scoped. Unwinding a text-only stack is a real architectural project with real money behind it — but the reference diagram circulating to describe it (lakehouse, Kafka, Flink, Materialize, done) does not survive contact with the documentation of its own components. Three of the four boxes are doing jobs they do not do.
Here are the 3 components the design actually turns on:
* 📼 Apache Kafka: Learn why "Kafka ingests the video stream" tells a design review you have not sized a broker, and the pattern that already has a name you should be using instead.
* ⚡ Apache Flink: Discover why the part of the pitch that sounds most like vapour became true most recently — plus the two footnotes the marketing omits, one of which means you will pay for some tokens twice.
* 🧮 Materialize: See why listing it as a way to compute embeddings is the tell that a diagram was assembled from vendor landing pages, and where it genuinely belongs instead.
We also get specific about the distinction vendors blur between Lance and LanceDB, why "batch is deprecated" is contradicted by the flagship deployment of this very architecture, and the compliance citation that will get your design second-guessed by legal. The issue closes with a scorecard: what each component does, and what to check before you commit.
Parsing a PDF is a commodity, and it was one before this wave. The scarce thing is a system that embeds a live stream without a bespoke microservice holding it together and joins the result against governed relational data. That is buildable in 2026 — just not from the diagram everyone is drawing. Get the boxes right first.
Check out the video edition below 👇
0
0 comments
The multimodal lakehouse is real. The diagram everyone is drawing is not. 🎞️
skool.com/data-innovators-exchange
Your source for Data Management Professionals in the age of AI and Big Data. Comprehensive Data Engineering reviews, resources, frameworks & news.
Leaderboard (30-day)
1
+5
2
+1
3
+1
Powered by