Hi For some use case we are planning to put JSON data as variant column in Satellite without extracting into individual columns. We will then extract the required columns in the Business vault. Is the Is this approach acceptable in the data vault pattern. We are using Snowflake database

Of course there are dozen of tips available for Snowflake, but let me share the ones which came into my mind very quickly: 1) Understand how Snowflake stores the data! They are using micro-partitions, organized in a columnar way. Micro Partitions store statistics like distinct values and value-ranges for each column. Your goal should always be to prune as much as possible from both when querying data. For example: Only select columns you really need, and apply filters on columns where the values are mostly not overlapping multiple Micro Partitions. Also think on re-clustering your data if necessary, or creating your own values with a pattern to cluster your data on (usually only necessary for huge amounts of data in one table). 2) When data is spilled to local storage while querying, is a good indicator that a bigger warehouse makes sense. I assume here that the query itself is already optimized and we are just dealing with a lot of data and maybe complex logics. But keep in mind: Increasing the size of the Snowflake Virtual Warehouse by 1 step (i.e. M -> L), doubles the costs or the same runtime! (calculated per cluster). So, when the query time is less than 50%, we achieved a win-win: faster & cheaper result! If the runtime could not be reduced by 50% or more, then you have to decide whether the quicker response is worth the money you now spend. 3) Snowflakes no-copy clones allow you to test features and fixes against your production in a very easy and fast way. It should be part of your deployment pipelines. 4) Insert-only reduces the number of versions Snowflake has to create for the Micro Partitions. Updates and Deletes cause this versioning of already existing Micro Partitions what costs time and additional storage. That also means that Data Vault with its Insert-Only approach meets the scalability factors of Snowflake! 5) The QUALIFY statement improved the code writing a lot. It is using the result of a window-function as filter, means, you don't have to write nested sub-queries with and self-joins.

Hey community, I am wondering what your best practices are for requirement gathering from the business to enable data engineers to work? Imagine you build a data platform for several purposes: reporting, dashboarding, cleansed data for AI input, etc. The general output are Information Marts (however they are modeled). And we want to build it the agile way. How much and what kind of information would be necessary from the business? How much in-person communication you would expect during development? A User Story itself is always nice to understand the intention, but is way too less as details are missing (in my opinion). Looking forward to your opinions:)

Who we can welcome to the Data Vault Friday tomorrow 11am. CEST? :) Ask your quesion upfront and get an answer from Michael Olschimke: https://scalefr.ee/DataVaultFriday Or just attend and learn: https://us02web.zoom.us/webinar/register/WN_TsjULDy7Tuiwp0clye0BSQ#/registration Duration: Usually 10-20 minutes.

Currently tweaking datavault4dbt to work with Oracle. How come Oracle users are less interested in Data Vault than snowflake users?