Introduction “Data engineers are focused primarily on building and maintaining data pipelines that transport data through different steps and put it into a usable state … The data engineering process encompasses the overall effort required to create data pipelines that automate the transfer of data from place to place and transform…

Introduction Data engineering is a broad discipline that includes data ingestion, data transformation, and data consumption, along with the accompanying SDLC best practices (i.e. DevOps). Data engineers employ different tools and approaches depending on the phase. For this blog post (and related Quickstart) we will focus on the data transformation phase…

Introduction The recent craze around the data lakehouse seems to me like much ado about nothing. The idea is to have a single platform to support all your data and all of your workloads (Data Warehouse, BI, AI/ML, Streaming). Well, Snowflake has been able to support these scenarios for a long…

Introduction In this blog post I will try to make the case that we should both move beyond the file-based data lake, and stop using the terms “data lake” and “data lakehouse” altogether! Data lakes were created a little over 10 years ago to overcome real limitations in legacy data warehouses…

Introduction In this blog post I will provide step-by-step instructions for how to build a simple CI/CD pipeline for Snowflake with Azure DevOps. My hope is that this will provide you with enough details to get you started on your DevOps journey with Snowflake. DevOps is concerned with automating the development…

Borrowing an expression from Mugatu in Zoolander, “DevOps is so hot right now”! And the database DevOps space in particular is red hot. Like I mentioned in my previous article, A database change management (DCM) tool is critical for managing your Snowflake objects and implementing CI/CD pipelines. But DCM tools…

How do you currently manage your Snowflake objects (tables, views, stored procedures, and so on)? Are you using any tools to help with automation and DevOps? If the answer is no, you’re not alone. Just about every Snowflake customer is asking how to do DevOps with Snowflake. But the term…

(Originally posted on Apr 09, 2020 on Snowflake.com) If you asked almost any current leader in data engineering to draw a “modern” data architecture on a whiteboard (or you searched online for one), you would most certainly get something like the following: But what’s so modern about this systems-based architecture…

(Originally posted on Feb 28, 2019 on Snowflake.com) The topic of DevOps is trending right now and most of my clients are asking about it, especially those who want to build a modern data platform in the cloud. But actually doing DevOps for a data warehouse environment is difficult and…