snowchange, a Database Change Management Tool
How do you currently manage your Snowflake objects (tables, views, stored procedures, and so on)? Are you using any tools to help with automation and DevOps? If the answer is no, you’re not alone. Just about every Snowflake customer is asking how to do DevOps with Snowflake. But the term DevOps (and related terms such as CI/CD, agile, and release automation) can refer to a number of things depending on the context. When it comes to Snowflake, what most people are asking about is database change management (DCM) or database DevOps.
I previously published a blog post titled Embracing Agile Software Delivery and DevOps with Snowflake, which discusses the benefits of and challenges with database DevOps. In that blog post, I mentioned a few approaches for managing your Snowflake objects, but I didn’t provide any specific tool recommendations. Although there are a few different DCM tools out there that work with Snowflake (Flyway and Sqitch, in particular), I’m a big fan of snowchange.
Below, I introduce snowchange, but feel free to skip directly to the GitHub repository for snowchange.
Note: snowchange is not an official Snowflake product and is licensed under the Apache license.
Introducing snowchange
snowchange is a lightweight Python-based tool to manage all your Snowflake objects. It follows an imperative-style approach to database change management (DCM) and was inspired by the Flyway database migration tool. When snowchange is combined with a version control system and a CI/CD tool, database changes can be approved and deployed through a pipeline using modern software delivery practices. As such, snowchange plays a critical role in enabling database (or data) DevOps.
DCM tools (also known as database migration, schema change management, or schema migration tools) follow one of two approaches, declarative or imperative. As I mentioned above, snowchange follows an imperative-style approach to DCM. For a brief discussion on the differences between the declarative and imperative approaches, please read my previous blog post, Embracing Agile Software Delivery and DevOps with Snowflake.
snowchange was originally created in 2018 by James Weakley (check out James’ other exciting Snowflake related projects on GitHub). I recently became a co-maintainer of snowchange and have worked closely with James on this updated version of snowchange. We’re both excited to share this new version with you!
How does snowchange work?
This section contains a very high-level overview of how snowchange works. For the details, see the snowchange GitHub repository.
A snowchange change script is nothing more than a file that contains one or more SQL statements. Each change script name must include a unique version number (for example, V1.1__initial_database_objects.sql) that is used to ensure the scripts are run in the correct order against a target Snowflake database. The change scripts are stored under source control in a flexible directory structure.
When snowchange runs, it loops through each database folder and applies any missing changes to the target Snowflake database. The state of each target database is tracked in a special change history table named SNOWCHANGE.CHANGE_HISTORY (which is automatically created and maintained by snowchange). snowchange run scripts only against the target database with a version number larger than the max version stored in the change history table. This way, only “new” scripts are applied to the target database each time snowchange is run.
Finally, because snowchange is written in Python it can be easily included in a CI/CD pipeline. All popular CI/CD tools provide hosted agents (or runners) with Python installed, so there is no need to create a self-hosted agent or container to run snowchange!
Getting started with snowchange
To get started with snowchange, please visit the GitHub repository for snowchange. The README.md file provides a detailed description of how to use the tool.
In addition to providing getting started information, the README.md file contains details about how to run the included demo and how to integrate snowchange with CI/CD tools and processes.
So what’s next?
A DCM tool is critical for managing your Snowflake objects and implementing CI/CD pipelines. But DCM tools have been around for a long time now and aren’t well suited for certain aspects of managing your database objects. I’m incredibly excited about dbt (data build tool) and think it’s a real game changer when it comes to managing your data transformations, including many of the related objects in your database.
In my next blog post, I will discuss a new approach for managing all your Snowflake database objects using a combination of snowchange and dbt! Stay tuned.
