Migrating to Balto

Balto is 100% compatible with existing dbt projects, however to use Balto specific features, like schedules and triggers, you'll need to install the Balto CLI tool.

Migrating to the Balto CLI

To install the Balto CLI simply install it via pip:

pip install balto-core

If you're project config specifies a dbt version you'll need to remove it, or adjust it for the current balto version:

require-dbt-version: [">=0.1.0", "<1.0.0"]

Migrating deploys to the Balto Snowflake Native App

You can immediately being deploying you project to a new target schema, however if you wish to use an existing schema you must grant a few required permissions. Additionally, since Balto deploys use the state:modified+ selector to determine which models to build, you must upload a current compiled manifest.json so that Balto can accurately determine what state has been modified.

Granting Permissions

Balto requires all permissions on its target database:

grant all on target_database to application balto;

For any existing schemas and schema objects you want to manage with Balto you must transfer ownership over to the Balto application (this is required to support certain materialization behavior, e.g. doing a full refresh on an incremental model requires ownership on the table so that it can be dropped).

Snowflake's permission model does not allow you to directly grant ownership on objects to an application, however the Balto application can create a clone of your existing tables/views and then the originals can be dropped. The Balto CLI provides built-in macro (transfer_ownership) that can be run via the run-operation command that automates this process for you. To use the macro you must first grant the Balto app all privileges on the existing objects:

grant all on schema jaffle.test to application balto;
grant all on all tables in schema jaffle.test to application balto;
grant all on all dynamic tables in schema jaffle.test to application balto;
grant all on all views in schema jaffle.test to application balto;

And then you can trigger the macro from the CLI

balto run-operation transfer_ownership --args '{database: jaffle, schema: test}'

Note

You must use a profile with the role that owns the existing objects when executing the transfer_ownership macro

This will transfer ownership of all tables and views in the jaffle.test schema to the Balto application by: 1. Cloning all tables and dynamic tables, and recreating any views using a stored procedure that executes as the Balto application 2. Dropping the original tables and views 3. Renaming the cloned objects to the original name

Uploading Existing state

To migrate an existing project to the Balto Snowflake Native App you must either rebuild the entire project from scratch on your first deploy, or upload the most recently deployed manifest to Balto so that in can be used to determine what state has changed on subsequent deploys.

To upload an existing manifest simply upload your manifest.json to a stage, ingest it into a table, and then call the balto.dbt.create_project procedure with the manifest as the only argument:

create or replace table manifest (manifest variant);
create temporary stage manifests;
PUT file://test_project/target/manifest.json @manifests;
copy into manifest from @manifests FILE_FORMAT = (TYPE = 'JSON');

CALL balto.dbt.create_project((select manifest from jaffle.ops.manifest limit 1));

The first time you call balto.dbt.deploy (or push changes via our Github integration), the manifest passed to create_project will be used to diff against the manifest being deployed which will determine what models need to be built.