Getting Started with Snowflake and SDF

SDF as a best-in-class transformation layer for Snowflake

Overview

SDF can hydrate table schemas and execute queries against a Snowflake DB. This guide will walk you through the steps of creating a Snowflake integration. By the end, you will be able to reference tables in your snowflake instance in SDF.

SDF will use remote Snowflake schema information to do type bindings and other static analysis checks on your workspace.

Prerequisites

In order to connect to Snowflake, you will need to have on hand the following:

1. A Snowflake account to connect to.
2. Valid credentials with read access to materialize tables in your Snowflake account.

Guide

1. Collect Required Information

   To connect to Snowflake, you need the following materials:

   • Account ID - the 8 digit organization identifier
   • Username - your Snowflake username
   • Password - your Snowflake password
   • Role - a role with appropriate permissions
   • Warehouse - The warehouse you are connecting to.

   Log in to your Snowflake account to find the relevant information.

   

   Run sdf auth login snowflake --help to see all login options.

2. Connect SDF to Snowflake

   Running the following command will prompt you to choose authentication method.

   sdf auth login snowflake --account-id <ACCOUNT_ID> --username <USERNAME> --role <ROLE> --warehouse <WAREHOUSE_NAME>
   

   Password Authentication

   Connect using username and password

   Choose the password option and enter your password when prompted.

   Alternatively include the password directly inline:

   sdf auth login snowflake --account-id <ACCOUNT_ID> --username <USERNAME> --role <ROLE> --warehouse <WAREHOUSE_NAME> --password <PASSWORD>
   

   Key Pair Authentication

   Connect using key pair authentication

   Choose private key path or private key pem option and enter the path or pem content to your private key when prompted.

   • Supported key formats: PKCS#1, PKCS#8

   If you have a PEM file, opt to use the --private-key-path flag to pass the path to the file.

   • If you are determined to paste the pem directly or choose to parameterize private-key-pem in a credential.sdf.yml file with an environment variable, replace newlines with \n and include key delimiters.
   • In both cases, ensure the PEM is a single line with \n as newline characters and no extra whitespace.

   Alternatively, you can provide the private key information directly inline:

   sdf auth login snowflake --account-id <ACCOUNT_ID> --username <USERNAME> --role <ROLE> --warehouse <WAREHOUSE_NAME> --private-key-path <PATH_TO_PRIVATE_KEY> --private-key-passphrase <PASSPHRASE>
   

   Ensure you have generated and registered your key pair with Snowflake before using this method. Please refer to the Snowflake documentation for more information on setting up key pair authentication.

   This will create a new credential in a ~/.sdf/ directory in the root of your system. This credential will be used to authenticate with Snowflake.

   SDF supports Duo push notifications for Snowflake Multi-Factor Authentication (MFA). When MFA is enabled for your Snowflake account, you’ll be prompted to complete the MFA step during authentication using Duo push.

   To minimize MFA prompts during authentication, Snowflake allows MFA token caching. For detailed instructions on setting this up, refer to the Snowflake documentation on MFA token caching.

3. Add Snowflake Provider in Workspace

   Once authenticated, add an integration block to your workspace.sdf.yml. This tells SDF to use Snowflake to hydrate missing table schemas.

   workspace:
       ...
       integrations:
           - provider: snowflake
             type: database
             sources: 
               - pattern: <DATABASE>.*.*
   

   Replace <DATABASE> with the name of the database you want to hydrate. Note this is configurable and can be changed to any database you have access to. For example, if I wanted SDF to pull from two databases, called db1 and db2, I would write:

   workspace:
       ...
       integrations:
           - provider: snowflake
             type: database
             sources: 
               - pattern: db1.*.*
               - pattern: db2.*.*
   

   Integrations can also be configured to use specific credentials by referencing the credential name in the integration block. For example, if you wanted to use a credential called snowflake-creds, you would write:

   workspace:
       ...
       integrations:
           - provider: snowflake
             type: database
             credential: snowflake-creds
             sources: 
               - pattern: db1.schema1.mytable
               - pattern: db2.schema2.*
   

   For more information on integration configuration, see the integration documentation.

4. Try it out!

   Now that you’re connected, let’s make sure SDF can pull the schema information it needs.

   Run sdf compile -q "select * from <DATABASE>.<SCHEMA>.<TABLE>" --show all

   If the connection is successful, you will see the schema for the table you selected.

The term ‘DATABASE’ in Snowflake is interchangeable with the term ‘CATALOG’ in SDF.

Next Steps

Now that you have connected to Snowflake, you can start materializing tables in Snowflake. Check out the Snowflake materialization guide to get started.