Metrics

Overview

Note

Balto's Metrics are different than dbt's concept of Metrics used in the Semantic Layer.

Metrics provide a built-in mechanism for observing both how your data changes and how you're data pipelines are running. Metrics let you push custom metrics directly to your monitoring stack from Snowflake when executing your Balto projects.

Pushing Metrics

The Balto Orchestrator provides a built-in UDF that allows you to push custom metrics to Datadog. Combine metrics and tiggers to monitor your data in near real time.

Configuration

Before you can start pushing metrics you must create an external access integration for the Datadog api and configure a Datadog api/app key.

First create an api and app key through the Datadog UI. Once you have your API and application key, run the following commands:

call balto_alpha.config.setup_datadog('API_KEY', 'APP_KEY');

CREATE OR REPLACE NETWORK RULE test.ops.datadog_egress
    MODE = EGRESS
    TYPE = HOST_PORT
    VALUE_LIST = ('us5.datadoghq.com');

CREATE OR REPLACE EXTERNAL ACCESS INTEGRATION datadog_access_integration
ALLOWED_NETWORK_RULES = (test.ops.datadog_egress)
ALLOWED_AUTHENTICATION_SECRETS = (balto_alpha.config.datadog_api_key, balto_alpha.config.datadog_app_key)
ENABLED = true;

CALL balto_alpha.config.register_single_reference(
    'datadog_integration',
    'ADD',
    SYSTEM$REFERENCE(
        'external_access_integration',
        'datadog_access_integration',
        'PERSISTENT',
        'USAGE'
    )
);

You should now be able to use the built-in balto.dbt.metric_handler procedure when defining your triggers.

Limitations

Balto currently only supports Datadog as a metric endpoint.