Validation
All transformations must be validated before they can can be merged to production.
Validating Transformations
Narrator runs a series basic tests to ensure all transformations will process correctly. After you save a transformation, you'll be able to run validation.
These tests check the core assumptions that are used when updating the activity stream, customer, and enrichment tables. When these assumptions fail then the data doesn't process correctly, so the validation step will help catch processing issues before you merge to production.
Successful Validation for an Activity Transformation
Tests
Validation tests required
| Test Name | Details |
|---|---|
| Check for LIMIT Clauses | LIMIT is not used to limit the data output of the transformation |
| Check for Missing Columns | All required columns are available |
| Check for NULL Values in Required Columns | activity_id, ts, and activity values are never NULL for Activity Transformations, enriched_activity_id , enriched_ts are never NULL for Enrichment Transformations, customer is never null for Customer Transformations |
| Check for ID Duplication | activity_id or enriched_activity_id is unique. For Customer Transformations, the test checks that customer is unique. |
Note: Additional tests can be manually run on the transformation from the Validation Log tab. More information on the additional validations can be found here.
Data Sample Size for Validation
By default, Narrator uses the last 3 months of data to run the validation tests. Users can control the validation months by going to Company Settings > Processing Configuration. Learn more.
Custom Validation / Custom Alerts
Set up a custom alert to get notified if assumptions about the data changes. You can define a custom SQL query and set up an email alert if the query returns rows or doesn't return rows. This is a helpful way to ensure the assumptions you made when defining the transformation stay true over time.
You can schedule your custom alert to run the query regularly using the cron schedule and it can be managed from the processing page (under the Alerts tab).
Custom Validation Set Up
How to: Use Custom Validations (Custom Alerts)
Watch this step-by-step tutorial to set up a custom validation.
Troubleshooting tips
Still have questions?
Our data team is here to help! Here are a couple ways to get in touch...
💬 Chat us from within Narrator
💌 Email us at [email protected]
🗓 Or schedule a 15 minute meeting with our data team
Updated over 2 years ago