A task is the job that processes the data in your data system. Each task runs on a schedule that can be managed from the Processing Tab.
Run Transformations is the main task that processes all of the data in your data system. It performs the following:
Updates Tables with New Data
Narrator uses each SQL transformation to transform your source data into a structure that is compatible with the Activity Stream or enrichment tables and inserts the new data into the relevant tables.
Applies Identity Resolution on the Activity Stream
Narrator applies identity resolution to the the Activity Stream. Using the
customercolumns on each activity, the data processing re-generates the user mapping and updates the customer values in the Activity Stream to associate each (previously anonymous) activity with a customer where possible.
Generates auto-computed columns
Once the data in the Activity Stream has been updated, Narrator computes a few additional columns to improve query efficiency when the table is used to construct datasets. Those columns are:
activity_occurrence: The instance of this activity for each customer (the number of times a customer completed a specific activity at the point that the activity occurred).
activity_repeated_at: The timestamp of the next time the customer completed that activity.
Runs Full Reconciliation on Activity and Enrichment Tables Once Nightly
This process diffs the data against what is in production over the last 15 days and reconciles conflicting records. This handles cases where data was backfilled or EL was delayed.
This operation is computationally expensive because it requires a comparison to all records historically, so it is performed once per day.
Redshift warehouses only
Runs a VACUUM command on your tables to improve query performance.
Finds all new customers that have been added to the activity stream since the last update and adds them to the customer index so they can be quickly searched in the Customer tab.
Our data team is here to help! Here are a couple ways to get in touch...
💬 Chat us from within Narrator
💌 Email us at [email protected]
🗓 Or schedule a 15 minute meeting with our data team
Updated 7 months ago