Documentation

Data Processing

Understand how Narrator uses transformations to update the tables in your data system

Suggest Edits

Processing Overview

A task is the job that processes the data in your data system. Each task runs on a schedule that can be managed from the Processing Tab.


Tasks

Run Transformations

Run Transformations is the main task that processes all of the data in your data system. It performs the following:

  • Updates Tables with New Data
    Narrator uses each SQL transformation to transform your source data into a structure that is compatible with the Activity Stream or enrichment tables and inserts the new data into the relevant tables.

  • Applies Identity Resolution on the Activity Stream
    Narrator applies identity resolution to the the Activity Stream. Using the anonymous_customer_id and customer columns on each activity, the data processing re-generates the user mapping and updates the customer values in the Activity Stream to associate each (previously anonymous) activity with a customer where possible.

  • Generates auto-computed columns
    Once the data in the Activity Stream has been updated, Narrator computes a few additional columns to improve query efficiency when the table is used to construct datasets. Those columns are:

    • activity_occurrence: The instance of this activity for each customer (the number of times a customer completed a specific activity at the point that the activity occurred).
    • activity_repeated_at: The timestamp of the next time the customer completed that activity.

  • Runs Full Reconciliation on Activity and Enrichment Tables Once Nightly
    This process diffs the data against what is in production over the last 15 days and reconciles conflicting records. This handles cases where data was backfilled or EL was delayed.
    This operation is computationally expensive because it requires a comparison to all records historically, so it is performed once per day.


Vacuum Tables

Redshift warehouses only
Runs a VACUUM command on your tables to improve query performance.


Indexing Customers

Finds all new customers that have been added to the activity stream since the last update and adds them to the customer index so they can be quickly searched in the Customer tab.



FAQ




Still have questions?

Our data team is here to help! Here are a couple ways to get in touch...

šŸ’¬ Chat us from within Narrator

šŸ’Œ Email us at [email protected]

šŸ—“ Or schedule a 15 minute meeting with our data team

Updated about 3 years ago

Whatā€™s Next