Transformation Processing

Each transformation is processed according to the settings in the Processing Config.

Update Type

Update types control how Narrator processes your transformation when inserting data into your activity stream, enrichment table, or customer table.

Note: Updates types are set by default based on the transformation type. Almost all activity and enrichment transformations will use the Incremental update type. Customer transformations will always use the Full Data Resync update type.

• Incremental - Update the Activity Stream by transforming new data that was generated since the last update (using the timestamp column) and appending these new activity records to the activity stream. When the transformation is modified, the processing will delete historical data and insert new data using the new SQL logic.

• Insert Missing Rows Only - (Rarely Used) Runs a full historical update with each activity stream update. This can be inefficient so this update type is only used in special circumstances. Use this update type when the incremental update has a high likelihood of missing activities. When the transformation is modified, the processing will delete historical data and insert new data using the new SQL logic.

• Sync Once - Used for a one-time insert of data into the activity stream. It should be used if the underlying data is static (ex. historical data source that is no longer in use). When the transformation is modified, the processing will delete historical data and insert new data using the new SQL logic.

• Full Data Resync (aka Materialized Views) - (Used for customer tables) With each run, the old data is deleted and all data is reprocessed and inserted. It updates according to the schedule set in your processing settings.
  You can also use this for activities if your data is always changing and you just want a fresh copy. If just recent data is changing I highly recommend doing Incremental with a "delete recent days" to only resync a window of the data.

Specify Enriched Activities

(Enrichment Transformations)
An enrichment transformation needs to be linked to the activities it enriches in order for the columns to show up on the platform.

Enrichment transformations can enrich multiple activities as long as they use the same activity_id as the enriched_activity_id. The enrichment table will be joined to the Activity Stream using these columns as the key.

How do I link an activity with its enrichment table?

Note: The link between Enrichment Transformations and Enriched Activities can be specified from the Processing Config or the Activity pages.

Advanced Configuration

Processing

• Requires Identity Resolution If an activity transformation's SQL includes a non-NULL source, then this setting will be automatically applied. Any transformation that has an anonymous_customer_id will have identity resolution applied after the data has been updated in the activity stream, but you can override the defaults in the Advanced Configuration settings.

• Process as an Aliasing Activity An aliasing is a special type of activity that can be used when the universal identifier for a customer changes, but you want to ensure the historical data and future data is associated with the same customer ID. An aliasing activity can be used to link the customer identifiers. More information on Aliasing Activities can be found here.

• Delete Recent Days This setting will delete the data from the most recent days (the number of days is specified in the settings) before incrementally updating the data in the activity stream or enrichment table. This can be helpful when it takes a few days for data to stabilize (usually counts or aggregates for daily reporting) and you don't want to add the most recent data until the data has stopped changing.

• Allow Future Data By default, Narrator will not add activities to the activity stream if the timestamp is in the future. If you want to override this session (ie. capture events that will happen in the future), you can check the allow future data setting. This will remove the timestamp constraint and allow you at add future data to your activity stream.

• Enable Anomaly Detection You can get automatically notified and (optionally) pause processing before your activity transformation makes an anomalous update. Learn More

Dependency

Dependency management can be important when you define an activity that is derived from another activity.

• Shares Dependencies With This can be helpful when an activity is derived from another activity. When the "Depends On" transformation is resynced or modified, the current transformation will also be resynced.
• Run After Ensures that the current transformation runs after the specified transformation. This is helpful when a transformation is derived from another activity AND user mapping must be completed in a specific order. When to use Run After
  
  
  

Resyncing

• Do Not Delete Old Data When Resyncing Check this setting if you don't want to reprocess the historical data.
• Only Process Data After Check this setting to only re-sync data starting after a certain date. This can be helpful if there is a lot of historical data and you only want to reprocess the more recent data.

Reconciliation

• Nightly Reconciliation
  Narrator runs a diff of the activity stream against the source data to see if records are missing. Because this process can be computationally expensive, Narrator will only diff the last 15 days by default. If you wish to change the Days to Diff for a particular transformation, you can do so under the Advanced Configurations.

Still have questions?

---

Our data team is here to help! Here are a couple ways to get in touch...

💬 Chat us from within Narrator

💌 Email us at [email protected]

🗓 Or schedule a 15 minute meeting with our data team