Transformation Processing

Each transformation is processed according to the settings in the Processing Config.

Processing Configuration TabProcessing Configuration Tab

Processing Configuration Tab



Update Type

Update types control how Narrator processes your transformation when inserting data into your activity stream, enrichment table, or customer table.

Update Type

Use for...

Incremental updates

Full Reconciliation

On merged to prod...

Regular

Recommended for most activities.

This processing type is suited for data that is generated chronologically.

On Activity Stream update

Nightly

Deletes historical data and insert new data

Mutable

Data sources where new data is generated in non-chronological order.

On Activity Stream Update

Deletes historical data and insert new data

Single Run

One-time insert. Great for static data or historical migrations where data isn't actively updating.

One-time

Inserts new data

Materialized View

Customer Tables

Deletes historical data and insert new data

Note: Updates types are set by default based on the transformation type. Almost all activity and enrichment transformations will use the Regular update type. Customer transformations will always use the Materialized View update type.


  • Regular - Update the Activity Stream by transforming new data that was generated since the last update (using the timestamp column) and appending these new activity records to the activity stream. When the transformation is modified, the processing will delete historical data and insert new data using the new SQL logic.

  • Mutable - (Rarely Used) Runs a full historical update with each activity stream update. This can be inefficient so this update type is only used in special circumstances. Use this update type when the incremental update has a high likelihood of missing activities. When the transformation is modified, the processing will delete historical data and insert new data using the new SQL logic.

  • Single Run - Used for a one-time insert of data into the activity stream. It should be used if the underlying data is static (ex. historical data source that is no longer in use). When the transformation is modified, the processing will delete historical data and insert new data using the new SQL logic.

  • Materialized View - (Used for customer tables) With each run, the old data is deleted and all data is reprocessed and inserted. It updates according to the schedule set in your processing settings.



Specify Unseen Activities

(Activity Transformations)
After validation has been run, the activities that are generated by the transformation will automatically show up under Activities in the Processing Config. Because the validation is only performed using the last 3 months of data, it is possible that some activities will be missed. In this case you can manually specify the unseen activities under the Create New Activity input. This will create an Activity on the Activities tab and you'll be able to modify the activity details on that page.



Specify Enriched Activities

(Enrichment Transformations)
An enrichment transformation needs to be linked to the activities it enriches in order for the columns to show up on the platform.

Enrichment transformations can enrich multiple activities as long as they use the same activity_id as the enriched_activity_id. The enrichment table will be joined to the Activity Stream using these columns as the key.

How do I link an activity with its enrichment table?

Note: The link between Enrichment Transformations and Enriched Activities can be specified from the Processing Config or the Activity pages.



Advanced Configuration

Processing

  • Requires Identity Resolution If an activity transformation's SQL includes a non-NULL source, then this setting will be automatically applied. Any transformation that has a source and source_id will have identity resolution applied after the data has been updated in the activity stream, but you can override the defaults in the Advanced Configuration settings.

  • Process as an Aliasing Activity An aliasing is a special type of activity that can be used when the universal identifier for a customer changes, but you want to ensure the historical data and future data is associated with the same customer ID. An aliasing activity can be used to link the customer identifiers. More information on Aliasing Activities can be found here.

  • Delete Recent Days This setting will delete the data from the most recent days (the number of days is specified in the settings) before incrementally updating the data in the activity stream or enrichment table. This can be helpful when it takes a few days for data to stabilize (usually counts or aggregates for daily reporting) and you don't want to add the most recent data until the data has stopped changing.

  • Allow Future Data By default, Narrator will not add activities to the activity stream if the timestamp is in the future. If you want to override this session (ie. capture events that will happen in the future), you can check the allow future data setting. This will remove the timestamp constraint and allow you at add future data to your activity stream.



Dependency

Dependency management can be important when you define an activity that is derived from another activity.

  • Shares Dependencies With This can be helpful when an activity is derived from another activity. When the "Depends On" transformation is resynced or modified, the current transformation will also be resynced.
  • Run After Ensures that the current transformation runs after the specified transformation. This is helpful when a transformation is derived from another activity AND user mapping must be completed in a specific order. When to use Run After


Resyncing

  • Do Not Delete Old Data When Resyncing Check this setting if you don't want to reprocess the historical data.
  • Only Process Data After Check this setting to only re-sync data starting after a certain date. This can be helpful if there is a lot of historical data and you only want to reprocess the more recent data.



Reconciliation

  • Nightly Reconciliation
    Narrator runs a diff of the activity stream against the source data to see if records are missing. Because this process can be computationally expensive, Narrator will only diff the last 15 days by default. If you wish to change the Days to Diff for a particular transformation, you can do so under the Advanced Configurations.




Still have questions?


Our data team is here to help! Here are a couple ways to get in touch...

๐Ÿ’ฌ Chat us from within Narrator

๐Ÿ’Œ Email us at [email protected]

๐Ÿ—“ Or schedule a 15 minute meeting with our data team


Whatโ€™s Next
Did this page help you?