Schema Change Detection

his article provides information about Nexla's schema change detection feature, which protects the consistency and reliability of data workflows when the structure of incoming data changes. It also includes instructions for setting up schema change notifications and configuration options that determine how the platform should handle schema changes.

1. Schema Change Detection in Nexla

Each time Nexla ingests data from a data source, it not only reads the data but also analyzes it. Through this analysis, ingested data is automatically organized into records, attributes, and overall schema in detected Nexsets.

Once Nexla has detected a Nexset from a data source, in each subsequent ingestion cycle, the platform uses active metadata intelligence to compare the structure of newly ingested data to the existing Nexset schema.

When a significant change in the structure of ingested data is detected, the newly ingested data is organized into a new detected Nexset. However, when smaller changes that are unlikely to impact downstream processes in the workflow are detected, such as the addition of a new attribute, Nexla will automatically update the existing Nexset schema to include the newly detected data. The platform will also create a log of the change, as well as send notifications of the schema change according to settings configured in the account (discussed in Section 2).

Nexla's schema change detection ability allows users to monitor and understand how their data changes over time, as well as ensure that any downstream operations remain stable and consistent.

2. Configure Schema Change Notifications

By default, Nexla is configured to produce a notification each time a schema change is detected. With the default settings, these notifications can be viewed inside the platform, and a notification email will be sent to the user. However, Nexla can be configured to deliver schema change notifications to Slack channels, webhook services, and multiple email IDs.

1. To configure schema change notifications in Nexla, navigate to the Monitor section by selecting from the platform menu on the left side of the screen.

2. Select from the Monitor toolbar on the left to access the frequency and management configuration options available in the Notification Settings screen.

     

3. Select the tab.

     

4. The following subsections provide instructions for configuring each of the available notification delivery methods in this tab.

     

2.1 Platform Notifications

1. To receive notifications about detected schema changes within the Nexla platform, check the box in the APP column for the "Schema Change" notification type.

   This setting is enabled by default. To disable platform notifications about detected schema changes, uncheck this box.

     

2. To view schema change detections in Nexla:

   1. Click the icon on the detected Nexset.

        

   2. Select the tab.

   3. Any schema change notifications for the selected Nexset will be shown in this tab.

        

   4. Click on the notification to view the modifications implemented in this schema change.

        

2.2 Email Notifications

1. To receive notifications about detected schema changes by email, check the box in the EMAIL column for the "Schema Change" notification type.

   This setting is enabled by default. To disable email notifications about detected schema changes, uncheck this box.

     

2. In the Channel Settings panel on the right side of the screen, click to view email notification settings.

   Email notification settings available in the Channel Settings panel apply to all notification types for which email delivery is enabled.

     

3. Enter one or more email addresses to which Nexla should send notifications in the Emails to Notify field.

     

   • Use commas to separate multiple email addresses in this field.

       

4. Click to save the email notification settings. Notifications about detected schema changes will now be sent to the email(s) entered in Step 3 above.

     

2.3 Slack Notifications

Note: To receive notifications about detected schema changes in a Slack channel, a Slack webhook must first be configured. If a Slack webhook has previously been configured and saved in Nexla, skip to Step 5.

1. Click in the Channel Settings panel on the right side of the screen to view Slack notification settings.

   Slack notification settings available in the Channel Settings panel apply to all notification types for which Slack delivery is enabled.

     

2. Click to create and configure the incoming webhook on the Slack website.

   For more information about configuring a Slack incoming webhook, see this Slack article.

     

3. Enter the URL of the configured Slack webhook in the Slack Webhook URL field.

     

4. Click to save the Slack notification settings.

     

5. Check the box in the SLACK column for the "Schema Change" notification type. Notifications about detected schema changes will now be sent to the Slack channel associated with the webhook URL entered in Step 3 above.

     

2.4 Webhook Notifications

1. To receive notifications about detected schema changes through a custom webhook for services such as Pager Duty, check the box in the WEBHOOKS column for the "Schema Change" notification type.

     

2. In the Channel Settings panel on the right side of the screen, click to view webhook notification settings.

   Webhook notification settings available in the Channel Settings panel apply to all notification types for which webhook delivery is enabled.

     

3. Enter the URL of the webhook that should be used to deliver notifications in the Webhook URL field.

     

4. Click to save the webhook notification settings. Notifications about detected schema changes will now be sent to the webhook URL entered in Step 3 above.