Data destination (also called data sink) resources describe external destinations for output data from specific datasets. Like data sources, they usually require client credentials to allow connecting and writing to the external system.
No matter where the data the data has to be written out, all information where, when, and how to write the data out is contained in these Nexla resources.
Both Nexla API and Nexla CLI support methods to list all destinations in the authenticated user's account. A successful call returns detailed information like id, owner, type, credentials, activation status, and output configuration about all destinations.
Fetch a specific destination accessible by the authenticated user. A successful call returns detailed information like id, owner, type, credentials, activation status, and output configuration about that destination.
In case of Nexla API, add an expand query param with a truthy value to get more details about the destination. With this parameter, full details about the related resources (destination's dataset, credentials, etc) will also be returned.
Both Nexla API and Nexla CLI support methods to create a new data destination in the authenticated user's account.
The only required attribute in the input object is the data destination name; all other attributes are set to default values.
Specify data_set_id to associate what data should be written into that destination, data_credential to authorize the destination location, and sink_config to control how the data should be written out to that destination.
Data destinations usually require some credentials for making a connection and ingesting data. You can refer to an existing data_credentials resource or create a new one in the create data destinations. In this example, an existing credentials object is used:
In either case, a successful POST on /data_sinks with credential information will return a response including the full data destination and the encrypted form of its associated data credentials resource:
Nexla API supports methods to delete any destination that the authenticated user has administrative/ownership rights to. A successful request to delete a data destination returns Ok (200) with no response body.
Nexla API
Delete Destination: Request
DELETE /data_sinks/{data_sink_id}
Nexla API
Delete Destination: Response
Empty response with status 200 for success
Error response with reason if destination could not be deleted
Associate a data set with a destination to control what data will be written out to a destination. Each destination can only have one dataset that writes data to it. You can associate a dataset with a destination by setting the data_set_id property of the destination.
You can control data from being written out by one of two ways:
You can control the status of the associated dataset (see relevant methods in the dataset page). This will prevent dataset from even processing data to be written out.
You can use the methods below to activate or pause the destination. This method is better suited for scenarios where same dataset is configured for writing to multiple destinations.
Nexla API
Nexla CLI
Activate Destination: Request
PUT /data_sinks/{data_sink_id}/activate
On the flip side, call the pause method to immediately stop data write on that destination.
All configuration about where and when to scan data is contained with the sink_config property of a data destination.
As Nexla provides quite a few options to fine tune and control exactly how and where you want to write your data out, it is important to ensure the sink_config contains all required parameters to successfully scan data. To validate the configuration of a given data destination, send a POST request on endpoint /data_sinks/<data_sink_id>/config/validate.
You can send optional json config as input body, if there is no input config in request then stored sink_config will be used for validation.
Nexla API
Validate Destination Configuration: Request
POST /data_sinks/{data_sink_id}/config/validate
Nexla API
Validate Destination Configuration: Response
{
"status":"ok",
"output":[
{
"name":"credsEnc",
"value":null,
"errors":[
"Missing required configuration \"credsEnc\" which has no default value."
],
"visible":true,
"recommendedValues":[]
},
{
"name":"credsEncIv",
"value":null,
"errors":[
"Missing required configuration \"credsEncIv\" which has no default value."
],
"visible":true,
"recommendedValues":[]
},
{
"name":"sink_typ",
"value":null,
"errors":[
"Missing required configuration \"sink_type\" which has no default value.",
"Invalid value null for configuration sink_type: Invalid enumerator"
Lifetime write metrics methods return information about total data written out through a destination since its creation. Metrics contain information about the number of records written out as well the estimated volume of data.
Aggregated write metrics methods return information about total data written out every day from a destination. Metrics contain information about the number of records written out as well the estimated volume of data.
Aggregations can be fetched in different aggregation units. Use the method below to fetch reports aggregated daily:
Nexla API
Nexla CLI
Daily Write Metrics: Request
GET /data_sink/5001/metrics?aggregate=1
...
Optional Payload Parameters:
{
"from": <UTC datetime in '%Y-%m-%dT%H:%M:%S' format>,
"to": <UTC datetime in '%Y-%m-%dT%H:%M:%S' format>,
"page": <integer page number>,
"size": <number of entries in page>
}
Nexla API
Nexla CLI
Daily Write Metrics: Response
{
"status":200,
"metrics":[
{
"time":"2017-02-08",
"record":53054,
"size":12476341
},
{
"time":"2017-02-09",
"record":66618,
"size":15829589
},
{
"time":"2017-02-10",
"record":25832,
"size":6645994
}
]
}
Destination metrics can also be batched by the ingestion frequency of the originating data. Use the methods below to view destination metrics per ingestion cycle.
Nexla API
Nexla CLI
Aggregated By Ingestion Frequency: Request
GET /data_sinks/5001/metrics/run_summary
...
Optional Payload Parameters:
{
"runId": <starting from unix epoch time of ingestion events>,
"from": <UTC datetime in '%Y-%m-%dT%H:%M:%S' format>,
"to": <UTC datetime in '%Y-%m-%dT%H:%M:%S' format>,
Apart from aggregated write metrics methods above that provide visibility into total number of records and total volume of data written out over a period of time, Nexla also provides methods to view granular details about data write events.
You can retrieve data write status of a file destination to find information like how many files have been written out fully, or are queued for being written out.
Nexla API
File Destination Write Status: Request
GET /data_sinks/6745/metrics/files_stats
...
Optional Parameters
{
"from": <UTC datetime in '%Y-%m-%dT%H:%M:%S' format>,
"to": <UTC datetime in '%Y-%m-%dT%H:%M:%S' format>,
"status": "one of NOT_STARTED/IN_PROGRESS/COMPLETE/ERROR/PARTIAL"
}
Nexla API
File Destination Write Status: Response
{
"status":200,
"metrics":{
"data":{
"COMPLETE":17
},
"meta":{
"currentPage":1,
"totalCount":1,
"pageCount":1
}
}
}
You can view write status and history per file of a file destination. The file destination write history methods below return one entry per file by aggregating all write events for each file.
Nexla API
Nexla CLI
Write History Per File: Request
/data_sinks/<data_sink_id>/metrics/files
...
Optional Parameters
{
"from": <UTC datetime in '%Y-%m-%dT%H:%M:%S' format>,
"to": <UTC datetime in '%Y-%m-%dT%H:%M:%S' format>,
"status": "one of NOT_STARTED/IN_PROGRESS/COMPLETE/ERROR/PARTIAL",
You can also bypass per file aggregation and fetch full ingestion history of each file even if it was written out multiple times. This is relevant for scenarios where the destination has been configured to write out the same file name in every ingestion cycle
Nexla API
Raw File Write Status: Request
GET /data_sinks/<data_sink_id>/metrics/files_raw
...
Optional Parameters
{
"from": <UTC datetime in '%Y-%m-%dT%H:%M:%S' format>,
"to": <UTC datetime in '%Y-%m-%dT%H:%M:%S' format>,
"status": "one of NOT_STARTED/IN_PROGRESS/COMPLETE/ERROR/PARTIAL",
Nexla