File Formats (Supported Formats & File Ingestion)
1. Automatic File Format Detection
By default, Nexla automatically detects the format of files ingested from data sources and parses the data contained in the files accordingly.
Nexla detects the format of each file ingested from a data source based on its file extension—for example, .json, .csv, .avro, etc. Format detection is performed for each ingested file, meaning that automatic file format detection works no matter whether the data source contains one or many file types.
In step 3. Configure when creating a new data source, the Automatically Detect Format option is selected as the default setting in the File Content Format menu located in the Advanced Settings panel on the right side of the screen.
No further action is required to enable automatic file format detection.
Nexla's automatic file format detection is appropriate for most use cases and should typically be left as the selected File Content Format setting.
However, the platform can be forced to parse ingested files according to a specified file format, as discussed in Section 2.
1.1 Nexsets & Using Automatic File Detection
Nexla's automatic file format detection feature is suitable for most use cases, including when data sources contain files in more than one format. Automatic file format detection enables the ingestion and processing of data from the source regardless of the file format(s) and should be used whenever possible.
Once data is ingested, it is organized into a Nexset, which is a data product that does not have a file type. Nexsets provide a complete understanding of ingested data, and the data that they contain can be sent to destinations in a file format different from that of the ingested file.
For more information about sending data to a destination in a different format from that of the ingested file, see Convert Data into a Different File Format.
2. Specifying a File Format
For specialized use cases, users can designate a specific incoming file format for a data source, forcing Nexla to read all files ingested from the data source according to the designated file format.
Nexla's automatic file format detection feature is suitable for most use cases, including when data sources contain files in more than one format. Automatic file format detection enables the ingestion and processing of data from the source regardless of the file format(s) and should be used whenever possible.
To specify a file format for a data source when creating or editing a data source:
In step 3. Configure, select the desired file format from the File Content Format pulldown menu in the Advanced Settings panel on the right side of the screen.
- See Section 3 for a list of file format options currently available in the File Content Format menu.
For more information about the advanced settings available for file-based data sources in Nexla, see Advanced Settings for File-Based Sources.
3. Supported File Formats
Nexla is continuously updated to support new file formats, so the list shown in this section may not include all supported file formats.
Below is a list of file format settings that Nexla currently supports under the File Content Format menu:
- AVRO (.avro)
- Comma-Separated Values (.csv)
- Custom Text File (.txt) – can be used to support arbitrarily delimited files, such as pipe-separated, caret-separated, or any other character-separated file formats
- EDI (.edi)
- Excel (.xlsx, .xls)
- Fixed-Width Files – files containing data in a format specified by column widths, pad characters, and/or left–right alignment
- JSON (.json)
- Log Files (.log)
- ORC (.orc)
- Parquet (.parquet)
- PDF (.pdf)
- SWIFT (.swift)
- Tab-Separated Values (.tsv)
- Unstructured Files – files with unstructured data content, i.e, files with an internal structure that does not follow a format predefined by data models
- XML (.xml)
3.1 Custom File Formats
Nexla can easily be extended to work with any custom file format. To discuss support for a custom file format, contact your Nexla account representative.