Recover failed ingestion data

Review DLQ entries, download failed data, and reingest it after resolving the issue.

When data fails to ingest into Sift, the error is logged in the Dead Letter Queue (DLQ) and surfaced in the Data Processing page of the Manage workspace. From here you can review DLQ entries, download the data along with the reasons why it failed, resolve the issue, and reingest the data.

Before you begin

• You are familiar with how Sift organizes data into Assets, Channels, and Runs.

How Data Processing works

Data processing errors are logged in the Dead Letter Queue (DLQ) when ingestion fails due to invalid values or parameters. These errors can be reviewed by navigating to the Manage section and selecting Data Processing in the Sift interface.

The following table describes the most common error types:

The following table describes how long error data is retained in Sift:

Review and download DLQ data

1. Click your profile icon.
2. Select Manage.
3. In the Manage navigation menu, click Data Processing.
4. In the Data Processing table, locate the error record you want to investigate.
5. Click Export to download the data and the reasons why it was sent to the DLQ.
6. Review the downloaded data to understand the issue.
7. Resolve the issue independently or work with Sift to resolve it, then reingest the data.

Recover data lost during a streaming interruption

A brief network interruption during streaming ingestion is not always retried indefinitely. The streaming ingestor’s retry policy uses exponential backoff: if a connection cannot reconnect quickly, retry delays reach the multi-second range, and data for that window can be permanently dropped instead of automatically backfilled.

To recover a gap caused by a streaming interruption, export the affected time range (for example, as a Parquet file) and reimport it. See Import telemetry from a file.

Next steps

• To monitor DLQ metrics as time-series data, see Monitor ingestion health.