Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Frequently missing Records from streamSets pipeline with a custom ldap connector as the origin stage

Hi everyone,

I am observing that everyday there are few records that have not been ingested (with respect to the day before) by my ingestion streamSets pipeline. In other words, it succeds to ingest a large majority of data from the source, but for the rest, it fails to ingest. The pipeline uses a custom Ldap connector (java code) as its origin stage. Thre are no signs of the missing records in the log file. Do you have any tips or guesses about the cause of this issue? Of course, I know that there might be dozens of things as the root cause of the issue, but any hint is appreciated! :) Let me add that the pipeline works well with limited amount of data, but it fails to ingest some records (about 1%) when the flow of data is continous.

Thanks, Amir