Frequently missing Records from streamSets pipeline with a custom ldap connector as the origin stage

asked 2019-04-03 04:27:51 -0500

Amir gravatar image

Hi everyone,

I am observing that everyday there are few records that have not been ingested (with respect to the day before) by my ingestion streamSets pipeline. In other words, it succeds to ingest a large majority of data from the source, but for the rest, it fails to ingest. The pipeline uses a custom Ldap connector (java code) as its origin stage. Thre are no signs of the missing records in the log file. Do you have any tips or guesses about the cause of this issue? Of course, I know that there might be dozens of things as the root cause of the issue, but any hint is appreciated! :) Let me add that the pipeline works well with limited amount of data, but it fails to ingest some records (about 1%) when the flow of data is continous.

Thanks, Amir

edit retag flag offensive close merge delete

Comments

Really hard to say without seeing your code!

metadaddy gravatar imagemetadaddy ( 2019-04-04 00:30:03 -0500 )edit