No-more-data Event Generation Delay (seconds) Not Working

asked 2019-11-12 21:02:00 -0500

Meihong gravatar image

I used JDBC Query Consumer origin to generate events for data exception/log handling. I set "No-more-data Event Generation Delay (seconds)" configuration to be "180" instead of default 10. But it doesn't work.

The problem I met is: when there is no data, it continues generate file every 10 sec instead of every 180 seconds. Therefore, I have a lot of useless files collected.

edit retag flag offensive close merge delete


Event generation is not the same as files being created. Can you share more details? What's the destination, batch size, etc? Also, what are you doing with "No-more-data" events?

iamontheinet gravatar imageiamontheinet ( 2019-11-13 09:47:55 -0500 )edit

Hi Iamontheinet, The destination is a SDC PRC pipeline, which will connect to another SDC PRC pipeline and distinated to S3 bucket which will collect files generated by jdbc query consumer (if i'm not wrong).

Meihong gravatar imageMeihong ( 2019-11-13 19:04:11 -0500 )edit

JDBC Query Consumer has "query interval" under "JDBC" tab set to 10 seconds, so when no new data coming in, every 10 seconds a new file is created and send to S3. I suspect it's bcz I tick the "Produce Events" option under "General" tab which I can receive the files. Correct me if I'm wrong.

Meihong gravatar imageMeihong ( 2019-11-13 19:05:59 -0500 )edit