Ask Your Question

Pipeline fetches data repeatedly

asked 2018-06-21 07:44:57 -0500

Shruthi gravatar image

updated 2018-07-05 23:56:02 -0500

I am fetching data from mysql using JDBC Query Consumer. If it had to fetch some 10 records it fetches 10. After that it doesn't stop fetching and the count keep on increasing. After fetching the actual 10 records it again fetches same 10 records and the count goes like on..Why it is fetching data repeatedly?

Thanks in advance

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-06-21 09:20:44 -0500

metadaddy gravatar image

It sounds like the JDBC Query Consumer origin is in Full mode, and is configured to repeat the query according to the configured Query Interval. It's doing exactly what it's supposed to do with this configuration. This is intended for cases where you need to overwrite data in a destination periodically.

If you want the pipeline to stop once the query is complete, you need to add a Pipeline Finisher executor, triggered by the origin's no-more-data event. This case study shows how.

edit flag offensive delete link more



It worked. Thank you @metadaddy

Shruthi gravatar imageShruthi ( 2018-06-22 00:23:22 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-06-21 07:44:57 -0500

Seen: 158 times

Last updated: Jul 05 '18