Ask Your Question
1

Pipeline fetches data repeatedly

asked 2018-06-21 07:44:57 -0600

Shruthi gravatar image

updated 2018-07-05 23:56:02 -0600

I am fetching data from mysql using JDBC Query Consumer. If it had to fetch some 10 records it fetches 10. After that it doesn't stop fetching and the count keep on increasing. After fetching the actual 10 records it again fetches same 10 records and the count goes like 20...30...40..so on..Why it is fetching data repeatedly?

Thanks in advance

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2018-06-21 09:20:44 -0600

metadaddy gravatar image

It sounds like the JDBC Query Consumer origin is in Full mode, and is configured to repeat the query according to the configured Query Interval. It's doing exactly what it's supposed to do with this configuration. This is intended for cases where you need to overwrite data in a destination periodically.

If you want the pipeline to stop once the query is complete, you need to add a Pipeline Finisher executor, triggered by the origin's no-more-data event. This case study shows how.

edit flag offensive delete link more

Comments

1

It worked. Thank you @metadaddy

Shruthi gravatar imageShruthi ( 2018-06-22 00:23:22 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2018-06-21 07:44:57 -0600

Seen: 142 times

Last updated: Jul 05