Ask Your Question

Pipeline fetches data repeatedly

asked 2018-06-21 07:44:57 -0500

Shruthi gravatar image

updated 2018-07-05 23:56:02 -0500

I am fetching data from mysql using JDBC Query Consumer. If it had to fetch some 10 records it fetches 10. After that it doesn't stop fetching and the count keep on increasing. After fetching the actual 10 records it again fetches same 10 records and the count goes like on..Why it is fetching data repeatedly?

Thanks in advance

edit retag flag offensive close merge delete

2 Answers

Sort by ยป oldest newest most voted

answered 2018-06-21 09:20:44 -0500

metadaddy gravatar image

It sounds like the JDBC Query Consumer origin is in Full mode, and is configured to repeat the query according to the configured Query Interval. It's doing exactly what it's supposed to do with this configuration. This is intended for cases where you need to overwrite data in a destination periodically.

If you want the pipeline to stop once the query is complete, you need to add a Pipeline Finisher executor, triggered by the origin's no-more-data event. This case study shows how.

edit flag offensive delete link more



It worked. Thank you @metadaddy

Shruthi gravatar imageShruthi ( 2018-06-22 00:23:22 -0500 )edit

answered 2019-07-10 15:47:25 -0500

DDgreat gravatar image

updated 2019-07-10 15:56:39 -0500

metadaddy gravatar image

Stream set is for continuous streaming the records from source. I believe you want to use it as a batch mode. In your JDBCQueryConsumer origin if you do not have delta processing logic or requirement then please remove any value or column defined as Offset column. Enable the Produce events check box from general tab. And second event output link from JDBCQueryConsumer you can connect to Pipeline finisher executer stage. And in General tab Pre-condition of Pipeline finisher executer stage define this expression : ${record:eventType() == 'no-more-data'}. What it does is after executing your query output (all source records) it gracefully deactivates the pipeline when it encounters no more data from source.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-06-21 07:44:57 -0500

Seen: 912 times

Last updated: Jul 10 '19