Ask Your Question
2

Pipeline Finisher not working with JDBC Query Consumer

asked 2017-11-03 09:25:44 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I have a simple pipeline which uses a JDBC query consumer with Offset value. I have a pipeline finisher with a precondition of ${record:eventType() == 'no-more-data'}. If the jdbc query consumer picks up source data then the pipeline finisher works properly and ends the pipeline after the data is processed. However, if the pipeline is started and the jdbc query consumer does not pick up any new data, the pipeline never finishes and continues to send 1 record to the pipeline finisher at each query interval. I changed the 'On Record Error' option to 'Send to Error' within the pipeline finisher to see the error. Now when I run the pipeline and there is no new data being picked up from jdbc query conusmer, it generates the following error in the pipeline finisher stage:

COMMON_0001 - Stage precondition: CONTAINER_0051 - Unsatisfied precondition(s) '${record:eventType() == 'no-more-data'}'

Any ideas?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
1

answered 2017-11-04 09:55:12 -0500

bob gravatar image

There was an outstanding issue with the Jdbc Query Origin when running in incremental mode -- SDC-7603. This issue causes the problem you're seeing.

As you identified, the current behavior requires at least one record to be processed, then the "no-more-data" event will be trigged when no more records are available. The new behavior will be to trigger "no-more-data" event even if no records were processed by the query.

In non-incremental mode, this was not a problem.

SDC-7603 is resolved and is available in the 3.0.0.0-SNAPSHOT code line, and will be back ported to the latest builds of 2.7.2.1.

edit flag offensive delete link more

Comments

Thanks Bob. Do you know when the fix will be back ported to 2.7.2.1?

jshivers gravatar imagejshivers ( 2017-11-06 08:42:35 -0500 )edit
1

hi jshivers. The back port is available now in http://nightly.streamsets.com/2.7/170/ Please be aware that nightly builds are not appropriate for production; you should run this only in a development environment. thanks.

bob gravatar imagebob ( 2017-11-07 15:01:30 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-11-03 09:25:44 -0500

Seen: 115 times

Last updated: Nov 04 '17