Ask Your Question

JDBC Consumer Query - run once?

asked 2019-05-09 09:40:52 -0500

Carol gravatar image

I have created a pipeline that reads a Salesforce data extension (successfully) and then writes records to SQL Server database (successfully). I can either read ALL the records from salesforce (which i do not need) and then filter out on write to the database table, or filter the read from DE and then write all records fetched. Either will work.

The piece of information that I need to use to filter out records (at either end) is to first check the MAX value of a column in the database. I created a JDBC Consumer Query and it indeed fetches the information fine, but it does so for each record that moves through the pipeline. I need to only fetch the value once and use it within the pipeline.

Ideas or suggestions on a better way to do this ?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2019-05-09 14:11:09 -0500

metadaddy gravatar image

You could look at using one of the Dev origins as a 'timer' and using the JDBC Lookup processor instead of the JDBC Consumer origin. The benefit here is that the Lookup processor can cache the data, and the cache is configurable to your requirements.

edit flag offensive delete link more


Thank you for the suggestion. The JDBC Lookup Processor would also execute for each/all records flowing through the pipeline, yes ? But if i enable the cache, it just saves the re-executing the lookup?

Carol gravatar imageCarol ( 2019-05-09 14:15:15 -0500 )edit

Exactly - the records would flow through the Lookup processor, but there would be no actual JDBC Lookup after the initial one.

metadaddy gravatar imagemetadaddy ( 2019-05-09 14:25:43 -0500 )edit

thanks so much.

Carol gravatar imageCarol ( 2019-05-09 15:12:41 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2019-05-09 09:40:52 -0500

Seen: 71 times

Last updated: May 09