Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

JDBC Consumer Query - run once?

I have created a pipeline that reads a Salesforce data extension (successfully) and then writes records to SQL Server database (successfully). I can either read ALL the records from salesforce (which i do not need) and then filter out on write to the database table, or filter the read from DE and then write all records fetched. Either will work.

The piece of information that I need to use to filter out records (at either end) is to first check the MAX value of a column in the database. I created a JDBC Consumer Query and it indeed fetches the information fine, but it does so for each record that moves through the pipeline. I need to only fetch the value once and use it within the pipeline.

Ideas or suggestions on a better way to do this ?