How to store status in ETL pipeline?

asked 2018-09-20 08:25:32 -0600

Sergey gravatar image

I can't find good example of ETL pipeline.

There are two databases: one with statuses, second data storage with data. ETL steps: 1. Get entries from statuses table with status (state/flag) 'UNPROCESSED' (JDBC Query Consumer) 2. Execute some queries against data storage database to reprocess data (JDBC Query) 3. Mark entry as 'FINISHED''

There is no anything like 'JDBC Query processor'. How should I update entry status as soon as some query had finished processing data in the data storage database?

Sincerely, Sergey.

edit retag flag offensive close merge delete

Comments

What exactly are you doing in step 2 -- JDBC Query?

iamontheinet gravatar imageiamontheinet ( 2018-09-20 17:40:18 -0600 )edit

Have you looked at the JDBC Lookup processor? https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Processors/JDBCLookup.html#concept_ysc_ccy_hw

metadaddy gravatar imagemetadaddy ( 2018-09-21 20:46:10 -0600 )edit
add a comment