Create a new unique row for loading into JDBC Table

asked 2017-12-11 15:14:26 -0500

jay1988 gravatar image

updated 2017-12-13 12:00:12 -0500

metadaddy gravatar image

I have 30 JDBC tables to be consumed from SQL Server. A JDBC Multitable consumer is being used in my pipeline to source these 30 tables and data is loaded into 30 HDFS Locations.There is another Teradata LOG Table which should hold the Load start and Load end dtm with a Unique Load ID.

I need to first load the Teradata LOG Table with unique load id and STRT dtm and then proceed with data ingestion of 30 Tables. Loading into Teradata Table and the 30 HDFS locations should happen in a single pipeline.

A Jython evaluator is used to create the value for Unique Load ID. My issue is that different id's are getting created for each row in the Tables since I'm using Multitable consumer. I need only 1 Row to be inserted into Teradata LOG table and then file ingestion for the 30 Tables should happen. These 2 process has to be done in a single pipeline.

edit retag flag offensive close merge delete


To clarify: when the pipeline starts, you need to write a single row to the Teradata LOG table? What does this row need to include, apart from the load id. What is a STRT dtm?

metadaddy gravatar imagemetadaddy ( 2017-12-13 11:59:16 -0500 )edit

The row needs to include the Load ID and start dtm. Start DTM here refer to current timestamp. The same pipeline has to write the Unique row to the Log table and should be able to Pull the data from SQL Server for 30 tables and write into HDFS.

jay1988 gravatar imagejay1988 ( 2017-12-13 12:17:12 -0500 )edit