Ask Your Question
1

jdbc.tables in JDBC Multitable Consumer

asked 2019-01-11 04:13:16 -0500

Deepak Rajendra gravatar image

Hi,

There are two tables in the Netezza database that I want to write the two tables as individual CSV files to hdfs. Below are the steps I follow (described in detail) but I still just see a single CSV file created with records from both tables in this single file and I do not see the record header attribute jdbc.tables written to the file. How should I use the jdbc.tables in the local FS destination to stream records to different files.

1) JDBC Multitable Consumer:

  • Set the JDBC Connection String
  • Per Batch Strategy = Process All available rows from the table
  • Set my Schema name under Table config
  • Set Table Name Pattern as ABC%
  • Enable Non-Incremental Load

2) Expression Evaluator:

  • Under Expressions, set Header Attribute as <jdbc.tables>
  • Set Header Attribute Expression as ${record:attribute('jdbc.tables')}

3) Local FS:

  • Set File Type = Text Files
  • Set Files Prefix = sdc-${sdc:id()}
  • Set Directory Template = /mapr/dsrx/data/ods/ph_contract/Test1/${YYYY()}-${MM()}-${DD()}-${hh()}
  • Set Data Format = Delimited CSV (ignore empty lines)
  • Set Header line = With Header line
  • Enable Replace New Line Characters

Thanks a lot.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
2

answered 2019-01-11 04:42:54 -0500

Deepak Rajendra gravatar image

updated 2019-01-13 22:17:46 -0500

Maithri gravatar image

I was able to figure this out. Here is the solution if someone else is interested.

  1. Removed expression evaluator as JDBC Multitable Consumer will create jdbc.tables record header attribute.
  2. Used ${record:attribute('jdbc.tables')} in directory template in Local FS.
edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2019-01-11 04:13:16 -0500

Seen: 115 times

Last updated: Jan 13