Error:Hadoop FS destination creates empty file.

asked 2018-01-11 22:40:28 -0600

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

I am trying to fetch table data using JDBC Query Consumer and dump the fetched data to HDFS using streamsets on different servers.But, the pipeline creates an empty file in the destination (HDFS ), without any data.Why is it so? What might be the solution for this? Please, someone suggest me the solution.

Thanks

edit retag flag offensive close merge delete

Comments

Is this because of permission issue? If yes ,my doubt is how a file got created in the specified destination?

Maithri gravatar imageMaithri ( 2018-01-13 08:23:34 -0600 )edit

Can you add the output files configuration in hdfs destination and show a example of how the empty file looks like ?

Roh gravatar imageRoh ( 2018-01-16 14:50:27 -0600 )edit

Configuration:Hadoop FS General: Stage library:5.10.1 Hadoop FS : Hadoop FS URI:hdfs://x.y.z:8020/streamsets Hadoop FS Configuration Directory:/home/streamsets/resources Output files: Directory Template:/streamsets/xyz/ Data Format : Data Format :JSON

Maithri gravatar imageMaithri ( 2018-01-16 22:51:08 -0600 )edit

Only these fields of configurations are edited,and the rest is default.And in the path /streamsets/xyz/sdc.... An sdc empty file got created .

Maithri gravatar imageMaithri ( 2018-01-16 22:52:05 -0600 )edit