Ask Your Question

Hadoop FS destination can NOT write data files

asked 2017-09-26 05:26:48 -0600

casel.chen gravatar image

I'm trying Replicating Relational Databases with StreamSets Data Collector, the pipeline run quietly without errors but Hadoop FS destination only create database and table folder in HDFS while NO data files generated in the table directory, why?

I'm sure there are many records from MySQL table. And checked Hive Metastore Destination, the hive table created successfully.

image description

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2017-09-26 05:28:00 -0600

casel.chen gravatar image

updated 2017-09-26 05:31:38 -0600

I set dfs.permission = false and chmod w+r for /user hdfs folder

Btw, how to extract mysql datetime field to yyyyMMdd and set it as partition field? Thanks!

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-09-26 05:26:48 -0600

Seen: 164 times

Last updated: Sep 26 '17