Hadoop FS destination can NOT write data files

casel.chen
197 ●13 ●26 ●33

I'm trying Replicating Relational Databases with StreamSets Data Collector, the pipeline run quietly without errors but Hadoop FS destination only create database and table folder in HDFS while NO data files generated in the table directory, why?

I'm sure there are many records from MySQL table. And checked Hive Metastore Destination, the hive table created successfully.

image description

1 Answer

Sort by » oldest newest most voted

casel.chen
197 ●13 ●26 ●33

I set dfs.permission = false and chmod w+r for /user hdfs folder

Btw, how to extract mysql datetime field to yyyyMMdd and set it as partition field? Thanks!

1 follower

Asked: 2017-09-26 05:26:48 -0500

Seen: 55 times

Last updated: Sep 26 '17