I'm trying Replicating Relational Databases with StreamSets Data Collector, the pipeline run quietly without errors but Hadoop FS destination only create database and table folder in HDFS while NO data files generated in the table directory, why?
I'm sure there are many records from MySQL table. And checked Hive Metastore Destination, the hive table created successfully.
Hello, I am facing similar issue as you mentioned in your post. Our Pipeline was working properly a couple of days ago and suddenly it stopped to work. We set full permition of the hdfs folder of hive (as showing on command: show create table [database].[table] in LOCATION property shows: /user/hive/warehouse/[database]/[table] ) Our job completed without any error (even on Preview than on Run options modes), and when we go to HDFS via command line issue command (hdfs dfs -ls /user/hive/warehouse/[database]/) we can see the folder was successfully created but no file (AVRO format) was added to this folder, and on HUE when we select data (SELECT * FROM [database].[table] returns empty...
Have you resolved this? Any idea/direction can you can help us figured out?
We have found a solution about this issue, on the Hadoop FS (Destination) on Configuration > Output Files > Directory in Header flag was needed to be checked (as it is not checked by default) and it was not adding the AVRO files once creating a temporary table in HDFS directory /user/hive/warehouse.
I set dfs.permission = false and chmod w+r for /user hdfs folder
Btw, how to extract mysql datetime field to yyyyMMdd and set it as partition field? Thanks!
Asked: 2017-09-26 05:26:48 -0500
Seen: 300 times
Last updated: Sep 26 '17
