Ask Your Question
1

I am trying to run a pipeline which does CSV -> Hive, Streamsets pipeline is not picking up CSV file which has a Header but has zero records.

asked 2019-03-20 18:38:51 -0500

Goth gravatar image

There are some CSV Files with Zero records , But they have a header , I am trying to convert them into tables and deploy them to Hive as empty tables which has only schema but no records , Streamsets is not picking up those CSV files . Can anyone suggest a solution to this ? Thanks !

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2019-03-20 18:49:21 -0500

metadaddy gravatar image

Data Collector will only create the Hive tables when it processes an actual data record. CSV files with headers but no data result in no data flowing through the pipeline.

edit flag offensive delete link more

Comments

Thanks for the response , I think i should do them manually now . Is there any better approach you can suggest ?

Goth gravatar imageGoth ( 2019-03-20 18:50:33 -0500 )edit

Not really. Note that, by default, even if you put a single dummy row in each CSV file, all the columns in Hive would be strings, which you likely don't want. You would have to use Field Type Converter to convert strings to integers or whatever,

metadaddy gravatar imagemetadaddy ( 2019-03-20 18:54:07 -0500 )edit
Login/Signup to Answer

Question Tools

2 followers

Stats

Asked: 2019-03-20 18:38:51 -0500

Seen: 99 times

Last updated: Mar 20