How to configure insert overwrite CRUD Operation on target Hadoop FS hive table through StreamSet?

asked 2019-07-10 15:23:14 -0500

DDgreat gravatar image

I am using JDBCQueryConsumer as source to select data from Oracle and creating Dynamic partition based on incoming data column (string type) from source with the help of partition column name and partition value expression and ingesting data into target HDFS Hive table. Our Requirement: In the re-run scenario of the same pipeline / same input data, our use case is if same partition data comes again from source we need to totally overwrite the target partition data with the new one if partition already existing in the target hive table. Target HDFS Hive supports hive action insert overwrite table <table_name> partition(partition key column) select SQL. How can we achieve same action through StreamSet pipe line? Any quick help will be appreciated

edit retag flag offensive close merge delete

Comments

Can some one please advice?

DDgreat gravatar imageDDgreat ( 2019-07-11 12:28:40 -0500 )edit

Hi Dave, It might be best if you open a support ticket for this. They should be able to work through this with you in a timely manner. Cheers, Dash

iamontheinet gravatar imageiamontheinet ( 2019-07-12 12:41:50 -0500 )edit