Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

How to configure insert overwrite CRUD Operation on target Hadoop FS hive table through StreamSet?

I am using JDBCQueryConsumer as source to select data from Oracle and creating Dynamic partition based on incoming data column (string type) from source with the help of partition column name and partition value expression and ingesting data into target HDFS Hive table. Our Requirement: In the re-run scenario of the same pipeline / same input data, our use case is if same partition data comes again from source we need to totally overwrite the target partition data with the new one if partition already existing in the target hive table. Target HDFS Hive supports hive action insert overwrite table <table_name> partition(partition key column) select SQL. How can we achieve same action through StreamSet pipe line? Any quick help will be appreciated