Ask Your Question

Revision history [back]

Use the Hive Metadata processor with the Hadoop FS and Hive Metastore processors as detailed in Drift Synchronization Solution for Hive. The Hadoop FS destination can be tuned to create large files, so you should not have any issues, and you can trigger a MapReduce job to convert the Avro output to Parquet.

Use the Hive Metadata processor with the Hadoop FS and Hive Metastore processors as detailed in Drift Synchronization Solution for Hive. The Hadoop FS destination can be tuned to create large files, so you should not have any issues, and you can trigger a MapReduce job to convert the Avro output to Parquet.

See the section Timeout to Close Idle Files for details on how to tune the Hadoop FS destination to create larger files. Increasing one or more of 'Idle Timeout', 'Max Records in a File' and 'Max File Size' should result in larger files.