Ask Your Question

Produce event is not generating in Hadoop FS to MapReduce (Avro to parquet conversion)

asked 2017-09-25 02:09:01 -0500

saikiran gravatar image

updated 2017-09-25 12:27:35 -0500

metadaddy gravatar image

I am trying to implement Parquet case study. I am able to generate avro file in (Paquet/.avro). But event is not generating to convert avro to parquet. Please help me to solve this issue

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2017-09-25 12:33:25 -0500

metadaddy gravatar image

The most likely issue is that the pipeline is waiting to close the file in Hadoop. By default, the Hadoop FS destination waits until the pipeline has been idle for 1 hour before closing the file, which will trigger the event. You can reduce this time, the maximum number of records/file, or the maximum file size, to make the event fire sooner - see Timeout to Close Idle Files in the documentation.

Note that fewer, larger files tend to be preferable in a production Hadoop environment rather than more, smaller files.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-09-25 02:09:01 -0500

Seen: 222 times

Last updated: Sep 25 '17