Ask Your Question
1

Produce event is not generating in Hadoop FS to MapReduce (Avro to parquet conversion)

asked 2017-09-25 02:09:01 -0600

saikiran gravatar image

updated 2017-09-25 12:27:35 -0600

metadaddy gravatar image

I am trying to implement Parquet case study. I am able to generate avro file in (Paquet/.avro). But event is not generating to convert avro to parquet. Please help me to solve this issue

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted
0

answered 2017-09-25 12:33:25 -0600

metadaddy gravatar image

The most likely issue is that the pipeline is waiting to close the file in Hadoop. By default, the Hadoop FS destination waits until the pipeline has been idle for 1 hour before closing the file, which will trigger the event. You can reduce this time, the maximum number of records/file, or the maximum file size, to make the event fire sooner - see Timeout to Close Idle Files in the documentation.

Note that fewer, larger files tend to be preferable in a production Hadoop environment rather than more, smaller files.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-09-25 02:09:01 -0600

Seen: 46 times

Last updated: Sep 25 '17