Ask Your Question

Revision history [back]

The most likely issue is that the pipeline is waiting to close the file in Hadoop. By default, the Hadoop FS destination waits until the pipeline has been idle for 1 hour before closing the file, which will trigger the event. You can reduce this time, the maximum number of records/file, or the maximum file size, to make the event fire sooner - see Timeout to Close Idle Files in the documentation.

Note that fewer, larger files tend to be preferable in a production Hadoop environment rather than more, smaller files.