Ask Your Question

Revision history [back]

directory origin widget not picking up files

Hi, I have created a simple flow that starts with a directory origin. It is supposed to pickup the files in a directory and then publish the records in the file to a Kafka publisher step. A second workflow starts with a Kafka consumer origin which reads all the records from the queue and sends it to a hdfs widget which then writes the files to a hdfs directory.

I am able to start both the flows and they do work end to end as designed. However with the first flow, I have to stop and start it about 3-4 times before it is able to successfully pick up the files from the directory. Want to know if someone else has encountered this before.