Ask Your Question

Does the MapR-FS destination generate an event when it recovers a file?

asked 2017-09-07 01:30:42 -0500

mstang gravatar image

updated 2017-09-07 10:51:38 -0500

metadaddy gravatar image

When a file is closed by the MapR-FS destination a "file-closed" event can be generated. Is this same event (or a different event) supposed to be generated when a file is recovered after a unexpected shutdown?

I'm using the events to move the files with a shell executor after they close, but it doesn't seem to pick up the files which are recovered/renamed after the pipeline restarts.

edit retag flag offensive close merge delete


Would you mind sharing your pipeline so that we can reproduce it on our end?

rupal gravatar imagerupal ( 2017-09-07 16:27:32 -0500 )edit

Sure Rupal, I may need to edit this question. I thought that the event generation wasn't occurring. What actually seems to happen is that the file switches from _tmp_sdc* to sdc* when it recovers and then it continues to write to the sdc* file (I think it may ignores the idle timeout after this).

mstang gravatar imagemstang ( 2017-09-08 13:30:07 -0500 )edit

I did not see my files closing for idle timeout after recovery, they seemed to keep growing. If I manually stop the pipeline after that it will trigger the event and rename/move my file with the executor.

mstang gravatar imagemstang ( 2017-09-08 13:33:08 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted

answered 2018-02-05 15:19:36 -0500

Jisun gravatar image

the "file-closed" is generated when a file is recovered. But, the events will be cached and then generated by the end of the batch

I suspect even after the file is recovered if there was no data. The events might be cached and waiting for the data.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2017-09-07 00:16:55 -0500

Seen: 80 times

Last updated: Feb 05 '18