Ask Your Question
1

Does the MapR-FS destination generate an event when it recovers a file?

asked 2017-09-07 01:30:42 -0500

mstang gravatar image

updated 2017-09-07 10:51:38 -0500

metadaddy gravatar image

When a file is closed by the MapR-FS destination a "file-closed" event can be generated. Is this same event (or a different event) supposed to be generated when a file is recovered after a unexpected shutdown?

I'm using the events to move the files with a shell executor after they close, but it doesn't seem to pick up the files which are recovered/renamed after the pipeline restarts.

edit retag flag offensive close merge delete

Comments

Would you mind sharing your pipeline so that we can reproduce it on our end?

rupal gravatar imagerupal ( 2017-09-07 16:27:32 -0500 )edit

Sure Rupal, I may need to edit this question. I thought that the event generation wasn't occurring. What actually seems to happen is that the file switches from _tmp_sdc* to sdc* when it recovers and then it continues to write to the sdc* file (I think it may ignores the idle timeout after this).

mstang gravatar imagemstang ( 2017-09-08 13:30:07 -0500 )edit

I did not see my files closing for idle timeout after recovery, they seemed to keep growing. If I manually stop the pipeline after that it will trigger the event and rename/move my file with the executor.

mstang gravatar imagemstang ( 2017-09-08 13:33:08 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
1

answered 2018-02-05 15:19:36 -0500

Jisun gravatar image

the "file-closed" is generated when a file is recovered. But, the events will be cached and then generated by the end of the batch https://github.com/streamsets/datacol...

I suspect even after the file is recovered if there was no data. The events might be cached and waiting for the data.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2017-09-07 00:16:55 -0500

Seen: 56 times

Last updated: Feb 05