Ask Your Question

Is the Origin Event trigger after all data sent to destination or after Data is processed by Origin

asked 2018-05-28 22:31:46 -0600

Trans gravatar image

suppose if I have Directory as origin which produces events such as new-file / processed-file and destination as MongoDB, now Is the event triggers after data is processed by Directory or after all the processed data saved to MongoDB.

I have seen that SDC is triggering event as soon as Directory reads all data.

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-05-29 03:55:59 -0600

Mufy gravatar image

Hello Trans. Not sure if you have gone over this. That quickly summarizes how event triggers are handled by different stages as they vary from stage to stage.

Just to make sure you have seen the difference, there is the Pipeline Event Generation and then there is the Stage Event Generation.

What you are observing is the latter where the event is generated as soon as a criteria defined for that particular stage is met.

Hope that helps.

edit flag offensive delete link more


I have already seen the pipeline and stage event generation, but in my case directory is origin and MongoDB is Destination. As per the event generations doc couldn't find event for mongodb Destination.

Trans gravatar imageTrans ( 2018-05-29 08:40:06 -0600 )edit

MongoDB D currently does not spew out any event at this point in time. That said, we'll be more than interested to know if you have any specific events in mind that you'd like to see generated from this stage.

Mufy gravatar imageMufy ( 2018-05-30 01:13:27 -0600 )edit

I am looking for an event which triggers as soon as all the data is inserted to Mongo DB and there is no more data left to insert

Trans gravatar imageTrans ( 2018-05-30 05:11:16 -0600 )edit

What you are looking for is an Origin's event. Here, events are triggered after a batch completion. So, if the origin is seeing the last batch and it sends the batch to destination, it will trigger the no-more-data event, but the event sits in an event lane until the batch completes ....

Mufy gravatar imageMufy ( 2018-06-04 00:05:14 -0600 )edit

... and then triggers the event. Please note that this works with normal delivery guarantees though.

Mufy gravatar imageMufy ( 2018-06-04 00:05:20 -0600 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-05-28 22:31:46 -0600

Seen: 2,008 times

Last updated: May 29 '18