events based pipeline execution

asked 2019-05-07 08:27:18 -0500

Jmbertoncelli gravatar image

Hello, I need to execute a pipeline only when a number of events generated by any number of other pipelines reach a specific trigger. I also need to know if Streamsets does have anything that could be used to perform a Kafka join between topics? To make a long story short I have to orchestrate pipelines execution following an event based architecture but I have some issues when it comes to execute a pipeline with predecessors dependencies... For instance data integration pipeline should start only when predecessors pipelines have completed their job...

Let me know if what I do ask make sense or not.

Thank you

JM

edit retag flag offensive close merge delete