Is there a way to trigger the start of another batch cluster pipeline after another finishes?
Like the question says, would like to trigger a pipeline after another finishes. I have an initial data ingestion pipeline that uses batch cluster mode to move data from one MapR-FS location to another (this gets started periodically using a script that uses the StreamSets CLI process). I then want to move it to another location from the previous pipeline's destination. Trying to avoid using the YARN streaming mode if possible, since I'm assuming that batch processing requires less resources over time and is faster. Is there a way to do this?
add a comment
