How to start multiple pipelines using parameter

asked 2019-06-01 20:04:23 -0500

Barry gravatar image

Hi. I am new to StreamSets so have only done a few basic piplelines. One of these piplelines extracts data from a database using JDBC query with a runtime parameter for the partition to be extracted, and outputs this to a Local FS destination for another purpose. This is run by partition due to the millions of records to be extracted taking too long to run the query without specifying the partition.

Rather than starting pipelines manually and specifying the parameter manually, I would like to automate this. Preferably scaling this to 3 or 4 pipelines running concurrently.

One option I have investigated is providing a file (or files) with the partition key specified as a record and using a 'control pipeline' to read the file, and starting the 'worker pipleline' via an HTTP Client API call. In order to check that the 'worker pipeline' is ready, I am checking the pipeline status has a status of 'FINISHED' using another HTTP Client processor. If the pipeline is not ready, I have added a 60 second delay before attempting to kill the 'control pipeline' with a RUN_ERROR and have it restart at the same record in the file (as I cannot see a way of looping back from the selector).

It feels like I am overcomplicating what needs to be done, so if there is a better way to achieve this, please let me know. If on the other hand this is the right approach, I need help with

  1. Getting the pipeline status exposed. I have been able to get the status of the pipeline using the restAPI page in StreamSets, but have had trouble translating that to a HTTP Client processor.

  2. Starting the pipeline. I have been able to start a pipeline using the restAPI page in StreamSets, but have had trouble translating that to a HTTP Client processor.

  3. Raising the error and making the pipeline restart. The pipeline hasn't even got this far, so I am not sure whether what I have coded will work.

Appreciate any guidance Thanks

edit retag flag offensive close merge delete