Can a Topology be called as a REST microservice?

asked 2020-07-21 14:36:53 -0500

Colin gravatar image

Hello,

I am able to create Data Collector REST microservice pipelines, as introduced in this tutorial: https://streamsets.com/blog/create-mi...

I also know how to connect Data Collector pipelines to other Data Collector or Transformer pipelines, forming a Topology, via Kafka.

However I am struggling to see how a Topology consisting of more than one pipeline can be called as a whole, via REST. In particular I would like to be able to call a REST service that combines processing in Data Collector and Transformer.

If anyone could confirm this is possible and share an example, it would be greatly appreciated.

edit retag flag offensive close merge delete

Comments

Are you simply trying to orchestrate execution of SDC and Transformer pipelines? If so, have you looked into this -- https://streamsets.com/documentation/datacollector/latest/help/datacollector/UserGuide/Orchestration_Pipelines/OrchestrationPipelines_Title.html?

iamontheinet gravatar imageiamontheinet ( 2020-07-21 16:23:49 -0500 )edit

Hi - I see there that Orchestration Pipelines can be used to trigger other pipelines. I can't see how those can be used to pass the output of the combination of pipelines in a response back to the original REST origin, however.

Colin gravatar imageColin ( 2020-07-22 04:57:21 -0500 )edit