Ask Your Question

Setup HA for StreamSet Data Collectors

asked 2018-05-24 05:27:48 -0500

Pradip gravatar image

updated 2018-05-24 10:30:23 -0500

metadaddy gravatar image

I have setup pipeline MySQLBinary Log -> Kafka Producer in standalone execution mode and working pretty well as expected. Next - I wanted to move this pipeline to production and have proper HA setup in place. Question - How do we enable HA like setup when node running data collector crashes for some reason and there is second one(standby) taking over automatically?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-05-24 10:32:50 -0500

metadaddy gravatar image

You should look at StreamSets Control Hub -

Control Hub lets you create jobs, which assign pipelines to Data Collector instances. When Data Collector instance 'A' goes down, Control Hub will restart the job on another instance, 'B', where it can pick up from the last offset that instance 'A' processed.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-05-24 05:27:48 -0500

Seen: 563 times

Last updated: May 24 '18