Ask Your Question

How can I run StreamSets Data Collector cluster for Kafka origin pipelines?

asked 2018-06-13 09:31:30 -0500

casel.chen gravatar image

updated 2018-06-13 09:53:08 -0500

metadaddy gravatar image

Does StreamSets Data Collector support cluster deployment for Kafka origin pipelines?

edit retag flag offensive close merge delete

1 Answer

Sort by ยป oldest newest most voted

answered 2018-06-13 09:45:49 -0500

metadaddy gravatar image

Yes - in fact, Kafka is the only origin supported for Cluster Streaming pipelines.

edit flag offensive delete link more


Thanks for quick response. Is there any tutorial or screen shot guide to follow? Thanks.

casel.chen gravatar imagecasel.chen ( 2018-06-13 09:53:53 -0500 )edit

I am using Apache Kafka 1.0.0 and hadoop/yarn 2.7.3 without CDH or HDP, but it reports VALIDATION_0071 - Stage 'Kafka Consumer' from 'Apache Kafka 1.0.0' library does not support 'Cluster Yarn Streaming' execution mode. Can't the stage library be used in cluster yarn streaming mode?

casel.chen gravatar imagecasel.chen ( 2018-06-14 05:30:13 -0500 )edit
Login/Signup to Answer

Question Tools

1 follower


Asked: 2018-06-13 09:31:30 -0500

Seen: 443 times

Last updated: Jun 13 '18