EOFException running Streamsets Kafka Stage Library in Yarn Streaming mode

asked 2018-02-21 14:46:09 -0600

Aj262 gravatar image

I'm trying to run the Streamsets work on Yarn Streaming Cluster Mode for Kafka client using SSL connection. I tried using both Sparks 1.6 and 2.2 but getting the error below. Streamsets doesn't seem to have a Kafka stage library that supports Sparks 2 yet. At least that appears from the stage library jar name being used in the classpath - file:$PWD/spark-streaming-kafka_2.10-1.6.0-cdh5.12.0.jar. It works fine in the standalone mode.

"Error trying to invoke BootstrapClusterStreaming.main: org.apache.spark.SparkException: java.io.EOFException org.apache.spark.SparkException: java.io.EOFException"

Does anyone know if Streamsets support Yarn Streaming Cluster Mode for Kafka client using SSL connection? If yes, I think I might be missing some configuration. Any suggestions what I might be missing?

edit retag flag offensive close merge delete