ClassNotFoundException when running streamsets on yarn

asked 2018-06-26 00:45:52 -0600

casel.chen gravatar image

I tried to use sdc 3.0.3.0 running on yarn 2.7.3 (kafka -> mongo) but met ClassNotFoundException. Missed scala library? How to fix it? Thanks!

image description

2018-06-26 13:40:07,547 [user:] [pipeline:] [thread:Driver] INFO VerifiableProperties - Property zookeeper.connect is overridden to Tue Jun 26 13:40:07 CST 2018: Error trying to invoke BootstrapClusterStreaming.main: java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class java.lang.NoClassDefFoundError: scala/collection/GenTraversableOnce$class at kafka.utils.Pool.<init>(Pool.scala:26) at kafka.consumer.FetchRequestAndResponseStatsRegistry$.<init>(FetchRequestAndResponseStats.scala:61) at kafka.consumer.FetchRequestAndResponseStatsRegistry$.<clinit>(FetchRequestAndResponseStats.scala) at kafka.consumer.SimpleConsumer.<init>(SimpleConsumer.scala:44) at org.apache.spark.streaming.kafka.KafkaCluster.connect(KafkaCluster.scala:52) at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$org$apache$spark$streaming$kafka$KafkaCluster$$withBrokers$1.apply(KafkaCluster.scala:345) at org.apache.spark.streaming.kafka.KafkaCluster$$anonfun$org$apache$spark$streaming$kafka$KafkaCluster$$withBrokers$1.apply(KafkaCluster.scala:342) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at org.apache.spark.streaming.kafka.KafkaCluster.org$apache$spark$streaming$kafka$KafkaCluster$$withBrokers(KafkaCluster.scala:342) at org.apache.spark.streaming.kafka.KafkaCluster.getPartitionMetadata(KafkaCluster.scala:125) at org.apache.spark.streaming.kafka.KafkaCluster.getPartitions(KafkaCluster.scala:112) at org.apache.spark.streaming.kafka.KafkaUtils$.getFromOffsets(KafkaUtils.scala:211) at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:484) at org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:607) at org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala) at com.streamsets.pipeline.spark.SparkStreamingBinding$JavaStreamingContextFactoryImpl.create(SparkStreamingBinding.java:178) at com.streamsets.pipeline.spark.AbstractStreamingBinding.init(AbstractStreamingBinding.java:129) at com.streamsets.pipeline.BootstrapClusterStreaming.main(BootstrapClusterStreaming.java:36) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:635) Caused by: java.lang.ClassNotFoundException: scala.collection.GenTraversableOnce$class at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 24 more

edit retag flag offensive close merge delete