An exception occurred during starting pipeline with spark

asked 2020-10-10 03:45:13 -0500

Chui gravatar image

Streamsets Transformer Version:StreamSets Transformer 3.14.0 ( built by ubuntu on 2020-06-15T21:21Z git (Docker)
Spark Version:2.4.7

image description image description

Because I don’t have permission to upload files, I can only use pictures to show the contents of pipeline config info. I clicked start button,waited a few seconds,throw some error message

 Pipeline Status: START_ERROR: Failed to start pipeline, check logs for error. The failure may be because the Transformer Spark application in the Spark cluster cannot reach Transformer at 'http://f83a2ce03255:19630'. If this is not the correct URL for Transformer, update the transformer.base.http.url property in the Transformer configuration file and restart Transformer.

image description

View logs, An exception was thrown

2020-10-10 08:38:31,136 [user:admin] [pipeline:Test_Spark/sdaae91884a4-58bb-455e-971e-fe326bbe89ca] [com.streamsets.datatransformer.dag.com.streamsets.datatransformer.dag.runner:] [thread:launcher-proc-4] [stage:] INFO sdaae91884a4-58bb-455e-971e-fe326bbe89ca_7 - Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Can only call getServletHandlers on a running MetricsSystem 2020-10-10 08:38:31,137 [user:admin] [pipeline:Test_Spark/sdaae91884a4-58bb-455e-971e-fe326bbe89ca] [com.streamsets.datatransformer.dag.com.streamsets.datatransformer.dag.runner:] [thread:launcher-proc-4] [stage:] INFO sdaae91884a4-58bb-455e-971e-fe326bbe89ca_7 - at scala.Predef$.require(Predef.scala:224) 2020-10-10 08:38:31,137 [user:admin] [pipeline:Test_Spark/sdaae91884a4-58bb-455e-971e-fe326bbe89ca] [com.streamsets.datatransformer.dag.com.streamsets.datatransformer.dag.runner:] [thread:launcher-proc-4] [stage:] INFO sdaae91884a4-58bb-455e-971e-fe326bbe89ca_7 - at org.apache.spark.metrics.MetricsSystem.getServletHandlers(MetricsSystem.scala:91) 2020-10-10 08:38:31,137 [user:admin] [pipeline:Test_Spark/sdaae91884a4-58bb-455e-971e-fe326bbe89ca] [com.streamsets.datatransformer.dag.com.streamsets.datatransformer.dag.runner:] [thread:launcher-proc-4] [stage:] INFO sdaae91884a4-58bb-455e-971e-fe326bbe89ca_7 - at org.apache.spark.SparkContext.<init>(SparkContext.scala:516) 2020-10-10 08:38:31,137 [user:admin] [pipeline:Test_Spark/sdaae91884a4-58bb-455e-971e-fe326bbe89ca] [com.streamsets.datatransformer.dag.com.streamsets.datatransformer.dag.runner:] [thread:launcher-proc-4] [stage:] INFO sdaae91884a4-58bb-455e-971e-fe326bbe89ca_7 - at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2520) 2020-10-10 08:38:31,137 [user:admin] [pipeline:Test_Spark/sdaae91884a4-58bb-455e-971e-fe326bbe89ca] [com.streamsets.datatransformer.dag.com.streamsets.datatransformer.dag.runner:] [thread:launcher-proc-4] [stage:] INFO sdaae91884a4-58bb-455e-971e-fe326bbe89ca_7 - at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935) 2020-10-10 08:38:31,137 [user:admin] [pipeline:Test_Spark/sdaae91884a4-58bb-455e-971e-fe326bbe89ca] [com.streamsets.datatransformer.dag.com.streamsets.datatransformer.dag.runner:] [thread:launcher-proc-4] [stage:] INFO sdaae91884a4-58bb-455e-971e-fe326bbe89ca_7 - at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:926) 2020-10-10 08:38:31,137 [user:admin] [pipeline:Test_Spark/sdaae91884a4-58bb-455e-971e-fe326bbe89ca] [com.streamsets.datatransformer.dag.com.streamsets.datatransformer.dag.runner:] [thread:launcher-proc-4] [stage:] INFO sdaae91884a4-58bb-455e-971e-fe326bbe89ca_7 - at scala.Option.getOrElse(Option.scala:121) 2020-10-10 08:38:31,137 [user:admin] [pipeline:Test_Spark/sdaae91884a4-58bb-455e-971e-fe326bbe89ca] [com.streamsets.datatransformer.dag.com.streamsets.datatransformer.dag.runner:] [thread:launcher-proc-4] [stage:] INFO sdaae91884a4-58bb-455e-971e-fe326bbe89ca_7 - at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:926) 2020-10-10 08:38:31,137 [user:admin] [pipeline:Test_Spark/sdaae91884a4-58bb-455e-971e-fe326bbe89ca] [com.streamsets.datatransformer.dag.com.streamsets.datatransformer.dag.runner:] [thread:launcher-proc-4] [stage:] INFO sdaae91884a4-58bb-455e-971e-fe326bbe89ca_7 - at com.streamsets.transformer.BootstrapSparkMain.main(BootstrapSparkMain.java:54) 2020-10-10 08:38:31,137 [user:admin] [pipeline:Test_Spark/sdaae91884a4-58bb-455e-971e-fe326bbe89ca] [com.streamsets.datatransformer.dag.com.streamsets.datatransformer.dag.runner:] [thread:launcher-proc-4] [stage:] INFO sdaae91884a4-58bb-455e-971e-fe326bbe89ca_7 - at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 2020-10-10 08:38:31,137 [user:admin] [pipeline ...

(more)
edit retag flag offensive close merge delete

Comments

Issues like this can be very difficult to diagnose on this forum. Please open a support request http://streamsets.com/support <-- here to get assistance.

jeff gravatar imagejeff ( 2020-10-12 16:57:48 -0500 )edit