Ask Your Question
0

ERROR DataTransformerLauncher - Unable to detect Spark version by running command

asked 2020-03-30 15:18:28 -0500

J3 gravatar image

Attempting to run the simplest of pipeline and everytime i attempt to start I get this error:

ERROR DataTransformerLauncher - Unable to detect Spark version by running command /opt/streamsets/spark/bin/spark-submit --version

Why?

When I SSH on to the AMI instance, and I run the same command it works:

[ec2-user@ip-xxx-xxx-xxx-xxx transformer]$ /opt/streamsets/spark/bin/spark-submit --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.5
      /_/

Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_242
Branch HEAD
Compiled by user centos on 2020-02-02T20:01:52Z
Revision cee4ecbb16917fa85f02c635925e2687400aa56b
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.
[ec2-user@ip-xxx-xxx-xxx-xxx transformer]$

What configuration setting am I missing?

edit retag flag offensive close merge delete

Comments

I ended up using the latest docker image (3.13.0-SNAPSHOT) instead of the AMI image, and that worked for me.

J3 gravatar imageJ3 ( 2020-03-31 17:13:25 -0500 )edit

1 Answer

Sort by ยป oldest newest most voted
0

answered 2020-03-31 17:12:56 -0500

J3 gravatar image

I ended up using the latest docker image (3.13.0-SNAPSHOT) instead of the AMI image, and that worked for me.

edit flag offensive delete link more
Login/Signup to Answer

Question Tools

1 follower

Stats

Asked: 2020-03-30 15:18:28 -0500

Seen: 44 times

Last updated: Mar 31