ERROR DataTransformerLauncher - Unable to detect Spark version by running command
Attempting to run the simplest of pipeline and everytime i attempt to start I get this error:
ERROR DataTransformerLauncher - Unable to detect Spark version by running command /opt/streamsets/spark/bin/spark-submit --version
Why?
When I SSH on to the AMI instance, and I run the same command it works:
[ec2-user@ip-xxx-xxx-xxx-xxx transformer]$ /opt/streamsets/spark/bin/spark-submit --version
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.5
/_/
Using Scala version 2.11.12, OpenJDK 64-Bit Server VM, 1.8.0_242
Branch HEAD
Compiled by user centos on 2020-02-02T20:01:52Z
Revision cee4ecbb16917fa85f02c635925e2687400aa56b
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.
[ec2-user@ip-xxx-xxx-xxx-xxx transformer]$
What configuration setting am I missing?
I ended up using the latest docker image (3.13.0-SNAPSHOT) instead of the AMI image, and that worked for me.