About Spark executor Argument

asked 2020-03-25 18:49:56 -0500

HI.

I'm running a SDC 3.13.0 in a local. Also I installed spark in a local. My pipeline is simple, 'Directory -> Spark executor'. I use 'spark executor' what was installed from 'CDH Spark 2.1.0 Release 1' library.

I just want to do a test for driving my spark stuff on a SDC. In other words, I try to embody some work what is complemented by a local terminal command 'bin/spark-submit --master spark://bdy:7077 application.py' on a SDC pipeline.

I set configurations of spark executor as below.

Additional Spark Arguments : 1 [ --master spark://BDY:7077 ] ( A figure '[ ]' means input box. )

And I got a error message as below.

Error: Unrecongnized option: --master spark://BDY:7077

My question is this : What expression is right in here?.

+) Before pipeline is started, i start a master and works in local.

Thanks.

edit retag flag offensive close merge delete