hadoop - Spark: Unknown/unsupported param error when setting conf.yarn.jar -

i have little application runs fine on spark cluster based on yarn when commit spark-submit this:

~/spark-1.4.0-bin-hadoop2.4$ bin/spark-submit --class myclass --master yarn-cluster --queue testing myapp.jar hdfs://nameservice1/user/xxx/readme.md_count 

however, avoid uploading spark-assembly.jar file each time, set spark.yarn.jar configuration parameter:

~/spark-1.4.0-bin-hadoop2.4$ bin/spark-submit --class myclass --master yarn-cluster --queue testing --conf "spark.yarn.jar=hdfs://nameservice1/user/spark/share/lib/spark-assembly.jar" myapp.jar hdfs://nameservice1/user/xxx/readme.md_count 

this seems fine @ first:

15/07/08 13:57:17 warn util.nativecodeloader: unable load native-hadoop library platform... using builtin-java classes applicable 15/07/08 13:57:18 info yarn.client: requesting new application cluster 24 nodemanagers 15/07/08 13:57:18 info yarn.client: verifying our application has not requested more maximum memory capability of cluster (8192 mb per container) 15/07/08 13:57:18 info yarn.client: allocate container, 896 mb memory including 384 mb overhead 15/07/08 13:57:18 info yarn.client: setting container launch context our 15/07/08 13:57:18 info yarn.client: preparing resources our container 15/07/08 13:57:18 info yarn.client: source , destination file systems same. not copying hdfs://nameservice1/user/spark/share/lib/spark-assembly.jar [...] 

however, fails eventually:

15/07/08 13:57:18 info yarn.client: submitting application 670 resourcemanager 15/07/08 13:57:18 info impl.yarnclientimpl: submitted application application_1434986503384_0670 15/07/08 13:57:19 info yarn.client: application report application_1434986503384_0670 (state: accepted) 15/07/08 13:57:19 info yarn.client:   client token: n/a  diagnostics: n/a  applicationmaster host: n/a  applicationmaster rpc port: -1  queue: testing  start time: 1436356638869  final status: undefined  tracking url: http://node-00a/cluster/app/application_1434986503384_0670  user: xxx 15/07/08 13:57:20 info yarn.client: application report application_1434986503384_0670 (state: accepted) 15/07/08 13:57:21 info yarn.client: application report application_1434986503384_0670 (state: accepted) 15/07/08 13:57:23 info yarn.client: application report application_1434986503384_0670 (state: failed) 15/07/08 13:57:23 info yarn.client:   client token: n/a  diagnostics: application application_1434986503384_0670 failed 2 times due container appattempt_1434986503384_0670_000002 exited  exitcode: 1 due to: exception container-launch. container id: container_1434986503384_0670_02_000001 exit code: 1 [...] 

in yarn log, find following error message indicating wrong usage of parameters:

container: container_1434986503384_0670_01_000001 on node-01b_8041 =================================================================================================== logtype:stderr log upload time:mi jul 08 13:57:22 +0200 2015 loglength:764 log contents:     unknown/unsupported param list(--arg, hdfs://nameservice1/user/xxx/readme.md_count, --executor-memory, 1024m, --executor-cores, 1, --num-executors, 2)     usage: org.apache.spark.deploy.yarn.applicationmaster [options]  options:   --jar jar_path       path application's jar file (required)   --class class_name   name of application's main class (required)   --args args          arguments passed application's main class.                        mutliple invocations possible, each passed in order.   --num-executors num    number of executors start (default: 2)   --executor-cores num   number of cores executors (default: 1)   --executor-memory mem  memory per executor (e.g. 1000m, 2g) (default: 1g)  end of logtype:stderr 

as same application runs when uploading local assembly file upon submission, seems come down assembly file. 1 on cluster wrong/different version? how validate that? other reasons might cause? warning warn util.nativecodeloader: ... possibly related?

the same happens when set (deprecated) environment variable spark_jar instead of setting spark.yarn.jar.

asking obvious question here: sure spark-assembly.jar on hdfs same 1 have locally? if not, can try uploading local spark-assembly home directory on hdfs , try again?