This task is intended to launch a Spark application. The task submits the Spark application to a YARN cluster
for execution. This task is appropriate for a deployment that has access to a Hadoop YARN cluster. The Spark application jar and the Spark Assembly jar should be referenced from an HDFS location.
The spark-yarn task has the following options:
- spark.app-args
- The arguments for the Spark application. (String[], default:
[]) - spark.app-class
- The main class for the Spark application. (String, default:
<none>) - spark.app-jar
- The path to a bundled jar that includes your application and its dependencies, excluding any Spark dependencies. (String, default:
<none>) - spark.app-name
- The name to use for the Spark application submission. (String, default:
<none>) - spark.assembly-jar
- The path for the Spark Assembly jar to use. (String, default:
<none>) - spark.executor-memory
- The memory setting to be used for each executor. (String, default:
1024M) - spark.num-executors
- The number of executors to use. (Integer, default:
1) - spark.resource-archives
- A comma separated list of archive files to be included with the app submission. (String, default:
<none>) - spark.resource-files
- A comma separated list of files to be included with the application submission. (String, default:
<none>)