How to run Spark jobs with Spark on YARN? This often requires trial and error in order to make it work.
Hue is leveraging Apache Oozie to submit the jobs. It focuses on the yarn-client mode, as Oozie is already running the spark-summit command in a MapReduce2 task in the cluster. You can read more about the Spark modes here.
Here is how to get started successfully:
PySpark
Simple script with no dependency.
Script with a dependency on another script (e.g. hello imports hello2).
For more complex dependencies, like Panda, have a look at this documentation.
Jars (Java or Scala)
Add the jars as File dependency and specify the name of the main jar:
Another solution is to put your jars in the ‘lib’ directory in the workspace (‘Folder’ icon on the top right of the editor).
4 Comments
-
Sorry to trouble you,could you help me?When I build a workflow on oozie using pyspark and sparksql,I get message saying :
The specified datastore driver (“org.apache.derby.jdbc.EmbeddedDriver”) was not found in the CLASSPATH.
Though I added the derby-10.10.1.1.jar to /user/oozie/share/lib and other directory ,it still fails.
I’m using CDH 5.9.0 and Hue 3.11.-
Author
Could you added to a ‘lib’ directory in the workflow?
Or have the Oozie property ‘oozie.libpath’ point to it in HDFS?
-
-
Hey team – I am getting same error as Jiang Hao. I uploaded hive-site.xml to hdfs, and attached in the FILE Attribute on Spark Action. Seems like, the Spark-Action, is not reading the hive-site.xml configuration file.
-
Could you include you hive-site.xml in a directory nameed ‘lib’ in the workspace of the workflow?
-
