our spark shell action job is failing with
ERROR SparkContext: Error initializing SparkContext.java.lang.IllegalArgumentException: requirement failed: Cannot read Hadoop config dir /var/run/cloudera-scm-agent/process/**-yarn-NODEMANAGER.
we included below line to the xml. but still its looking for /var location
can you please help here!!
We highly recommend that you use the Spark action and not the shell action for Spark.
Also make sure that you configure the gateways for Spark on the system.
If you need more detail you will need to provide a little more about what you are doing.
Is there a working example about how to start a pyspark program as spark action into oozie?