Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Spark shell action fails : ERROR SparkContext: Error initializing SparkContext

Spark shell action fails : ERROR SparkContext: Error initializing SparkContext

New Contributor

Hi,

 

our spark shell action job is failing with

 

ERROR SparkContext: Error initializing SparkContext.java.lang.IllegalArgumentException: requirement failed: Cannot read Hadoop config dir /var/run/cloudera-scm-agent/process/**-yarn-NODEMANAGER.

 

we included below line to the xml. but still its looking for /var location  

<env-var>HADOOP_CONF_DIR=/etc/hadoop/conf</env-var>

 

can you please help here!!

 

thanks 

sathish

 

2 REPLIES 2

Re: Spark shell action fails : ERROR SparkContext: Error initializing SparkContext

Super Collaborator

We highly recommend that you use the Spark action and not the shell action for Spark.

Also make sure that you configure the gateways for Spark on the system.

 

If you need more detail you will need to provide a little more about what you are doing.

 

Wilfred

Highlighted

Re: Spark shell action fails : ERROR SparkContext: Error initializing SparkContext

New Contributor

Is there a working example about how to start a pyspark program as spark action into oozie?