Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain]

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain]

New Contributor

Hi,

 

We have recently updated the cluster to cdh 5.5.2.

 

After the upgrading the workflows which contain a spark action are returning this error. 

 

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain], main() threw exception, requirement failed: Cannot read Hadoop config dir /var/run/cloudera-scm-agent/process/10902-yarn-NODEMANAGER.
java.lang.IllegalArgumentException: requirement failed: Cannot read Hadoop config dir /var/run/cloudera-scm-agent/process/10902-yarn-NODEMANAGER.
	at scala.Predef$.require(Predef.scala:233)

This is one of our workflows:

<workflow-app
        name="cust-info-cache-builder"
        xmlns="uri:oozie:workflow:0.5"
        xmlns:sla="uri:oozie:sla:0.2">

    <start to="build-cust-info"/>
    <action name="build-cust-info">
        <spark xmlns="uri:custom:spark-action:0.1">
            <job-tracker>${resourceManager}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>oozie.launcher.mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
                <property>
                    <name>spark.yarn.queue</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <master>yarn</master>
            <mode>cluster</mode>
            <name>Customer Information Cache Builder</name>
            <class>CustomerInformationBuilder</class>
            <jar>workflows.jar</jar>
            <spark-opts>--driver-memory 1G --executor-memory 1G --executor-cores 1 --num-executors 1 ${sparkGeneralOpts}</spark-opts>
            <arg>${1dir}</arg>
            <arg>${2dir}</arg>
            <arg>${3Dir}</arg>
        </spark>
        <ok to="end"/>
        <error to="email-error"/>
        <sla:info>
            <sla:nominal-time>${nominal_time}</sla:nominal-time>
            <sla:should-start>${10 * MINUTES}</sla:should-start>
            <sla:should-end>${30 * MINUTES}</sla:should-end>
            <sla:notification-msg>Notification Cache Building</sla:notification-msg>
            <sla:upstream-apps>UpStream Cache Building</sla:upstream-apps>
        </sla:info>
    </action>
    <action name="email-error">
        <email xmlns="uri:oozie:email-action:0.1">
            ...
        </email>
        <ok to="kill"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>  

 

We have checked yarn and oozie logs and we are not finding any clue. Does someone have a similar error?

 

Thanks in advance.

Regards,

 

2 REPLIES 2

Re: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain]

Master Guru
I've usually observed this over Shell actions running Spark commands but not on a Spark action. Could you reconfirm if it does happen on your quoted workflow that uses a direct Spark Action?

For the shell action, while https://issues.apache.org/jira/browse/OOZIE-2481 would help in future, the workaround is to simply add the below line into the shell script:

export YARN_CONF_DIR=$HADOOP_CONF_DIR
Highlighted

Re: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SparkMain]

New Contributor

Hi Harsh,

 

Thanks a lot for your response.

 

I am using a spark action: <spark xmlns="uri:custom:spark-action:0.1">

 

Do you know how to export this variable from oozie?

 

I think the problem is with YARN_CONF_DIR variable which is not being update with the proper path.

 

This YARN_CONF_DIR should have the same value that HADOOP_CONF_DIR: export YARN_CONF_DIR=$HADOOP_CONF_DIR

 

My problem is that I do not know how to do it in oozie. Do you know how to do it?

 

Thanks in advance.

Regards,