Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Who agreed with this topic

oozie issue with pyspark

New Contributor

Hello everyone,

I only have access to a HUE (3.7.0) interface and through this I’m trying to run a oozie workflow which contains only a Spark action to execute a python script.

 

My python file (consoli.py) is under the workspace “/user/hue/oozie/workspaces/hue-oozie-1470040761.25/lib »

Unfortunately, I have this issue (that I don’t have if I try it with a jar file instead of a python script …) :

 

>>> Invoking Spark class now >>>

 

  python: can't open file '/opt/yarn/nm/usercache/moet/appcache/application_1460044135313_0530/container_1460044135313_0530_01_000001/lib/consoli.py': [Errno 2] No such file or directory

  Intercepting System.exit(2)

 

The initial workflow.xml (the one generated by oozie) is:

 

<workflow-app name="Consolidation" xmlns="uri:oozie:workflow:0.5">

    <start to="spark-839a"/>

    <kill name="Kill">

        <message>L&#39;action a échoué, message d&#39;erreur[${wf:errorMessage(wf:lastErrorNode())}]</message>

    </kill>

    <action name="spark-839a">

        <spark xmlns="uri:oozie:spark-action:0.1">

            <job-tracker>${jobTracker}</job-tracker>

            <name-node>${nameNode}</name-node>

            <configuration>

                <property>

                    <name></name>

                    <value></value>

                </property>

                <property>

                    <name></name>

                    <value></value>

                </property>

            </configuration>

            <master>local[*]</master>

            <mode>client</mode>

            <name>MySpark</name>

              <class>main</class>

            <jar>lib/consoli.py</jar>

        </spark>

        <ok to="End"/>

        <error to="Kill"/>

    </action>

    <end name="End"/>

</workflow-app>

 

Any help would be appreciated :)

 

Who agreed with this topic