Unable to deploy Spark jobs with Oozie.

joseluis_navar1 — Tue, 15 Nov 2016 21:12:34 GMT

I need to run 24/7 a (java) spark application in my cluster, for this task Oozie seems to be a good option.

I wrote a workflow and a properties file, however when I try to run it I get the message configuration is not specified although I specify it and the path provided is a local path in the machine where this is being executed. So:

The file and path exists:

[oozie@hdpgw01 tst]$ ls /home/oozie/tst/job.properties
/home/oozie/tst/job.properties

Trying to submit without and with oozie server in the command:

[oozie@hdpgw01 tst]$ oozie job –config /home/oozie/tst/job.properties -submit
java.io.IOException: configuration is not specified
        at org.apache.oozie.cli.OozieCLI.getConfiguration(OozieCLI.java:816)
        at org.apache.oozie.cli.OozieCLI.jobCommand(OozieCLI.java:981)
        at org.apache.oozie.cli.OozieCLI.processCommand(OozieCLI.java:686)
        at org.apache.oozie.cli.OozieCLI.run(OozieCLI.java:639)
        at org.apache.oozie.cli.OozieCLI.main(OozieCLI.java:225)
configuration is not specified
[oozie@hdpgw01 tst]$ oozie job -oozie hdpgw01:11000/oozie –config /home/oozie/tst/job.properties -submit
java.io.IOException: configuration is not specified
        at org.apache.oozie.cli.OozieCLI.getConfiguration(OozieCLI.java:816)
        at org.apache.oozie.cli.OozieCLI.jobCommand(OozieCLI.java:981)
        at org.apache.oozie.cli.OozieCLI.processCommand(OozieCLI.java:686)
        at org.apache.oozie.cli.OozieCLI.run(OozieCLI.java:639)
        at org.apache.oozie.cli.OozieCLI.main(OozieCLI.java:225)
configuration is not specified

Trying to run without and with oozie server in the command:

[oozie@hdpgw01 tst]$ oozie job –config /home/oozie/tst/job.properties -run
java.io.IOException: configuration is not specified
        at org.apache.oozie.cli.OozieCLI.getConfiguration(OozieCLI.java:816)
        at org.apache.oozie.cli.OozieCLI.jobCommand(OozieCLI.java:1055)
        at org.apache.oozie.cli.OozieCLI.processCommand(OozieCLI.java:686)
        at org.apache.oozie.cli.OozieCLI.run(OozieCLI.java:639)
        at org.apache.oozie.cli.OozieCLI.main(OozieCLI.java:225)
configuration is not specified
[oozie@hdpgw01 tst]$ oozie job -oozie hdpgw01:11000/oozie –config /home/oozie/tst/job.properties -run
java.io.IOException: configuration is not specified
        at org.apache.oozie.cli.OozieCLI.getConfiguration(OozieCLI.java:816)
        at org.apache.oozie.cli.OozieCLI.jobCommand(OozieCLI.java:1055)
        at org.apache.oozie.cli.OozieCLI.processCommand(OozieCLI.java:686)
        at org.apache.oozie.cli.OozieCLI.run(OozieCLI.java:639)
        at org.apache.oozie.cli.OozieCLI.main(OozieCLI.java:225)
configuration is not specified

Even if I use some crazy path here I get the same result:

[oozie@hdpgw01 tst]$ oozie job -oozie hdpgw01:11000/oozie –config /home205723984u34oith3ot3/oozie/tst/job.properties -run
java.io.IOException: configuration is not specified
        at org.apache.oozie.cli.OozieCLI.getConfiguration(OozieCLI.java:816)
        at org.apache.oozie.cli.OozieCLI.jobCommand(OozieCLI.java:1055)
        at org.apache.oozie.cli.OozieCLI.processCommand(OozieCLI.java:686)
        at org.apache.oozie.cli.OozieCLI.run(OozieCLI.java:639)
        at org.apache.oozie.cli.OozieCLI.main(OozieCLI.java:225)
configuration is not specified

Is Oozie the best tool for this task? I am missing up something?

Re: Unable to deploy Spark jobs with Oozie.

joseluis_navar1 — Tue, 15 Nov 2016 21:59:47 GMT

This was already answered in stackoverflow. For some reason I there were two different type of dash character in the command I wrote.

question Unable to deploy Spark jobs with Oozie. in Archives of Support Questions (Read Only)

Unable to deploy Spark jobs with Oozie.

Re: Unable to deploy Spark jobs with Oozie.