Try running   curl http://oozie_host:oozie_port/oozie/v2/admin/status   ... View more

I'm not sure why that's not working.  I can see that guava 16 is being passed to Spark and guava 14 isn't there (FYI: you also replaced guava 11 from Hadoop with 16, which may cause problems for Hadoop).   Can you try yarn-client or yarn-cluster mode instead of local?  My understanding is that local mode doesn't always work right. ... View more

As I said, you should look at the stdout of the Launcher Job.  Can you post the stdout, stderr, and syslogs from the Launcher Job somewhere and link to them here?  It contains a log of useful information and might help narrow down your classpath problem. ... View more

As I said, you should look at the stdout of the Launcher Job.  Can you post the stdout, stderr, and syslogs from the Launcher Job somewhere and link to them here? ... View more

It sounds like you have the guava 16 jar in the classpath then. If you look at the stdout from the Launcher Job, you should see that it's listed there and that it's being passed to Spark.   | 1.built  spark-assembly-1.5.3-hadoop2.6.0.jar with guava 16.0.1 by myself |  2.renamed it as  | spark-assembly-1.5.0-cdh5.5.0-hadoop2.6.0-cdh5.5.0.jar under  | /opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/jars That's going to result in all kinds of problems.  You must use CDH Spark with CDH.   I can't speak to /etc/spark/conf/ classpath.txt (You'll have to ask on the Spark forum).  Though w hen run from Oozie, I don't think that Spark uses that file.     Keep in mind that what you're trying to do here (replacing jars we're shipping) isn't really supported or tested, so it might not even be possible to do what you want. ... View more

Hi,   I think you need to re-read those 4 ways a little more carefully.  #1 clearly states: There is no need to ever point [oozie.libpath] at the ShareLib location. (I see that in a lot of workflows.) Oozie knows where the ShareLib is and will include it automatically if you set oozie.use.system.libpath=true in job.properties. which is exactly what you tried.     Any of the 4 methods should work.  (Except for #3: we're currently aware of a known issue where the Spark Action does not allow <file> or <archive> tags; we're planning on fixing that in a later release.)   Though keeping in mind that you're trying to replace a jar in the Sharelib, you need to go with #4 anyway, and replace the jar in the Spark Sharelib subdir (which you did already).   Though remember that this means any Spark Action that anyone runs will have this modification.  If you want to protect other users and workflows from your changes, you can create a new dir in the Sharelib, say "spark_guava_16", and set " oozie.action.sharelib.for.spark" to "spark_guava_16".  This is also described in the blog post in the "Overriding the ShareLib" section.  If that's not a concern, then you don't need to bother.   Please check the following: Run the oozie admin -shareliblist spark command.  It will print out a list of the jars from the Spark sharelib directory that Oozie is currently aware of and using.  If you did replace the guava 14 jar with the 16 jar there, it should show up in that output.  If not, you need to restart the Oozie server or run the oozie admin -sharelibupdate command.  Also pay attention to the lib_<timestamp> directory in the output; perhaps you're changing an old directory When you run the job, look at the stdout from the Launcher Job.  It prints out a lot of useful information, including the classpath.  Do you see the guava 16 or 14 jar there?   However, as I said in the email thread in the oozie mailing list, Spark is expecting guava 14.  Guava tends to not be very compatible across major versions; so you may encounter other problems if you force it to use guava 16. ... View more

Hi,   Oozie only uses ZooKeeper when configured in HA (High Availability).  CM only emits oozie.zookeeper.connection.string when HA is enabled, even though the ZooKeeper Service option always showing up in Oozie's Configuration page in CM. And when CM doesn't emit this, Oozie uses the default, which is localhost; but Oozie's not actually using it or talking to ZooKeeper.   To investigate your slower job, you  should look through the Oozie Server logs and action output in Yarn to figure out what's actually happening.     ... View more

Hi,   The Launcher Job runs as Map task in an MR job, and runs the command from there (Sqoop in this case).  Sqoop can then launch addiitonal jobs.  Because this Launcher Job is sitting around waiting for all of the other jobs to finish, it's actually possible to deadlock the cluster or queue, depending on the size of your cluster and/or your scheduler settings.  A good sign that this happened is if you see the "heart beat" message over and over again in the Launcher Job, and one or more ACCEPTED jobs in the RM that are not starting because there are not enough resources.  It sounds like that might be happening here.  I'm not that familiar with the Capacity Scheduler (we typically recommend the Fair Scheduler), so I can't really advise you on that specifically. ... View more

The "timezone" attribute in the Coordiantor is a little misleading.  You still have to specify your dates in UTC (hence why they end with 'Z').  The "timezone" attribute is only used for daylight saving time calculations.     ... View more

Usually if there is a delay between actions, especially if it's 10 minutes, it means that Oozie didn't receive the callback  that the job finished .  To avoid having to poll the remote server frequently, Oozie only checks once every 10 minutes (by default).  Obviously we want the Oozie server to find out sooner, so the SSH action is configured to issue an http callback to the Oozie server once it's finished to let Oozie know.  If that callback gets blocked somehow, then Oozie will take 10 minutes to notice.  Can you make sure that the target machine is able to send a GET request to the Oozie server?  curl also needs to be installed (the SSH action uses curl to make the GET request). ... View more

Oozie can work with both MR1 and MR2 (YARN), though not at the same time.  The VM is configured for MR2 out of the box, but the examples that ship with Oozie target MR1.  You can point the example at MR2 by simply changing the "jobTracker" parameter from  jobTracker=localhost:8021 (the JT) to jobTracker=localhost:8032 (the RM).  Even though it's called "jobTracker", Oozie will use MR2 if you point it at an RM. ... View more

Usually if there is a delay between actions, it means that Oozie didn't receive the callback from Hadoop.  To not spam the JT/RM with job status requests, Oozie normally only queries it once every ten minutes (by default).  The reason you normally don't have to wait 10 minutes is that Oozie configures Hadoop to send back a message to Oozie over http once the job has finished.  If that callback doesn't go through for some reason, then you'll typically see a 10 minute delay between actions.   Can you check the launcher job logs for any error messages?  If it couldn't send the callback, it would probably say something there. ... View more

That blog post is a little outdated at this point.  It all depends on your Yarn configuration: - DefaultContainerExecutor: 'yarn' - LinuxContainerExectutor:     - With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=false (default), it runs as yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user (default is 'nobody')      - With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=true , it runs as the user submitting the workflow     I'd encourage you to use any of the other actions (e.g. Java action) if possible; they will all run as the user who submitted the workflow. ... View more

You need to copy or symlink the following jars from  /opt/cloudera/parcels/CDH/lib/hbase to  /opt/cloudera/parcels/CDH/lib/oozie/libserver hbase-common.jar hbase-client.jar hbase-server.jar hbase-protocol.jar lib/netty-*Final.jar lib/htrace-core.jar  and restart Oozie.   If using packages, the above paths would be /usr/lib/hbase and /usr/lib/oozie/libserver instead.   We're planning on doing this for users out-of-the-box in a future CDH release. ... View more

If you look closely, you'll see that your start node is trying to go to "sqoop-node" but your Sqoop node is named "sqoop-node1".  These should match. ... View more

Oozie runs on port 11000; IIRC, 8888 is Hue, and there's no /oozie on Hue 🙂   Try this: $ oozie admin -sharelibupdate -oozie http://vm-cluster-node1:11000/oozie   Also, instead of having to specify -oozie <url> everytime you do an Oozie command, you can set the OOZIE_URL environment variable. e.g. $ export OOZIE_URL=http://vm-cluster-node1:11000/oozie   ... View more

Hi,   The launcher job sticks around the entire time the sqoop job is running (because it's running the sqoop CLI).  So when using Oozie, you need to make sure that your cluster has capacity.  In MR1 this was quite common if you didn't have enough map slots.  In MR2, I've seen this happen if you haven't configured your Scheduler properly or if you don't have enough memory in your node manager. ... View more

I don't think that's the same problem described on that other forum; they said that it was OOZIE-1447 that fixed it.  However, CDH 4.4.0 has OOZIE-1447 already.     Can you post the error?  Also keep in mind that using eval isn't recommended/supported. ... View more

I haven't tested this, but can you try adding the following property to your aciton's <configuration> section? dfs.replication=3   You can also try setting the following property in your hdfs-site configuration; you may have to put it in the safety valve for HDFS if its not exposed in CM dfs.replication.max=10   As a general point, I'd strongly recommend you upgrade to a version of CDH 4.x (ideally the latest).  We've made some signficant improvements in Oozie since CDH 3uX, especially for Coordinators.  CDH 3 has also reached end-of-life.   ... View more

Hi Jared,   What do you mean that the JobID string is empty?  OOZIE-1447 is to fix an issue caused by the Sqoop action not launching an MR job, which if I understand correctly, should only be the case if you're using "eval" (which isn't meant for production use FYI).  Is that what you're doing?   In either case, OOZIE-1447 is in CDH 4.4.0 and later; so if this is what you're running into, upgrading to CDH 4.4.0 should fix it. ... View more