Member since
07-08-2013
35
Posts
19
Kudos Received
5
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
9785 | 03-14-2016 09:34 PM | |
4383 | 02-05-2016 12:11 PM | |
10041 | 08-12-2015 11:44 AM | |
3296 | 08-14-2014 10:14 AM | |
7026 | 10-14-2013 11:53 AM |
03-14-2016
09:34 PM
1 Kudo
Hi, You can use the SSH Action to execute shell scripts on specific nodes. See http://archive.cloudera.com/cdh5/cdh/5/oozie/DG_SshActionExtension.html And here's a very detailed example of using the SSH Action: http://hadooped.blogspot.com/2013/10/apache-oozie-part-13-oozie-ssh-action_30.html
... View more
02-05-2016
12:11 PM
We're aware of this issue and there's a fix coming in CDH 5.5.2. In the meantime, you can add the following to your oozie-site.xml (or safety-valve in CM's Oozie configuration) as a workaround: <property>
<name>oozie.email.smtp.password</name>
<value>aaa</value>
</property> The value can be any string (it won't actually be used if oozie.email.smtp.auth is false, the default). For those interested, this is fixed by OOZIE-2365.
... View more
08-12-2015
11:44 AM
1 Kudo
What you can do as a workaround, is split up your long-running Coordinators. For example, instead of making your Coordinator run for years? forever?, make it run for, say, 6 months. And have an identical Coordinator scheduled to start exactly when that one ends. This will allow Oozie to cleanup the old child Workflows from that Coordinator every 6 months. Otherwise, you can schedule a cron job to manually delete old jobs from the Database. However, please be careful about this. When deleting a workflow job from the WF_JOBS table, you'll also need to delete the workflow actions from the WF_ACTIONS table that belong to it, as well as the coordinator action from the WF_ACTIONS table that it belongs to. If you miss something, it will likely cause problems.
... View more
08-11-2015
10:44 AM
1 Kudo
Hi, By default, Oozie will not purge child jobs if the parent is not eligible to be purged. In your case, because the Coordinator job is still running, none of the child Workflow jobs will be purged. Which version of CDH are you using? Starting with CDH 5.2.0, you can change it so that Oozie will delete the child jobs even if the parent job is still running. To do that, you can set oozie.service.PurgeService.purge.old.coord.action=true in oozie-site. Also, starting with CM 5.4, the Oozie Configuration page has controls for these configs, so you don't need the safety-valve anymore here.
... View more
06-03-2015
04:54 PM
1 Kudo
By the way, OOZIE-2159 will fix the 'oozie validate' command not supporting custom actions by moving the check from the Oozie client to the Oozie server, where the custom action is configured.
... View more
05-21-2015
09:55 AM
That blog post is a little outdated at this point. It all depends on your Yarn configuration: - DefaultContainerExecutor: 'yarn' - LinuxContainerExectutor: - With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=false (default), it runs as yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user (default is 'nobody') - With yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users=true, it runs as the user submitting the workflow I'd encourage you to use any of the other actions (e.g. Java action) if possible; they will all run as the user who submitted the workflow.
... View more
05-21-2015
09:46 AM
1 Kudo
Hi, To test a custom action, I'd recommend simply running it in an Oozie server and seeing if you run into any problems. You can also attach a debugger to Oozie to help there. To save on turn-around time, you can also write a unit test that starts up LocalOozie and tries to submit your action. To attach a debugger to Oozie, you should add the following to the "Oozie Service Environment Advanced Configuration Snippet (Safety Valve)" in Cloudera Manager (if not using CM, then you'd export this in oozie-env.sh: CATALINA_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005" Besides using the debugger I mentioned above, looking at the launcher output (assuming you added print statements) can also be helpful; you can attach a debugger to the launcher job, but that's a bit harder because you need to attach it to the MR container instead of the Oozie server. Oozie is pretty easy to run anywhere. You can run it in a VM, on a cluster, or even on your Mac. When developing for Oozie, I typically run it on my Mac against a pseudo-distributed cluster, also on my Mac. You can use oozie.service.HadoopAccessorService.hadoop.configurations to point Oozie at any Hadoop cluster, or even multiple ones. Nothing special. Just make sure that your custom action's jar is in the Oozie servers classpath and that if your custom action has a sharelib, it's deployed in HDFS and Oozie sees it.
... View more
08-14-2014
11:53 AM
2 Kudos
IIRC, for Hue to switch between MR1 and MR2, you have to ensure that a few of the other services are also switched. For example, in CM's Hive configuration page, I believe it has a similar config property to Oozie's that lets you switch between MR1 and MR2. Make sure they are both set to MR2. You should also check if any other services have a similar property. Then restart Hue, of course.
... View more
08-14-2014
10:14 AM
1 Kudo
Hi, You can definately do this. I'd have to see the actual exception to be sure, but I'm guessing by "it doesn't find the main method of Sqoop." you were getting a ClassNotFoundException on SqoopMain? SqoopMain is actually an Oozie class and is in the oozie-sharelib-sqoop-4.0.0-cdh5.0.2.jar in the sqoop sharelib. You'll need to copy this jar into your sqoopPatched sharelib. You may also need some of the other jars from the Sqoop sharelib. The safest thing is to copy the "sqoop" sharelib to "sqoopPatched" and then just replace your custom sqoop-1.4.4-cdh5.0.2.jar in there; this way, you'll be sure to have everything. - Robert
... View more
02-14-2014
05:49 PM
Can you past your workflow.xml? It sounds like you're trying to use the hcat/hive metastore crededentials stuff (i.e. a <credentials> section) without adding the Credential classes to oozie-site.xml: i.e. <property>
<name>oozie.credentials.credentialclasses</name>
<value>
hcat=org.apache.oozie.action.hadoop.HCatCredentials,
hbase=org.apache.oozie.action.hadoop.HbaseCredentials,
hive2=org.apache.oozie.action.hadoop.Hive2Credentials
</value>
</property> This page has more details: http://archive.cloudera.com/cdh4/cdh/4/oozie/DG_UnifiedCredentialsModule.html
... View more