Member since
06-27-2023
5
Posts
0
Kudos Received
0
Solutions
01-24-2024
06:08 PM
Hello, I'm new to using oozie and I've been trying to figure out how to use the output of an action as the input of the next action in Oozie. I'm actually coming from using traditional ETLs, where the output of one Step can be used as arguments in the next Step. My question is whether the following points are possible with Oozie: Use the output of one action as the input of the next action Let a subsequent action be executed "n" times according to the output of the first action. Like sub jobs or something. Actually, in order to know if this is possible, we are trying to run a very basic dump example where we read and insert with two hive actions, one for read a table and another one to write. Something like the following example created with Hue Oozie Editor. <workflow-app name="Workflow" xmlns="uri:oozie:workflow:0.5"> <start to="hive-04f4"/> <kill name="Kill"> <message>Error al realizar la acción. Mensaje de error [${wf:errorMessage(wf:lastErrorNode())}]</message> </kill> <action name="hive-04f4" cred="hive2"> <hive2 xmlns="uri:oozie:hive2-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <jdbc-url>jdbc:hive2://host:10000/default</jdbc-url> <script>${wf:appPath()}/hive-04f4.sql</script> </hive2> <ok to="hive-1c24"/> <error to="Kill"/> </action> <action name="hive-1c24" cred="hive2"> <hive2 xmlns="uri:oozie:hive2-action:0.1"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <jdbc-url>jdbc:hive2://host:10000/default</jdbc-url> <script>${wf:appPath()}/hive-1c24.sql</script> <param>fecha=${fecha_insercion}</param> <param>registro=${now}</param> </hive2> <ok to="End"/> <error to="Kill"/> </action> <end name="End"/> </workflow-app>
... View more
Labels:
- Labels:
-
Apache Oozie
12-11-2023
11:32 AM
Hi @mszurap I had troubles because the Oozie UI does not allow me to create the sqoop action with "command" space in blank, but once created and putting everything in "args" it works! Thanks.
... View more
12-11-2023
09:42 AM
Hi @mszurap Given the issue ticket you mentioned, it should have been solved back in 2017 but we are using Hue™ 4.10.0 with version cdh7.1.8.42 and it still happens in the "Sqoop 1" editor. I tried it from the Oozie workflow and with both modes, "Actions" and "Documents", but it gave the exact same error when it reaches the sqoop action. Thanks for your response.
... View more
12-11-2023
09:36 AM
Hi @paras I tried your suggestion but it didn't work. It returned a similar error. Sqoop command arguments :
import
--connect
jdbc:oracle:thin:@server/service
--query
'SELECT
*
from
schema.mytable
where
$CONDITIONS'
[..... <<more log>> ........]
Fetching child yarn jobs
tag id : oozie-176af7fc4166c6a2f20a49ccecd8c42109b5ab3fa82786eb191d19f4732d9afa
No child applications found
=================================================================
<<< Invocation of Sqoop command completed <<<
No child hadoop job is executed.
Intercepting System.exit(1)
null
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:417)
at org.apache.oozie.action.hadoop.LauncherAM.access$400(LauncherAM.java:55)
at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:232)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:226)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:156)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:144)
Caused by: java.lang.SecurityException: Intercepted System.exit(1)
at org.apache.oozie.action.hadoop.security.LauncherSecurityManager.checkExit(LauncherSecurityManager.java:57)
at java.lang.Runtime.exit(Runtime.java:101)
at java.lang.System.exit(System.java:987)
at org.apache.sqoop.Sqoop.main(Sqoop.java:260)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:221)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:206)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:135)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:52)
... 16 more
... View more
12-08-2023
04:00 PM
Hello, I'm running a sqoop job through Hue. This imports data from Oracle, but the value of --query is being divided by whitespace and produces sqoop flag error because it does not recognize these parameters import --connect jdbc:oracle:thin:@server/service --query 'select * from mytable' The logs show somethng like this: Sqoop command arguments :
import
--connect
jdbc:oracle:thin:@server/service
--query
'SELECT
*
from
mytable' ------------------------------------------------------------------------------------------------------- 17:48:29.329 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing arguments for import:
17:48:29.329 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: select
17:48:29.329 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: *
17:48:29.329 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: from I have tried to scape with single, double quotes, backslash but nothing works.
... View more
Labels:
- Labels:
-
Apache Sqoop
-
Cloudera Hue