Hi everyone,
Looks like we're hitting this bug - https://issues.apache.org/jira/browse/OOZIE-2380 - running a hive script in Oozie to insert data into a partitioned, managed Hive table, ORC format, snappy compression (SparkTestORCSnappy) from an external partitioned Hive tables (dingo_01_detections and dingo_01_parameters), text format. This Hive script works fine from the Hue editor, as well as from the beeline and hive command lines.
Here's the hive script =
set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.exec.max.dynamic.partitions.pernode=1000;
set hive.exec.max.dynamic.partitions=1000;
SET hive.exec.parallel=true;
SET hive.vectorized.execution.enabled=true;
use skynet;
INSERT OVERWRITE table SparkTestOrcSnappy PARTITION(filename, RA_Range , Dec_Range)
select d.x,d.y,
concat(cast(d.x as STRING), '-',cast(d.y as string )) ObjectPosition,
d.z freq, vel wavelengths,f_peak flux, p.threshold,
d.filename,
case
when floor(d.x) between 0 and 20 then 20
when floor(d.x) between 21 and 40 then 40
when floor(d.x) between 41 and 60 then 60
when floor(d.x) between 61 and 80 then 80
else 100
end RA_Range,
case
when floor(d.y) between 0 and 20 then 20
when floor(d.y) between 21 and 40 then 40
when floor(d.y) between 41 and 60 then 60
when floor(d.y) between 61 and 80 then 80
else 100
end Dec_Range
from dingo_01_detections d
inner join dingo_01_parameters p
on d.idmsb=p.idmsb and d.idlsb=p.idlsb
where d.filename='${Partition}' and p.filename='${Partition}';
Here's the error section of the Oozie job log -
16914 [uber-SubtaskRunner] ERROR org.apache.hadoop.hive.ql.Driver - FAILED: IllegalArgumentException java.net.URISyntaxException: Relative path in absolute URI: file:./tmp/yarn/0083d553-cf02-4730-a1b9-aa5bd854904a/hive_2016-02-14_11-44-01_759_9215421282887214185-1
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: file:./tmp/yarn/0083d553-cf02-4730-a1b9-aa5bd854904a/hive_2016-02-14_11-44-01_759_9215421282887214185-1
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.<init>(Path.java:197)
at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:267)
at org.apache.hadoop.hive.ql.Context.getLocalScratchDir(Context.java:302)
at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinTaskDispatcher.processCurrentTask(CommonJoinTaskDispatcher.java:539)
at org.apache.hadoop.hive.ql.optimizer.physical.AbstractJoinTaskDispatcher.dispatch(AbstractJoinTaskDispatcher.java:179)
at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.dispatch(TaskGraphWalker.java:111)
at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.walk(TaskGraphWalker.java:180)
at org.apache.hadoop.hive.ql.lib.TaskGraphWalker.startWalking(TaskGraphWalker.java:125)
at org.apache.hadoop.hive.ql.optimizer.physical.CommonJoinResolver.resolve(CommonJoinResolver.java:79)
at org.apache.hadoop.hive.ql.optimizer.physical.PhysicalOptimizer.optimize(PhysicalOptimizer.java:107)
at org.apache.hadoop.hive.ql.parse.MapReduceCompiler.optimizeTaskPlan(MapReduceCompiler.java:273)
at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:225)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10348)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10141)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:222)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:421)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1110)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1158)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1047)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1037)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:305)
at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:419)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:708)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:321)
at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:298)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:49)
at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:378)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:296)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:181)
at org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:224)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.URISyntaxException: Relative path in absolute URI: file:./tmp/yarn/0083d553-cf02-4730-a1b9-aa5bd854904a/hive_2016-02-14_11-44-01_759_9215421282887214185-1
at java.net.URI.checkPath(URI.java:1804)
at java.net.URI.<init>(URI.java:752)
at org.apache.hadoop.fs.Path.initialize(Path.java:203)
... 51 more
16914 [uber-SubtaskRunner] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=compile start=1455450241758 end=1455450245952 duration=4194 from=org.apache.hadoop.hive.ql.Driver>
16914 [uber-SubtaskRunner] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
16914 [uber-SubtaskRunner] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=releaseLocks start=1455450245952 end=1455450245952 duration=0 from=org.apache.hadoop.hive.ql.Driver>
16967 [uber-SubtaskRunner] INFO org.apache.hadoop.hive.ql.log.PerfLogger - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
16967 [uber-SubtaskRunner] INFO org.apache.hadoop.hive.ql.log.PerfLogger - </PERFLOG method=releaseLocks start=1455450246005 end=1455450246005 duration=0 from=org.apache.hadoop.hive.ql.Driver>
<<< Invocation of Hive command completed <<<
Hadoop Job IDs executed by Hive:
Intercepting System.exit(40000)
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.HiveMain], exit code [40000]
Oozie Launcher failed, finishing Hadoop job gracefully
Oozie Launcher, uploading action data to HDFS sequence file: hdfs://artemis.rc.pawsey.org.au:8020/user/prometheus/oozie-oozi/0000000-160214114141716-oozie-oozi-W...
Oozie Launcher ends
Any ideas? Configuration issue, or do we have to patch and rebuild Oozie as per the Apache JIRA?
Thanks in advance
Cheers
Geoff
