Member since
08-11-2016
2
Posts
1
Kudos Received
0
Solutions
11-18-2016
02:41 PM
1 Kudo
We are trying to run a Cascading job using Tez flow connector. Job is failing with the following error. but the same job run fine with mapreduce flow connector. After looking through Tez Source code LocalDirAllocator is initialized with key tez.runtime.framework.local.dirs. Not sure where an when this property is set or it has any impact to resolve the issue. any idea to revolve the issue is appreciated. Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for output/attempt_1479221177536_0001_1_01_000000_2_10015_0/file.out
at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:402)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150)
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131)
at org.apache.tez.runtime.library.common.task.local.output.TezTaskOutputFiles.getSpillFileForWrite(TezTaskOutputFiles.java:207)
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.spill(PipelinedSorter.java:447)
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.sort(PipelinedSorter.java:231)
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.collect(PipelinedSorter.java:327)
at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.write(PipelinedSorter.java:283)
at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput$1.write(OrderedPartitionedKVOutput.java:164)
at cascading.flow.tez.stream.element.OldOutputCollector.collect(OldOutputCollector.java:57)
at cascading.tap.hadoop.util.MeasuredOutputCollector.collect(MeasuredOutputCollector.java:70)
at cascading.flow.tez.stream.element.TezGroupGate.wrapGroupingAndCollect(TezGroupGate.java:125)
at cascading.flow.hadoop.stream.HadoopGroupGate.receive(HadoopGroupGate.java:109)
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Tez
08-11-2016
12:25 PM
I am
facing class incompatibility issue when cascading-hive job is run using TEZ on HDP
Sandbox version 2.4. jackson-mapper-asl.jar is in both hive–exec.jar(bundled)
and Hadoop lib path. Hadoop uses version 1.9.13 and hive-exec seems to
use either 1.9.10 or lower. Currently stopped using Timeline Service for TEZ to overcome the problem. any better alternative? java.lang.AbstractMethodError: org.codehaus.jackson.map.AnnotationIntrospector.findSerializer(Lorg/codehaus/jackson/map/introspect/Annotated;)Ljava/lang/Object;
at org.codehaus.jackson.map.ser.BasicSerializerFactory.findSerializerFromAnnotation(BasicSerializerFactory.java:366)
at org.codehaus.jackson.map.ser.BeanSerializerFactory.createSerializer(BeanSerializerFactory.java:252)
at org.codehaus.jackson.map.ser.StdSerializerProvider._createUntypedSerializer(StdSerializerProvider.java:782)
at org.codehaus.jackson.map.ser.StdSerializerProvider._createAndCacheUntypedSerializer(StdSerializerProvider.java:735)
at org.codehaus.jackson.map.ser.StdSerializerProvider.findValueSerializer(StdSerializerProvider.java:344)
at org.codehaus.jackson.map.ser.StdSerializerProvider.findTypedValueSerializer(StdSerializerProvider.java:420)
at org.codehaus.jackson.map.ser.StdSerializerProvider._serializeValue(StdSerializerProvider.java:601)
at org.codehaus.jackson.map.ser.StdSerializerProvider.serializeValue(StdSerializerProvider.java:256)
at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1613)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$EntityLogFD.writeEntities(TimelineClientImpl.java:924)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.createSummaryFDAndWrite(TimelineClientImpl.java:1438)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.writeSummmaryEntityLogs(TimelineClientImpl.java:1418)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl$LogFDsCache.writeSummaryEntityLogs(TimelineClientImpl.java:1397)
at org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.putEntities(TimelineClientImpl.java:823)
at org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingService.handleEvents(ATSV15HistoryLoggingService.java:375)
at org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingService.access$900(ATSV15HistoryLoggingService.java:53)
at org.apache.tez.dag.history.logging.ats.ATSV15HistoryLoggingService$1.run(ATSV15HistoryLoggingService.java:195)
at java.lang.Thread.run(Thread.java:745)
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Tez
