Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hive CLI Snappy Error

avatar
Rising Star

While running select count(*) from lte where date_id = '20160524'; experiencing below error where as select without where clause works fine.

Caused by: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec. at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:217) at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:121) at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:116) at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:142) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:142) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149) ... 14 more Caused by: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65) at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134) at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:208) ... 19 more ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:9, Vertex vertex_1464716100177_0034_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1464716100177_0034_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1464716100177_0034_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1

1 ACCEPTED SOLUTION

avatar
Rising Star

solved the issue by updating mapred.admin.user.env , since the cluster was upgraded from HDP 2.1 to HDP 2.3

View solution in original post

3 REPLIES 3

avatar
Guru

You can check with

hadoop checknative -a

to see if Snappy native is installed.

Also check if you have org.apache.hadoop.io.compress.SnappyCodec in io.compression.codecs in core-site.xml

avatar
Rising Star

i did check, they do exists

16/05/31 18:39:50 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 16/05/31 18:39:50 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /usr/hdp/2.3.4.7-4/hadoop/lib/native/libhadoop.so.1.0.0 zlib: true /lib64/libz.so.1 snappy: true /usr/hdp/2.3.4.7-4/hadoop/lib/native/libsnappy.so.1 lz4: true revision:99 bzip2: true /lib64/libbz2.so.1

avatar
Rising Star

solved the issue by updating mapred.admin.user.env , since the cluster was upgraded from HDP 2.1 to HDP 2.3