Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive CLI Snappy Error

Solved Go to solution

Hive CLI Snappy Error

Contributor

While running select count(*) from lte where date_id = '20160524'; experiencing below error where as select without where clause works fine.

Caused by: java.io.IOException: Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec. at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:217) at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:121) at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>(PipelinedSorter.java:116) at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start(OrderedPartitionedKVOutput.java:142) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:142) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149) ... 14 more Caused by: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65) at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134) at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>(ExternalSorter.java:208) ... 19 more ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:9, Vertex vertex_1464716100177_0034_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1464716100177_0034_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1464716100177_0034_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Hive CLI Snappy Error

Contributor

solved the issue by updating mapred.admin.user.env , since the cluster was upgraded from HDP 2.1 to HDP 2.3

View solution in original post

3 REPLIES 3
Highlighted

Re: Hive CLI Snappy Error

Guru

You can check with

hadoop checknative -a

to see if Snappy native is installed.

Also check if you have org.apache.hadoop.io.compress.SnappyCodec in io.compression.codecs in core-site.xml

Highlighted

Re: Hive CLI Snappy Error

Contributor

i did check, they do exists

16/05/31 18:39:50 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 16/05/31 18:39:50 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /usr/hdp/2.3.4.7-4/hadoop/lib/native/libhadoop.so.1.0.0 zlib: true /lib64/libz.so.1 snappy: true /usr/hdp/2.3.4.7-4/hadoop/lib/native/libsnappy.so.1 lz4: true revision:99 bzip2: true /lib64/libbz2.so.1

Highlighted

Re: Hive CLI Snappy Error

Contributor

solved the issue by updating mapred.admin.user.env , since the cluster was upgraded from HDP 2.1 to HDP 2.3

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here