Support Questions

Find answers, ask questions, and share your expertise

Hive CLI Snappy Error


While running select count(*) from lte where date_id = '20160524'; experiencing below error where as select without where clause works fine.

Caused by: Unable to get CompressorType for codec ( This is most likely due to missing native libraries for the codec. at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>( at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>( at org.apache.tez.runtime.library.common.sort.impl.PipelinedSorter.<init>( at org.apache.tez.runtime.library.output.OrderedPartitionedKVOutput.start( at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init( at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor( ... 14 more Caused by: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at at at org.apache.tez.runtime.library.common.sort.impl.ExternalSorter.<init>( ... 19 more ]], Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:9, Vertex vertex_1464716100177_0034_1_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]Vertex killed, vertexName=Reducer 2, vertexId=vertex_1464716100177_0034_1_01, diagnostics=[Vertex received Kill while in RUNNING state., Vertex did not succeed due to OTHER_VERTEX_FAILURE, failedTasks:0 killedTasks:1, Vertex vertex_1464716100177_0034_1_01 [Reducer 2] killed/failed due to:OTHER_VERTEX_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:1



solved the issue by updating mapred.admin.user.env , since the cluster was upgraded from HDP 2.1 to HDP 2.3

View solution in original post



You can check with

hadoop checknative -a

to see if Snappy native is installed.

Also check if you have in io.compression.codecs in core-site.xml


i did check, they do exists

16/05/31 18:39:50 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 16/05/31 18:39:50 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library Native library checking: hadoop: true /usr/hdp/ zlib: true /lib64/ snappy: true /usr/hdp/ lz4: true revision:99 bzip2: true /lib64/


solved the issue by updating mapred.admin.user.env , since the cluster was upgraded from HDP 2.1 to HDP 2.3