Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

CDH 5.7.0 native library configuration on Snappy

CDH 5.7.0 native library configuration on Snappy

Expert Contributor

I got the following error when runing a MR job on CDH 5.7.0. However, the same job doesn't fail on quickstart 5.8 VM. Is there a known bug for 5.7.0 and fixed in 5.7.2?(MAPREDUCE-6577)

 

I ran command "hadoop checknative -a | grep snapp" and it seems to be installed.

 snappy:  true /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop/lib/native/libsnappy.so.1

 

Here is the stack trace.

2016-10-11 03:17:05,622 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.

                    at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)

                    at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)

                    at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)

                    at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)

                    at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)

                    at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)

                    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1606)

                    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1486)

                    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:460)

                    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)

                    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)

                    at java.security.AccessController.doPrivileged(Native Method)

                    at javax.security.auth.Subject.doAs(Subject.java:415)

                    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)

                    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

4 REPLIES 4

Re: CDH 5.7.0 native library configuration on Snappy

Guru

Hi

 

Can you try to run the following commands in Hive to see if it also gives you the same error?

 

 

SET hive.exec.compress.output=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET mapred.output.compression.type=BLOCK;

CREATE TABLE test_snappy ( a int);
insert into test_snappy values (1);

 

This is to confirm if Snappy is really available in your CDH version

 

Re: CDH 5.7.0 native library configuration on Snappy

Expert Contributor

Eric,

Thanks for looking into this. I am on CDH 5.7.0. The beeline query failed as my PIG load/store script using snappy.

$ rpm -qa | grep snappy
snappy-1.1.0-1.el6.x86_64
$ hadoop checknative -a | grep snappy
16/10/13 22:16:56 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
16/10/13 22:16:56 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
snappy: true /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop/lib/native/libsnappy.so.1

 

Here is the stacktrace of the mapper for Beeline query.

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"tmp_values_col1":"1"} at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"tmp_values_col1":"1"} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170) ... 8 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:525) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:623) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497) ... 9 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:248) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:570) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:514) ... 16 more Caused by: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65) at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150) at org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131) at org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:99) at org.apache.hadoop.hive.ql.exec.Utilities.createCompressedStream(Utilities.java:1394) at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:260) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:245) ... 18 more

Re: CDH 5.7.0 native library configuration on Snappy

Expert Contributor

Upgraded to 5.7.4, bz2 and gz work now, but snappy still doesn't work.

ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2999: Unexpected internal error. native snappy library not available: this version of libhadoop was built without snappy support.

 

 

Re: CDH 5.7.0 native library configuration on Snappy

Expert Contributor

Eric,

Could you check again? I am using Oralce Enterprise 6.5.