Reply
Expert Contributor
Posts: 68
Registered: ‎10-04-2016

CDH 5.7.0 native library configuration on Snappy

I got the following error when runing a MR job on CDH 5.7.0. However, the same job doesn't fail on quickstart 5.8 VM. Is there a known bug for 5.7.0 and fixed in 5.7.2?(MAPREDUCE-6577)

 

I ran command "hadoop checknative -a | grep snapp" and it seems to be installed.

 snappy:  true /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop/lib/native/libsnappy.so.1

 

Here is the stack trace.

2016-10-11 03:17:05,622 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support.

                    at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65)

                    at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134)

                    at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150)

                    at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:165)

                    at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:114)

                    at org.apache.hadoop.mapred.IFile$Writer.<init>(IFile.java:97)

                    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1606)

                    at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1486)

                    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:460)

                    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)

                    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)

                    at java.security.AccessController.doPrivileged(Native Method)

                    at javax.security.auth.Subject.doAs(Subject.java:415)

                    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)

                    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

Cloudera Employee
Posts: 770
Registered: ‎03-23-2015

Re: CDH 5.7.0 native library configuration on Snappy

Hi

 

Can you try to run the following commands in Hive to see if it also gives you the same error?

 

 

SET hive.exec.compress.output=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
SET mapred.output.compression.type=BLOCK;

CREATE TABLE test_snappy ( a int);
insert into test_snappy values (1);

 

This is to confirm if Snappy is really available in your CDH version

 

Expert Contributor
Posts: 68
Registered: ‎10-04-2016

Re: CDH 5.7.0 native library configuration on Snappy

Eric,

Thanks for looking into this. I am on CDH 5.7.0. The beeline query failed as my PIG load/store script using snappy.

$ rpm -qa | grep snappy
snappy-1.1.0-1.el6.x86_64
$ hadoop checknative -a | grep snappy
16/10/13 22:16:56 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
16/10/13 22:16:56 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
snappy: true /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/hadoop/lib/native/libsnappy.so.1

 

Here is the stacktrace of the mapper for Beeline query.

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"tmp_values_col1":"1"} at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:179) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"tmp_values_col1":"1"} at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:507) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:170) ... 8 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:525) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:623) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95) at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:497) ... 9 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:248) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:570) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:514) ... 16 more Caused by: java.lang.RuntimeException: native snappy library not available: this version of libhadoop was built without snappy support. at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:65) at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:134) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:150) at org.apache.hadoop.io.compress.CompressionCodec$Util.createOutputStreamWithCodecPool(CompressionCodec.java:131) at org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:99) at org.apache.hadoop.hive.ql.exec.Utilities.createCompressedStream(Utilities.java:1394) at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat.getHiveRecordWriter(HiveIgnoreKeyTextOutputFormat.java:80) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getRecordWriter(HiveFileFormatUtils.java:260) at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:245) ... 18 more

Expert Contributor
Posts: 68
Registered: ‎10-04-2016

Re: CDH 5.7.0 native library configuration on Snappy

Upgraded to 5.7.4, bz2 and gz work now, but snappy still doesn't work.

ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2999: Unexpected internal error. native snappy library not available: this version of libhadoop was built without snappy support.

 

 

Expert Contributor
Posts: 68
Registered: ‎10-04-2016

Re: CDH 5.7.0 native library configuration on Snappy

Eric,

Could you check again? I am using Oralce Enterprise 6.5.

Announcements