question Re: Spark saveAsTextFile with SnappyCodec on YARN getting : "native snappy library not available: this version of libhadoop was built without snappy support." in Archives of Support Questions (Read Only)

Spark saveAsTextFile with SnappyCodec on YARN getting : "native snappy library not available: this version of libhadoop was built without snappy support."

David_Tam — Mon, 19 Aug 2019 10:00:43 GMT

Hello,

I have a spark job that is run by spark-submit and at the end it saves some output using SparkContext saveAsTextFile with SnappyCodec. To test I am using sandbox 2.3.4

This is running fine using master = local[x], after following the suggestion in this thread. However Once I changed master = yarn-client I am getting the same "native snappy library not available: this version of libhadoop was built without snappy support." on YARN.

However I thought I have done all the necessary setup - any suggestions welcome!

When I check the Spark History server I can see the following in the environment:

Spark properties:

system properties:

Furthermore in Ambari -> Yarn -> config -> Advanced yarn-env -> yarn-env template -> LD_LIBRARY_PATH:

Anything else I could do to make snappy available as a compression codec on YARN?

Thank you.

Re: Spark saveAsTextFile with SnappyCodec on YARN getting : "native snappy library not available: this version of libhadoop was built without snappy support."

rajkumar_singh — Wed, 08 Jun 2016 16:08:15 GMT

can you check whether snappy lib is installed on the cluster nodes using command 'hadoop checknative'?

Re: Spark saveAsTextFile with SnappyCodec on YARN getting : "native snappy library not available: this version of libhadoop was built without snappy support."

David_Tam — Mon, 19 Aug 2019 10:00:19 GMT

yes it is - as I said I was able to save in local mode.

Re: Spark saveAsTextFile with SnappyCodec on YARN getting : "native snappy library not available: this version of libhadoop was built without snappy support."

rajkumar_singh — Wed, 08 Jun 2016 16:21:58 GMT

can you add the following parameters in spark conf and see if it works

  spark.executor.extraClassPath /usr/hdp/current/hadoop-client/lib/snappy-java-*.jar 
  spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/native 
  spark.executor.extraJavaOptions -Djava.library.path=/usr/hdp/current/hadoop-client/lib/native/lib

Re: Spark saveAsTextFile with SnappyCodec on YARN getting : "native snappy library not available: this version of libhadoop was built without snappy support."

David_Tam — Mon, 19 Aug 2019 10:00:11 GMT

Thanks @Rajkumar Singh I have tried the mapred.child.java.opts and a few of the mapreduce settings suggested by @Jitendra Yadav but at the end just adding this spark.executor.extraLibraryPath would do the job.

Re: Spark saveAsTextFile with SnappyCodec on YARN getting : "native snappy library not available: this version of libhadoop was built without snappy support."

imartinez — Fri, 07 Dec 2018 20:24:35 GMT

spark.executor.extraLibraryPath did the job. Thanks! @David Tam and @Jitendra Yadav