Created on 06-08-2016 08:59 AM - edited 08-19-2019 03:00 AM
I have a spark job that is run by spark-submit and at the end it saves some output using SparkContext saveAsTextFile with SnappyCodec. To test I am using sandbox 2.3.4
This is running fine using master = local[x], after following the suggestion in this thread. However Once I changed master = yarn-client I am getting the same "native snappy library not available: this version of libhadoop was built without snappy support." on YARN.
However I thought I have done all the necessary setup - any suggestions welcome!
When I check the Spark History server I can see the following in the environment:
Furthermore in Ambari -> Yarn -> config -> Advanced yarn-env -> yarn-env template -> LD_LIBRARY_PATH:
Anything else I could do to make snappy available as a compression codec on YARN?
can you add the following parameters in spark conf and see if it works
spark.executor.extraClassPath /usr/hdp/current/hadoop-client/lib/snappy-java-*.jar spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/native spark.executor.extraJavaOptions -Djava.library.path=/usr/hdp/current/hadoop-client/lib/native/lib