@akeezhadath spark assume the your file is on hdfs by default if you have not specified any uri(file:///,hdfs://,s3://) so it your file is on hdfs, you can refrenced it using absolute path like sc.textFile("/user/xyz/data.txt")
... View more
can you add the following parameters in spark conf and see if it works spark.executor.extraClassPath /usr/hdp/current/hadoop-client/lib/snappy-java-*.jar
spark.executor.extraLibraryPath /usr/hdp/current/hadoop-client/native
spark.executor.extraJavaOptions -Djava.library.path=/usr/hdp/current/hadoop-client/lib/native/lib
... View more
@Roberto Sancho it seems that all your queries has been answered, could you please spare some time and accept a best answer in this thread.
... View more
@Roberto Sancho it seems that all your queries has been answered, could you please spare some time and accept a best answer in this thread.
... View more
could you please modify your program in this way and see if you still see any excepton from pyspark import SparkConf, SparkContext
from pyspark.sql import HiveContext
sc = SparkContext()
sqlContext = HiveContext(sc)
sqlContext.sql("select * from default.aaa limit 3").write.format("orc").save("test_orc2")
... View more
@alain TSAFACK please use saveAsHadoopFile while will write to hdfs saveAsHadoopFile(<file-name>,
<file output format>",
compressionCodecClass="org.apache.hadoop.io.compress.GzipCodec") or hive_context.write.format("orc").save("test_orc")
... View more
