Hi ,   Please share the Download Link for Sandbox image of HDP 2.6.4 VMware?   Thanks & Regards, Neha Jain ... View more

Hi,   We are currently using Spark 2.2.0 and we are trying to write 4GB data to Hive using below command: dataFrame.write.option("compression", "snappy").format("parquet").mode(SaveMode.Append).saveAsTable(dbName.tableName).   Configuration used to run the job from Spark Submit: Executor Memory = 15g Driver Memory = 10g Num-executors = 3 Number of cores = 3 spark.yarn.executor.memoryOverhead=8192 shuffle partitions = 30   Our job gets stucked for hours during data write to Hive which is strange as the two node cluster being used for testing is of huge capacity and has lot of available space. The job doesnot do much shuffling. Also, we do not see any memory spikes during execution of job. What could be the possible reason for the same? Any help will be appreciated.   ... View more

We are facing certain challenges in sorting of data on dataframes in Spark 1.6 . We are using df. orderBy(userColumn, rankColumn). The sorting of data is proper when the dataframe data is in one partition. As soon as the partition size increases , the dataframe sorting is not working on clustered environment. We tried Distribute by and sort by approach as well as per the below post: http://saurzcode.in/2015/01/hive-sort-vs-order-vs-distribute-vs-cluster/. This is also not working. Please suggest. ... View more

I am facing some issues while running Spark(1.6) jobs in Yarn cluster mode with below configurations: --master yarn --deploy-mode cluster --executor-cores 8 --num-executors 3 --executor-memory 25G --driver-memory 6g --conf spark.network.timeout=10000000 --conf spark.cores.max=35 --conf spark.memory.fraction=0.6 --conf spark.memory.storageFraction=0.5 --conf spark.shuffle.memoryFraction=1 Also, I am giving spark.sql.shuffle.partitions=30 in spark config.xml. I am running the job with above command on a three node cluster setup of hortonworks where each node has around 51GB of memory available. The input data records is approx 254 million. The job crashes when inserting data to Hive with Executor Lost issue and Exit code as 143. There is very high shuffling of data during processing. Can you please suggest what can be done to resolve this issue? Also, how can we determine based on input size , the memory parameters to be used for running the job? ... View more