@saivenkatg55Can you please  share the hdfs-site.xml ... View more

Please check if this file exists /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid . If not create the directory   mkdir /var/run/hadoop-yarn/yarn/ chown -R yarn:hadoop /var/run/hadoop-yarn/yarn/ touch hadoop-yarn-nodemanager.pid chown yarn:hadoop /var/run/hadoop-yarn/yarn/hadoop-yarn-nodemanager.pid   This will work.     ... View more

You need to follow these as those are for spark thrift Configuring Cluster Dynamic Resource Allocation Manually To configure a cluster to run Spark jobs with dynamic resource allocation, complete the following steps: Add the following properties to the spark-defaults.conf file associated with your Spark installation (typically in the $SPARK_HOME/conf directory): Set spark.dynamicAllocation.enabled to true . Set spark.shuffle.service.enabled to true . (Optional) To specify a starting point and range for the number of executors, use the following properties: spark.dynamicAllocation.initialExecutors spark.dynamicAllocation.minExecutors spark.dynamicAllocation.maxExecutors Note that initialExecutors must be greater than or equal to minExecutors , and less than or equal to maxExecutors . For a description of each property, see Dynamic Resource Allocation Properties. Start the shuffle service on each worker node in the cluster: In the yarn-site.xml file on each node, add spark_shuffle to yarn.nodemanager.aux-services , and then set yarn.nodemanager.aux-services.spark_shuffle.class to org.apache.spark.network.yarn.YarnShuffleService . Review and, if necessary, edit spark.shuffle.service.* configuration settings. For more information, see the Apache Spark Shuffle Behavior documentation. Restart all NodeManagers in your cluster. ... View more

Have you followed these steps before adding nodelabels to yarn cluster ? https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.5/bk_yarn-resource-management/content/configuring_node_labels.html ... View more

Follow this https://github.com/ehiggs/spark-terasort ... View more

For this you need to use the spark dynamic allocation. Dynamic Allocation (of Executors) (aka Elastic Scaling) is a Spark feature that allows for adding or removing Spark executors dynamically to match the workload. Unlike the "traditional" static allocation where a Spark application reserves CPU and memory resources upfront (irrespective of how much it may eventually use), in dynamic allocation you get as much as needed and no more. It scales the number of executors up and down based on workload, i.e. idle executors are removed, and when there are pending tasks waiting for executors to be launched on, dynamic allocation requests them ... View more

In /etc/yum.repos.d, remove all .repo files pointing to the Internet and copy only .repo files from other servers which are already using your local repo. For HDP nodes, initially you need only 2 .repo files, one for the OS, and ambari.repo. When Ambari adds a new node to the cluster it will copy there HDP.repo and HDP-UTILS.repo. Also, have you set your repository URLs in Ambari-> Admin-> Stack and versions-> Versions -> Manage Versions -> [click on your current version] ? ... View more

You need to have a look at user limit factor for the queue and min max capacity https://community.hortonworks.com/content/supportkb/49640/what-does-the-user-limit-factor-do-when-used-in-ya.html https://hortonworks.com/blog/yarn-capacity-scheduler/ ... View more