@Muhammad Umar Each executor can have 1 or more threads to perform parallel computation. In yarn master mode the default is 1. These threads can be increased by using the command line parameter --executor-cores. a) if I have specified 8 num_executors for an application and I dont set executor-cores so will each executor going to use all the cores ? In yarn master mode the default is 1, therefore each executor will use only 1 core by default. b) As each node has 8 cores, so what if I specify executor_cores = 4 so that means it will limit executor cores to be used for an executor should not exceed 4 while total cores per node are 8 ? Assignment of cores is static, not dynamic. And it will remain same during the execution of the application. If you set executor cores to 4 this means that each executor will start and run using 4 cores/threads to perform parallel computation. c) What is the criteria to specify executor_cores for a spark application? Increasing executor cores affects performance. You need to take in consideration the number of virtual cores available in each node and my recommendation is you should not increase this over 4 in most cases. HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more

@Nikhil Have you checked the Executor tab in Spark UI, does this helps? RM UI also displays the total memory per application. HTH ... View more

@Aditya Sirna I reproduced the problem. Turns out the issue was caused due metastore location uri poitning to one of the nodes only. To change this you need to run: hive --config /etc/hive/conf/conf.server --service metatool -listFSRoot The above will list and you will be able to spot the locations you need to change. Then issue: hive --config /etc/hive/conf/conf.server --service metatool -updateLocation <new-location> <old-location> For example: hive --config /etc/hive/conf/conf.server --service metatool -updateLocation hdfs://c14/apps/spark/warehouse hdfs://c14-node2.squadron-labs.com:8020/apps/spark/warehouse HTH ... View more

Could you try setting nameservice id for the following properties: spark.history.fs.logDirectory=hdfs://<name_service_id>/spark2-history/ spark.eventLog.dir=hdfs://<name_service_id>/spark2-history/ ... View more

@Shantanu Sharma You should not copy the hive-site.xml from hive conf directory for spark. Spark uses a smaller and rather simple hive-site.xml. cat /etc/spark2/conf/hive-site.xml <configuration xmlns:xi="http://www.w3.org/2001/XInclude"> <property> <name>hive.exec.scratchdir</name> <value>/tmp/spark</value> </property> <property> <name>hive.metastore.client.connect.retry.delay</name> <value>5</value> </property> <property> <name>hive.metastore.client.socket.timeout</name> <value>1800</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://hive-trhift-fqdn:9083</value> </property> <property> <name>hive.server2.enable.doAs</name> <value>false</value> </property> <property> <name>hive.server2.thrift.http.port</name> <value>10002</value> </property> <property> <name>hive.server2.thrift.port</name> <value>10016</value> </property> <property> <name>hive.server2.transport.mode</name> <value>binary</value> </property> </configuration> The above is an example, just make sure you change values accordingly to your env (hive.metastore.uris) HTH ... View more

@Michael Bronson Using Spark UI you can go to executor tab and there is a column with GC time. Also, by using the above configurations I shared the gc details will be printed as part of the log ouput. You can review those using any tool like http://gceasy.io/ HTH ... View more