@Nikhil Have you checked the Executor tab in Spark UI, does this helps? RM UI also displays the total memory per application. HTH ... View more

@Aditya Sirna I reproduced the problem. Turns out the issue was caused due metastore location uri poitning to one of the nodes only. To change this you need to run: hive --config /etc/hive/conf/conf.server --service metatool -listFSRoot The above will list and you will be able to spot the locations you need to change. Then issue: hive --config /etc/hive/conf/conf.server --service metatool -updateLocation <new-location> <old-location> For example: hive --config /etc/hive/conf/conf.server --service metatool -updateLocation hdfs://c14/apps/spark/warehouse hdfs://c14-node2.squadron-labs.com:8020/apps/spark/warehouse HTH ... View more

Could you try setting nameservice id for the following properties: spark.history.fs.logDirectory=hdfs://<name_service_id>/spark2-history/ spark.eventLog.dir=hdfs://<name_service_id>/spark2-history/ ... View more

@Shantanu Sharma You should not copy the hive-site.xml from hive conf directory for spark. Spark uses a smaller and rather simple hive-site.xml. cat /etc/spark2/conf/hive-site.xml <configuration xmlns:xi="http://www.w3.org/2001/XInclude"> <property> <name>hive.exec.scratchdir</name> <value>/tmp/spark</value> </property> <property> <name>hive.metastore.client.connect.retry.delay</name> <value>5</value> </property> <property> <name>hive.metastore.client.socket.timeout</name> <value>1800</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://hive-trhift-fqdn:9083</value> </property> <property> <name>hive.server2.enable.doAs</name> <value>false</value> </property> <property> <name>hive.server2.thrift.http.port</name> <value>10002</value> </property> <property> <name>hive.server2.thrift.port</name> <value>10016</value> </property> <property> <name>hive.server2.transport.mode</name> <value>binary</value> </property> </configuration> The above is an example, just make sure you change values accordingly to your env (hive.metastore.uris) HTH ... View more

@Michael Bronson Using Spark UI you can go to executor tab and there is a column with GC time. Also, by using the above configurations I shared the gc details will be printed as part of the log ouput. You can review those using any tool like http://gceasy.io/ HTH ... View more

@Daniel Müller As I thought the PushedFilters are empty. I checked the spark.sql.orc.filterPushdown implementation details and looks like LIMIT is not supported. You can read more by looking at inline comments here: https://github.com/apache/spark/blob/master/sql/hive/src/main/scala/org/apache/spark/sql/hive/orc/OrcFilters.scala HTH *** If you found this answer addressed your question, please take a moment to login and click the "accept" link on the answer. ... View more