https://community.cloudera.com/t5/Support-Questions/Spark-Sql-for-ETL-performance-tuning/m-p/238617#M200428 <P><A rel="user" href="https://community.cloudera.com/users/112382/barath51777.html" nodeid="112382">@Barath Natarajan</A></P><P>Check out how many executors and memory that <STRONG><U>spark-sql cli</U></STRONG> has been initialized(it seems to be running on local mode with one executor).</P><OL><LI>To debug the query run an <STRONG>explain plan</STRONG> on the query.</LI><LI>Check out how <STRONG>many files</STRONG> in hdfs directory for each table, if too many files then consolidate them to smaller number.</LI></OL><P><STRONG><U><EM>Another approach would be:</EM></U></STRONG><BR />-&gt; Run <STRONG>spark-shell (or) pys</STRONG><STRONG>park</STRONG> with <STRONG>local mode/yarn-clie</STRONG>nt mode with more number of executors/more memory<BR />-&gt; Then load the tables into <STRONG>dataframe</STRONG> and then <STRONG>registerTempTable(spark1.X)/createOrReplaceTempView(if using spark2)</STRONG><BR />-&gt; Run your join using spark.sql("&lt;join query&gt;")<BR />-&gt; Check out the performance of the query.</P> Fri, 19 Apr 2019 08:56:41 GMT Shu_ashu 2019-04-19T08:56:41Z