No worries @PR_224  Glad it's fixed : ) ... View more

That was the solution for me. I just forgot to run this "connecting" command after system restart    sudo /opt/cloudera/cm/schema/scm_prepare_database.sh postgresql scm scm   ...Dummy in Linux and Cloudera ... View more

My original issue of heap space is now fixed , seems my driver memory was not optimum . Setting driver memory from pyspark client does not take effect as container is already created by that time , thus I had to set it at spark environmerent properties in CDH manager console. To set that I went to Cloudera Manager > Spark > Configuration > Gateway > Advanced > in Spark Client Advanced Configuration Snippet (Safety Valve) for spark-conf/spark-defaults.conf I added spark.driver.memory=10g and Java heap issue was solved . I think this will work when you're running your spark application on Yarn-Client.   However after spark job is finished the application hangs on toPandas , does anyone has any idea what specific properties need to set for conversion of dataframe toPandas ?   -Rahul ... View more