Hi Srowen, Thanks for the update, I am new to spark, can you please guide where should I set "--master yarn-client" in configuration file or ?   thanks Pal ... View more

Hi Sowen,   Below are the steps I launched “pyspark” command shell by executing “/opt/cloudera/parcels/CDH/bin/pyspark” In pyspark command I executed simple python “sc.parallelize(range(1000)).count()” , this simple one line program successfully in pyspark command shell. Since Apache Spark in CDH5.2 is configured to run on YARN , I was expecting to see the pyspark app in spark history server http://localhost:18088   Since its single node installation master and worker are in same node   Please advice if I am running this simple python program incorrectly   thanks Pal ... View more

Hi Sowen, Thanks for your reply,   Are you running Spark on YARN, or using Spark standalone? if the latter, you won't see any YARN history since it's not using YARN.   Yes Spark is running on YARN (MR2 Included) I checked this in Cloudera Manager Web console Spark à Configuration   Is that mean I have to configure Spark to user YARN (MR2 Included) according in Cloudera Manager its already OR am I missing something   I did a default installation and followed wizard.   Can you please advice   thanks Pal ... View more