Those scripts, master and slave, are for running a Standalone Spark cluster.  They are not needed if you are running Spark in YARN.  The preference is to have Spark running in YARN so you don't have to divy up the cluster resources.    The history script is for the Spark History server.  You can start and stop that through CM.   My best guess re: the UI not showing up when you start the slave is that the master was no longer running.  Were you able to launch the spark-shell afterwards but not access the UI? ... View more

Hi Shilpa,   I am assuming that you selected the Spark service from the base CDH 5.9 parcels/packages and didn't fetch is separate and therefor are running a standalone Spark cluster alongside the Hadoop cluster.   In any case, the Spark Gateway, as well as the other Gateway's (HDFS, YARN, HBase, etc.), only install the libraries, binaries, and configuration files required to use the command line tools.  In this case it is spark-submit and spark-shell (maybe the pyspark files as well, not 100% sure on it).  There is not service to start and stop or monitor.  That is why you can't start it and why it is gray instead of green.   In the case of the above assumption, it automatically sets up the tools to run in YARN mode.  This means that the spark application runs in YARN instead of a standalone Spark cluster.  Find the ResourceManager UI.  You will find the application listed there and can access the AM and worker logs.  You will be direct to the Spark History server UI (so you could go directly here if you wanted but it is good to get used to the RM interface). ... View more

Yes. Go through your process. It is granting more accessible which is generally less risky. Also, it is the correct way to install Hadoop/CDH. https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_sg_cm_users_principals.html ... View more