Microsoft HDInsights provides the four different options under Cluster Type and they are:
Hadoop, Hbase, Storm and Spark (Preview);
Some of HDP components are not standard in the HDInsight distribution of choice.
In this particular case, the customer was interested in Spark / Spark Notebooks Components and especially Zeppelin.
When choosing the Spark Option, only these services will be provided under Ambari:
HDFS, MR2, YARN, Tez, Hive, Pig, Sqoop, OOzie, Zookeeper, Ambari Metrics, Jupyter, Livy (Livy is the remote job submission server for Spark), and Spark;
Zeppelin is missing from the HDI Spark distro at the time of writing this article;
So in order to install Zeppelin When choosing 'Spark' as the HDI ‘Cluster Type' , and under 'New HDInsights Cluster' >> 'Optional Configuration’ >> 'Scripts Action' put the following URI as shown below:
https://hdiconfigactions.blob.core.windows.net/linuxincubatorzeppelinv01/install-zeppelin-spark151-v01.sh
Further services can be added for the Spark Cluster Type through Ambari (Add Service):
Accumulo, Atlas, Mahout, Ranger, Ranger KMS and Slider.
(notice that Hbase is not a default install when choosing the Spark HDI install option, and has to be installed seperatly or through chosing the Hadoop cluster Type instead of Spark)
Here are other 'Script Actions' URLs for installing Spark, R, Solr, and Giraph while installing HDInsight:
Install Spark >> https://hdiconfigactions.blob.core.windows.net/sparkconfigactionv03/spark-installer-v03.ps1.
Install R >> https://hdiconfigactions.blob.core.windows.net/rconfigactionv02/r-installer-v02.ps1.
Install Solr>> https://hdiconfigactions.blob.core.windows.net/solrconfigactionv01/solr-installer-v01.ps1.
Install Giraph >> https://hdiconfigactions.blob.core.windows.net/giraphconfigactionv01/giraph-installer-v01.ps1.
... View more
