Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
avatar
Contributor

Microsoft HDInsights provides the four different options under Cluster Type and they are:

Hadoop, Hbase, Storm and Spark (Preview);

Some of HDP components are not standard in the HDInsight distribution of choice.

In this particular case, the customer was interested in Spark / Spark Notebooks Components and especially Zeppelin.

When choosing the Spark Option, only these services will be provided under Ambari:

HDFS, MR2, YARN, Tez, Hive, Pig, Sqoop, OOzie, Zookeeper, Ambari Metrics, Jupyter, Livy (Livy is the remote job submission server for Spark), and Spark;

Zeppelin is missing from the HDI Spark distro at the time of writing this article;

So in order to install Zeppelin When choosing 'Spark' as the HDI ‘Cluster Type' , and under

'New HDInsights Cluster' >> 'Optional Configuration’ >> 'Scripts Action' put the following URI as shown below:

https://hdiconfigactions.blob.core.windows.net/linuxincubatorzeppelinv01/install-zeppelin-spark151-v...

1592-zeppelin-actionscripts.png

Further services can be added for the Spark Cluster Type through Ambari (Add Service):

Accumulo, Atlas, Mahout, Ranger, Ranger KMS and Slider.

(notice that Hbase is not a default install when choosing the Spark HDI install option, and has to be installed seperatly or through chosing the Hadoop cluster Type instead of Spark)

Here are other 'Script Actions' URLs for installing Spark, R, Solr, and Giraph while installing HDInsight:

Install Spark >> https://hdiconfigactions.blob.core.windows.net/sparkconfigactionv03/spark-installer-v03.ps1.

Install R >> https://hdiconfigactions.blob.core.windows.net/rconfigactionv02/r-installer-v02.ps1.

Install Solr>> https://hdiconfigactions.blob.core.windows.net/solrconfigactionv01/solr-installer-v01.ps1.

Install Giraph >> https://hdiconfigactions.blob.core.windows.net/giraphconfigactionv01/giraph-installer-v01.ps1.

3,262 Views
Comments

@Alexander Do we have similar script to install Hbase on Spark HD Insights cluster ?

@Alexander is there a full list of these hdi scripts available? If not how did you discover the ones above?

Version history
Last update:
‎09-16-2022 01:34 AM
Updated by:
Contributors