This article is a follow-up on my original article about the visualization of a cluster and its services/components ( https://community.hortonworks.com/articles/2010/visualizing-hdp-cluster-service-allocation.html). In the first part I am particularly focusing on a new feature that enables users to build and plan a cluster by using a drag-n-drop Web UI.
Build a Cluster
Until now visualizing a cluster and its service allocation either meant exporting the information from Ambari or writing a JSON file that outlines the details of the nodes. Planning and deploying a cluster should be easier, right?
I'll introduce, Build a Cluster 🙂
This simple Web UI is based on different drag-n-drop functionalities and allows the creation of a new cluster by simply dragging Hadoop components from the elements list to the indivdual nodes.
Lets go over the different features....
User Interface
The UI is divided into two sections Elements & Settings and Cluster:
- Elements & Settings (left): Contain the available services and components of the environment (remember these can be edited by simply importing a different env.) as well as cluster settings (HDP version, cluster name, security enabled yes/no). Additionaly this section provides some action buttons to finalize the cluster and add nodes
- Cluster (right): This is the current cluster with all its components. Elements can be dragged from the elements list and dropped in the individual nodes. Nodes can be edited or removed.
Nodes
Note: The data structure of Nodes has changed in this version, one node does not have to represent a single physical machine anymore, a node in this app can now represent many physical machines that all share the same components.
Adding Nodes
The number of nodes is curently limited to 1000.
New nodes can be added by pressing the "+ Node"-button in the elements section.
Editing the hostname and cardinality
Simply click on the hostname or cardinality. (Note: ever
Hostname Syntax
Hostnames allow some special syntax, which automatically generates multiple hostnames (only if cardinality is set > 1)
#{x} => Number with trailing zeros
{x} => Number without trailing zeros
X => defines the start of the counter
Examples:
1) datanode#{0}.example.com (Cardinality = 2)
- datanode1.example.com
- datanode2.example.com
2) datanode#{0}.example.com (Cardinality = 30)
- datanode01.example.com
- datanode02.example.com
- ...
3) datanode{100}.example.com (Cardinality = 20)
- datanode100.example.com
- datanode101.example.com
- ...
4) datanode.example.com (Cardinality = 2)
- datanode.example.com1
- datanode.example.com2
Adding components to a Node
- Select a service in the Elements list, this will bring up the list of components of this service
- Select & Drag a component to any of the nodes
Removing a component from a Node
Drag the component from the node and drop it outside the node or over the "Trash" area, inside the elements section
Finalize a Cluster
When your cluster is finished, press the "Finalize"-button inside the elements section, this will convert the built cluster into the same data format as any exported or JSON-specified cluster. Additionaly this imports or basically transfers the new cluster to the main page "Cluster". Finalizing a cluster also regenerates the Ambari Blueprint (read more in the next section)
Note: You can press the button multiple times while you're developing a new cluster 🙂 This might be helpful, e.g. if you want to see different views (service, component, list) on the Cluster-page during the development.
Generating Ambari Blueprints
In this section I am focusing on another new feature that will more than simplfy the creation of Ambari Blueprints.
The blueprint section contains the actual blueprint (left) as well as the cluster creation template or hostgroup mapping (right). Hitting the "Copy"-button in the upper-right corner will copy the individual content to the clipboard (might not work with all browsers)
Cluster-Configuration
No blueprint is complete without any configuration! The Cluster-Configuration page provides the necessary functionalities to add general or host-group specific configurations to the blueprint or the cluster creation template (hostgroup mapping).
I have seen plenty of blueprints that had typos within the configuration section, e.g. instead of dfs.blocksize the blueprint included a dfs.blcksize configuration. This is why the typeahead feature for the config location and name was added. Simply start typing and the app will come up with some suggestions.
HDFS HA & Yarn HA configurations (automation)
A nice little gimmick that has been added to this application is the automatic config generation for HDFS HA and Yarn HA clusters. Whenever the app recognizes a specified set of service components (e.g. 2 Namenodes, 3 Journalnodes, etc.) in the cluster, it will automatically generate the necessary configuration for HDFS or Yarn High Availability.
Project & Setup: https://github.com/mr-jstraub/ambari-node-view
I hope you enjoy these new features and find them useful.
Looking forward to your feedback and feature requests 🙂
