Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

devops tools used in Ambari

Solved Go to solution
Highlighted

devops tools used in Ambari

Expert Contributor

Hi All,

With all the DevOps tools available (Docker, swram, kubernetes, Ansible, Puppet, Chef etc.) around configuration management, cluster management, container clustering etc.

I was wondering where does Amabri fits in, how is it developed? What about Ambari blue prints?

Thanks,

Avijeet

1 ACCEPTED SOLUTION

Accepted Solutions

Re: devops tools used in Ambari

@Avijeet Dash

At its core, Ambari is able to perform the same common ops that DevOps are used to with Chef and Puppet. Chef, Puppet, and Ansible are great for small clusters and limited operations, but as soon as you need to do perform complex operations like Kerberize, HA, Stack Upgrade, then those tools are not sufficient; this is where Ambari shines since it has deep knowledge about the cluster and Hadoop in order to perform things like:

  • Rolling Restart (pick any component, like DataNodes, and pick your batch size, sleep time, and failures tolerated)
  • High Availability for NameNode, Hive, Oozie, Resource Manager, etc (wizards to configure these and restart affected services)
  • Config Management (this is where it truly brings value; if a config changes, Ambari can suggest other configs to change, and validate dependent configs)
  • Kerberos Security: create and distribute keytabs, also handle use cases like adding hosts and more services
  • Rolling and Express Upgrade: the orchestration and config changes here alone, plus handling a downgrade, would easily take weeks to develop and test. Ambari has all of the tools to visualize the orchestration, auto-retry, handle failures, skip optional steps, etc.
  • Metrics: dashboards to visualize and analyze metrics, trends
  • Alerts: notifications, SNMP, port checks, liveliness checks, and UI to enable/disable
  • Maintenance Mode: put hosts and services in maintenance mode to suppress alerts and operations from running on certain hosts
  • LDAP: active directory for enterprise clusters
  • Service Auto-Start: automatically restart down services

and more.

Ambari was developed with scale in mind to minimize to support clusters with several thousand nodes. The agents heartbeat to the server with the status of commands and daemons; while the server sends commands down to the agents.

Blueprints is the mechanism to export an existing cluster and re-create it (or several others just like it) with a predefined topology, configs, etc. This is extremely useful in cloud environments.

I hope this answers your question.

View solution in original post

3 REPLIES 3

Re: devops tools used in Ambari

@Avijeet Dash

At its core, Ambari is able to perform the same common ops that DevOps are used to with Chef and Puppet. Chef, Puppet, and Ansible are great for small clusters and limited operations, but as soon as you need to do perform complex operations like Kerberize, HA, Stack Upgrade, then those tools are not sufficient; this is where Ambari shines since it has deep knowledge about the cluster and Hadoop in order to perform things like:

  • Rolling Restart (pick any component, like DataNodes, and pick your batch size, sleep time, and failures tolerated)
  • High Availability for NameNode, Hive, Oozie, Resource Manager, etc (wizards to configure these and restart affected services)
  • Config Management (this is where it truly brings value; if a config changes, Ambari can suggest other configs to change, and validate dependent configs)
  • Kerberos Security: create and distribute keytabs, also handle use cases like adding hosts and more services
  • Rolling and Express Upgrade: the orchestration and config changes here alone, plus handling a downgrade, would easily take weeks to develop and test. Ambari has all of the tools to visualize the orchestration, auto-retry, handle failures, skip optional steps, etc.
  • Metrics: dashboards to visualize and analyze metrics, trends
  • Alerts: notifications, SNMP, port checks, liveliness checks, and UI to enable/disable
  • Maintenance Mode: put hosts and services in maintenance mode to suppress alerts and operations from running on certain hosts
  • LDAP: active directory for enterprise clusters
  • Service Auto-Start: automatically restart down services

and more.

Ambari was developed with scale in mind to minimize to support clusters with several thousand nodes. The agents heartbeat to the server with the status of commands and daemons; while the server sends commands down to the agents.

Blueprints is the mechanism to export an existing cluster and re-create it (or several others just like it) with a predefined topology, configs, etc. This is extremely useful in cloud environments.

I hope this answers your question.

View solution in original post

Highlighted

Re: devops tools used in Ambari

Expert Contributor

Thanks @Alejandro Fernandez

It will be great if you could share what technology/language it uses underline, if any. also custom ambari-services. How are those built.

Highlighted

Re: devops tools used in Ambari

@Avijeet Dash, you may find this presentation useful, http://events.linuxfoundation.org/sites/events/files/slides/ambari_big_data_2017.pdf Custom Services can be added to the stack, as long as they support lifecycle commands like stop, start, restart, status, etc., and can also contain Custom Commands.

Technology-wise, Ambari Server is all Java and supports multiple DB types (Postgres, MySQL, Oracle, Azure). The Agents use Python for running, heartbeating, and executing the scripts for managing the Hadoop Services. The Agents send heartbeat messages to the Server every 4-10 seconds (configurable).

The web UI uses AngularJS and Node.js to responsiveness and auto-refresh, and relies on the RESTful web service for all data.

Don't have an account?
Coming from Hortonworks? Activate your account here