Ambari is a great tool to manage the Hortonworks Data Platform. However, as the complexity of the applications that run on HDP grows and more and more components from the stack are required, it may be necessary to begin automating tasks like configuration changes. Using the Ambari REST interface it is possible to install, control, interrogate, and even change configuration of HDP services.
This example demonstrates how to automate the install and configuration of Nifi. However, the same approach can be applied to any service. This example uses curl to make all REST API request.
Make sure that the bits required to actually install the service in Ambari are available on the host running the Ambari server. In the case of Nifi, the bits should be present in /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/NIFI.
The service is merely the container that holds all of the process that comprise it. When a request to start the service is issued, it actually attempts to start each of the components defined for that service, which are the actual processes that provide functionality.
4. Configure the Components:
This is the tricky part. One of the great values of Ambari is that it provides the ability for administrators to add and change configuration for HDP services from a single place, This means that when a service is being installed, the configurations and the files they are contained in must be defined as well. Each component of each service will have its own configurations files each file will have a unique set of properties and formats. This means that each configuration file that the Ambari wrapped service expects must be defined and applied to each component of the target service prior to install. The simplest way to obtain the required configuration files is to install the service via the Ambari UI and then use the Ambari Config utility located at: /var/lib/ambari-server/resources/scripts/configs.sh. This utility will use the REST API to pull the configurations that Ambari requires from and existing service. In the case of an already installed NIFI service it is possible to to get the required configuration files as follows:
In Ambari UI, click on the NIFI service and then navigate to the configuration section
Take note of each of the configuration sections listed.
Use the Ambari Configs utility to export each of the configuration sections to file as follows: (not shown below... script takes a username and password parameter, this should be the Ambari admin user)
/var/lib/ambari-server/resources/scripts/configs.sh get sandbox.hortonworks.com Sandbox nifi-ambari-config >> nifi-ambari-config.json
/var/lib/ambari-server/resources/scripts/configs.sh get sandbox.hortonworks.com Sandbox nifi-bootstrap-env >> nifi-bootstrap-env.json
/var/lib/ambari-server/resources/scripts/configs.sh get sandbox.hortonworks.com Sandbox nifi-flow-env /root/CreditCardTransactionMonitor/Nifi/config/nifi-flow-env.json
/var/lib/ambari-server/resources/scripts/configs.sh get sandbox.hortonworks.com Sandbox nifi-logback-env >> nifi-logback-env.json
/var/lib/ambari-server/resources/scripts/configs.sh get sandbox.hortonworks.com Sandbox nifi-properties-env >> nifi-properties-env.json
Make sure to strip out the header that gets added to these files (########## Performing 'GET' on (Site:nifi-ambari-config, Tag:version1461337652473983733)).
These configuration definitions can now be used to complete the automation the NIFI installation as follow:
/var/lib/ambari-server/resources/scripts/configs.sh set sandbox.hortonworks.com Sandbox nifi-ambari-config >> nifi-ambari-config.json
/var/lib/ambari-server/resources/scripts/configs.sh set sandbox.hortonworks.com Sandbox nifi-bootstrap-env >> nifi-bootstrap-env.json
/var/lib/ambari-server/resources/scripts/configs.sh set sandbox.hortonworks.com Sandbox nifi-flow-env >> nifi-flow-env.json
/var/lib/ambari-server/resources/scripts/configs.sh set sandbox.hortonworks.com Sandbox nifi-logback-env nifi-logback-env.json
/var/lib/ambari-server/resources/scripts/configs.sh set sandbox.hortonworks.com Sandbox nifi-properties-env nifi-properties-env.json
5. Add Role to Member Hosts
Since HDP is a distributed platform, most services will be installed across multiple hosts. Each host may host different components or all of the same components. Which component of the service run on which service must be defined prior to install. In this case, we are assigning the NIFI-MASTER role to the same host where Ambari server is running.
This request will return a task id as one of the return parameters. This task may take a while and is asynchronous, thus it is necessary to get a handle on the task, periodically check to see if it has been completed before, and loop/sleep until the task is complete. Check for task status as follows: