Working with Ambari REST API - Automate NIFI Install and Config on Ambari

[#1]

Ambari is a great tool to manage the Hortonworks Data Platform. However, as the complexity of the applications that run on HDP grows and more and more components from the stack are required, it may be necessary to begin automating tasks like configuration changes. Using the Ambari REST interface it is possible to install, control, interrogate, and even change configuration of HDP services.

This example demonstrates how to automate the install and configuration of Nifi. However, the same approach can be applied to any service. This example uses curl to make all REST API request.

Make sure that the bits required to actually install the service in Ambari are available on the host running the Ambari server. In the case of Nifi, the bits should be present in /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/NIFI.

1. Check Service Status:

curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/services/NIFI

Use regex of some JSON parsing library/tool to parse the response. In this particular case you want to get a status 404 as that means that the NIFI service does not yet exist.

2. Create Service:

curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/services/NIFI

A check of Ambari UI should confirm that the NIFI service is visible on the left hand pane.

3. Add Components to Service

curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/services/NIFI/components/NIFI_MASTER

The service is merely the container that holds all of the process that comprise it. When a request to start the service is issued, it actually attempts to start each of the components defined for that service, which are the actual processes that provide functionality.

4. Configure the Components:

This is the tricky part. One of the great values of Ambari is that it provides the ability for administrators to add and change configuration for HDP services from a single place, This means that when a service is being installed, the configurations and the files they are contained in must be defined as well. Each component of each service will have its own configurations files each file will have a unique set of properties and formats. This means that each configuration file that the Ambari wrapped service expects must be defined and applied to each component of the target service prior to install. The simplest way to obtain the required configuration files is to install the service via the Ambari UI and then use the Ambari Config utility located at: /var/lib/ambari-server/resources/scripts/configs.sh. This utility will use the REST API to pull the configurations that Ambari requires from and existing service. In the case of an already installed NIFI service it is possible to to get the required configuration files as follows:

• In Ambari UI, click on the NIFI service and then navigate to the configuration section
• Take note of each of the configuration sections listed.
• Use the Ambari Configs utility to export each of the configuration sections to file as follows: (not shown below... script takes a username and password parameter, this should be the Ambari admin user)

/var/lib/ambari-server/resources/scripts/configs.sh get sandbox.hortonworks.com Sandbox nifi-ambari-config >> nifi-ambari-config.json

/var/lib/ambari-server/resources/scripts/configs.sh get sandbox.hortonworks.com Sandbox nifi-bootstrap-env >> nifi-bootstrap-env.json

/var/lib/ambari-server/resources/scripts/configs.sh get sandbox.hortonworks.com Sandbox nifi-flow-env /root/CreditCardTransactionMonitor/Nifi/config/nifi-flow-env.json

/var/lib/ambari-server/resources/scripts/configs.sh get sandbox.hortonworks.com Sandbox nifi-logback-env >> nifi-logback-env.json

/var/lib/ambari-server/resources/scripts/configs.sh get sandbox.hortonworks.com Sandbox nifi-properties-env >> nifi-properties-env.json  

Make sure to strip out the header that gets added to these files (########## Performing 'GET' on (Site:nifi-ambari-config, Tag:version1461337652473983733)).

These configuration definitions can now be used to complete the automation the NIFI installation as follow:

/var/lib/ambari-server/resources/scripts/configs.sh set sandbox.hortonworks.com Sandbox nifi-ambari-config >> nifi-ambari-config.json

/var/lib/ambari-server/resources/scripts/configs.sh set sandbox.hortonworks.com Sandbox nifi-bootstrap-env >> nifi-bootstrap-env.json

/var/lib/ambari-server/resources/scripts/configs.sh set sandbox.hortonworks.com Sandbox nifi-flow-env >> nifi-flow-env.json

/var/lib/ambari-server/resources/scripts/configs.sh set sandbox.hortonworks.com Sandbox nifi-logback-env nifi-logback-env.json

/var/lib/ambari-server/resources/scripts/configs.sh set sandbox.hortonworks.com Sandbox nifi-properties-env nifi-properties-env.json 

5. Add Role to Member Hosts

Since HDP is a distributed platform, most services will be installed across multiple hosts. Each host may host different components or all of the same components. Which component of the service run on which service must be defined prior to install. In this case, we are assigning the NIFI-MASTER role to the same host where Ambari server is running.

curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/hosts/sandbox.hortonworks.com/host_compo...

6. The NIFI service is now ready to be installed

curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context" :"Install Nifi"}, "Body": {"ServiceInfo": {"maintenance_state" : "OFF", "state": "INSTALLED"}}}' http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/services/NIFI

This request will return a task id as one of the return parameters. This task may take a while and is asynchronous, thus it is necessary to get a handle on the task, periodically check to see if it has been completed before, and loop/sleep until the task is complete. Check for task status as follows:

curl -u admin:admin -X GET http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/requests/$TASKID

Once the task comes back as COMPLETE, the NIFI service has been installed and is ready to be started.

7. Start the Service

curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context" :"Start NIFI"}, "Body": {"ServiceInfo": {"maintenance_state" : "OFF", "state": "STARTED"}}}' http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/services/NIFI

The NIFI service is now up and running and ready to build data flows. This same approach can be used to install, stop, start, and change configuration for any service in Ambari.

[#2]

Step 1 : Check Service Status: should use get request

curl -u admin:admin -H "X-Requested-By:ambari"-i -X GET http://sandbox.hortonworks.com:8080/api/v1/clusters/Sandbox/services/NIFI