Community Articles

Find and share helpful community-sourced technical articles.
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.
Labels (2)

Ambari is a great tool to manage the Hortonworks Data Platform. However, as the complexity of the applications that run on HDP grows and more and more components from the stack are required, it may be necessary to begin automating tasks like configuration changes. Using the Ambari REST interface it is possible to install, control, interrogate, and even change configuration of HDP services.

This example demonstrates how to automate the install and configuration of Nifi. However, the same approach can be applied to any service. This example uses curl to make all REST API request.

Make sure that the bits required to actually install the service in Ambari are available on the host running the Ambari server. In the case of Nifi, the bits should be present in /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/NIFI.

1. Check Service Status:

curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST

Use regex of some JSON parsing library/tool to parse the response. In this particular case you want to get a status 404 as that means that the NIFI service does not yet exist.

2. Create Service:

curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST

A check of Ambari UI should confirm that the NIFI service is visible on the left hand pane.

3. Add Components to Service

curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST

The service is merely the container that holds all of the process that comprise it. When a request to start the service is issued, it actually attempts to start each of the components defined for that service, which are the actual processes that provide functionality.

4. Configure the Components:

This is the tricky part. One of the great values of Ambari is that it provides the ability for administrators to add and change configuration for HDP services from a single place, This means that when a service is being installed, the configurations and the files they are contained in must be defined as well. Each component of each service will have its own configurations files each file will have a unique set of properties and formats. This means that each configuration file that the Ambari wrapped service expects must be defined and applied to each component of the target service prior to install. The simplest way to obtain the required configuration files is to install the service via the Ambari UI and then use the Ambari Config utility located at: /var/lib/ambari-server/resources/scripts/ This utility will use the REST API to pull the configurations that Ambari requires from and existing service. In the case of an already installed NIFI service it is possible to to get the required configuration files as follows:

  • In Ambari UI, click on the NIFI service and then navigate to the configuration section
  • Take note of each of the configuration sections listed.
  • Use the Ambari Configs utility to export each of the configuration sections to file as follows: (not shown below... script takes a username and password parameter, this should be the Ambari admin user)
/var/lib/ambari-server/resources/scripts/ get Sandbox nifi-ambari-config >> nifi-ambari-config.json

/var/lib/ambari-server/resources/scripts/ get Sandbox nifi-bootstrap-env >> nifi-bootstrap-env.json

/var/lib/ambari-server/resources/scripts/ get Sandbox nifi-flow-env /root/CreditCardTransactionMonitor/Nifi/config/nifi-flow-env.json

/var/lib/ambari-server/resources/scripts/ get Sandbox nifi-logback-env >> nifi-logback-env.json

/var/lib/ambari-server/resources/scripts/ get Sandbox nifi-properties-env >> nifi-properties-env.json  

Make sure to strip out the header that gets added to these files (########## Performing 'GET' on (Site:nifi-ambari-config, Tag:version1461337652473983733)).

These configuration definitions can now be used to complete the automation the NIFI installation as follow:

/var/lib/ambari-server/resources/scripts/ set Sandbox nifi-ambari-config >> nifi-ambari-config.json

/var/lib/ambari-server/resources/scripts/ set Sandbox nifi-bootstrap-env >> nifi-bootstrap-env.json

/var/lib/ambari-server/resources/scripts/ set Sandbox nifi-flow-env >> nifi-flow-env.json

/var/lib/ambari-server/resources/scripts/ set Sandbox nifi-logback-env nifi-logback-env.json

/var/lib/ambari-server/resources/scripts/ set Sandbox nifi-properties-env nifi-properties-env.json 

5. Add Role to Member Hosts

Since HDP is a distributed platform, most services will be installed across multiple hosts. Each host may host different components or all of the same components. Which component of the service run on which service must be defined prior to install. In this case, we are assigning the NIFI-MASTER role to the same host where Ambari server is running.

curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST

6. The NIFI service is now ready to be installed

curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context" :"Install Nifi"}, "Body": {"ServiceInfo": {"maintenance_state" : "OFF", "state": "INSTALLED"}}}'

This request will return a task id as one of the return parameters. This task may take a while and is asynchronous, thus it is necessary to get a handle on the task, periodically check to see if it has been completed before, and loop/sleep until the task is complete. Check for task status as follows:

curl -u admin:admin -X GET$TASKID

Once the task comes back as COMPLETE, the NIFI service has been installed and is ready to be started.

7. Start the Service

curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context" :"Start NIFI"}, "Body": {"ServiceInfo": {"maintenance_state" : "OFF", "state": "STARTED"}}}'

The NIFI service is now up and running and ready to build data flows. This same approach can be used to install, stop, start, and change configuration for any service in Ambari.


Step 1 : Check Service Status: should use get request

curl -u admin:admin -H "X-Requested-By:ambari"-i -X GET
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.
Version history
Last update:
‎04-29-2016 01:03 AM
Updated by:
Top Kudoed Authors