Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to use Ambari REST API to add NiFi service to an existing HDP cluster

How to use Ambari REST API to add NiFi service to an existing HDP cluster

New Contributor

Hi all

Help with the steps to add NiFi service to a an existing HDP cluster using the Ambari REST API.

I have used cloudbreak to deploy a HDP cluster and would like to use Ambari API to automate the installation of NIFI services on the cluster.

Cluster details:

Ambari Version: 2.6.1.3 Base Url: http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.3

HDP Version: 2.6

Stack Repo Id: HDP Stack Repository Version: 2.6.4.0-91


I am able to install NiFi services successfully to the HDP cluster but have errors when I tried to start the services.

Here are my steps:

#install mpack

wget http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.1.0/tars/hdf_ambari_mp/hdf-ambari-m...

ambari-server install-mpack --mpack=hdf-ambari-mpack-3.1.1.0-35.tar.gz --verbose

ambari-server restart


#install Nifi Services
curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X POST https://localhost/ambari/api/v1/clusters/test-cluster/services/NIFI

curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X POST https://localhost/ambari/api/v1/clusters/test-cluster/services/NIFI/components/NIFI_MASTER

curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X POST https://localhost/ambari/api/v1/clusters/test-cluster/hosts/ip-10-4-2-254.eu-west-1.compute.internal...


#as part of the prerequisites I have exported NIFI configurations from an existing HPD cluster. I am using those configs to setup this new NIFI install.

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-ambari-config --file=nifi-ambari-config.json

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-ambari-ssl-config --file=nifi-ambari-ssl-config.json

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-authorizers-env --file=nifi-authorizers-env.json

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-bootstrap-env --file=nifi-bootstrap-env.json

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-bootstrap-notification-services-env --file=nifi-bootstrap-notification-services-env.json

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-env --file=nifi-env.json

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-flow-env --file=nifi-flow-env.json

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-jaas-conf --file=nifi-jaas-conf.json

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-login-identity-providers-env --file=nifi-login-identity-providers-env.json

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-node-logback-env --file=nifi-node-logback-env.json

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-properties --file=nifi-properties.json

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-state-management-env --file=nifi-state-management-env.json

/var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-toolkit-env --file=nifi-toolkit-env.json



#Install Nifi

curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context" :"Install Nifi"}, "Body": {"ServiceInfo": {"maintenance_state" : "OFF", "state": "INSTALLED"}}}' https://localhost/ambari/api/v1/clusters/test-cluster/services/NIFI

curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context" :"Start NIFI"}, "Body": {"ServiceInfo": {"maintenance_state" : "OFF", "state": "STARTED"}}}' https://localhost/ambari/api/v1/clusters/test-cluster/services/NIFI

The install is successful but the last command to start the nifi service gives the following error on Ambari.

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 231, in <module>
    Master().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 152, in start
    self.configure(env, is_starting = True)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 120, in locking_configure
    original_configure(obj, *args, **kw)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 101, in configure
    if not sudo.path_isfile(params.nifi_flow_config_dir+'/flow.xml.gz') and nifi_toolkit_util.existing_cluster(params):
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util.py", line 314, in existing_cluster
    zk_command_location = os.path.join(params.zk_root, params.zk_stack_version_buildnum, "zookeeper", "bin", "zkCli.sh")
  File "/usr/lib64/python2.7/posixpath.py", line 75, in join
    if b.startswith('/'):
AttributeError: 'NoneType' object has no attribute 'startswith'

Has anyone else seen this issue, or have a suggestion on how to get this to work?

Many thanks in advance.

TL

2 REPLIES 2
Highlighted

Re: How to use Ambari REST API to add NiFi service to an existing HDP cluster

@Tok Luo

If you do not stick to this two-phased solution, another approach you can take is to install HDF with Cloudbreak 2.5 itself.

Here is a tutorial as well: https://community.hortonworks.com/articles/182221/create-a-nifi-cluster-on-aws-azure-google-or-opens...

Hope this helps!

Highlighted

Re: How to use Ambari REST API to add NiFi service to an existing HDP cluster

Expert Contributor

Did you ever find a solution?

Don't have an account?
Coming from Hortonworks? Activate your account here