Created 04-05-2018 04:34 PM
Hi all
Help with the steps to add NiFi service to a an existing HDP cluster using the Ambari REST API.
I have used cloudbreak to deploy a HDP cluster and would like to use Ambari API to automate the installation of NIFI services on the cluster.
Cluster details:
Ambari Version: 2.6.1.3 Base Url: http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.3
HDP Version: 2.6
I am able to install NiFi services successfully to the HDP cluster but have errors when I tried to start the services.
Here are my steps:
#install mpack wget http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.1.1.0/tars/hdf_ambari_mp/hdf-ambari-m... ambari-server install-mpack --mpack=hdf-ambari-mpack-3.1.1.0-35.tar.gz --verbose ambari-server restart #install Nifi Services curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X POST https://localhost/ambari/api/v1/clusters/test-cluster/services/NIFI curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X POST https://localhost/ambari/api/v1/clusters/test-cluster/services/NIFI/components/NIFI_MASTER curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X POST https://localhost/ambari/api/v1/clusters/test-cluster/hosts/ip-10-4-2-254.eu-west-1.compute.internal... #as part of the prerequisites I have exported NIFI configurations from an existing HPD cluster. I am using those configs to setup this new NIFI install. /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-ambari-config --file=nifi-ambari-config.json /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-ambari-ssl-config --file=nifi-ambari-ssl-config.json /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-authorizers-env --file=nifi-authorizers-env.json /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-bootstrap-env --file=nifi-bootstrap-env.json /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-bootstrap-notification-services-env --file=nifi-bootstrap-notification-services-env.json /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-env --file=nifi-env.json /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-flow-env --file=nifi-flow-env.json /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-jaas-conf --file=nifi-jaas-conf.json /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-login-identity-providers-env --file=nifi-login-identity-providers-env.json /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-node-logback-env --file=nifi-node-logback-env.json /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-properties --file=nifi-properties.json /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-state-management-env --file=nifi-state-management-env.json /var/lib/ambari-server/resources/scripts/configs.py --user=admin --password=admin --port=8080 --action=set --host=localhost --cluster=test-cluster --config-type=nifi-toolkit-env --file=nifi-toolkit-env.json #Install Nifi curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context" :"Install Nifi"}, "Body": {"ServiceInfo": {"maintenance_state" : "OFF", "state": "INSTALLED"}}}' https://localhost/ambari/api/v1/clusters/test-cluster/services/NIFI curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"RequestInfo": {"context" :"Start NIFI"}, "Body": {"ServiceInfo": {"maintenance_state" : "OFF", "state": "STARTED"}}}' https://localhost/ambari/api/v1/clusters/test-cluster/services/NIFI
The install is successful but the last command to start the nifi service gives the following error on Ambari.
Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 231, in <module> Master().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 152, in start self.configure(env, is_starting = True) File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 120, in locking_configure original_configure(obj, *args, **kw) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 101, in configure if not sudo.path_isfile(params.nifi_flow_config_dir+'/flow.xml.gz') and nifi_toolkit_util.existing_cluster(params): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi_toolkit_util.py", line 314, in existing_cluster zk_command_location = os.path.join(params.zk_root, params.zk_stack_version_buildnum, "zookeeper", "bin", "zkCli.sh") File "/usr/lib64/python2.7/posixpath.py", line 75, in join if b.startswith('/'): AttributeError: 'NoneType' object has no attribute 'startswith'
Has anyone else seen this issue, or have a suggestion on how to get this to work?
Many thanks in advance.
TL
Created 04-05-2018 09:13 PM
If you do not stick to this two-phased solution, another approach you can take is to install HDF with Cloudbreak 2.5 itself.
Here is a tutorial as well: https://community.hortonworks.com/articles/182221/create-a-nifi-cluster-on-aws-azure-google-or-opens...
Hope this helps!
Created 09-21-2018 12:39 PM
Did you ever find a solution?