Install Nifi 3.0 on HDP 2.6.3 Getting "resource_management.core.exceptions.Fail: Configuration parameter 'availableServices' was not found in configurations dictionary!"

[#1]

I'm trying to install HDF 3.0 onto an existing HDP cluster. I followed the directions here https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.1.1/bk_installing-hdf-and-hdp/content/ch_add-... and was able to install storm, kafka, registry, and sam with no problem. However installing nifi fails. It's obvious that there's some configuration information missing somewhere, but I don't know where. The Nifi certificate authority does install.

This is stderr:
Traceback (most recent call last):

  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 309, in <module>
    Master().execute()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 367, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 71, in install
    self.configure(env, True)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 120, in locking_configure
    original_configure(obj, *args, **kw)
  File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 112, in configure
    PropertiesFile(params.nifi_config_dir + '/nifi.properties', properties = params.nifi_properties, mode = 0600, owner = params.nifi_user, group = params.nifi_group)
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/properties_file.py", line 54, in action_create
    mode = self.resource.mode
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 123, in action_create
    content = self._get_content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 160, in _get_content
    return content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 52, in __call__
    return self.get_content()
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 144, in get_content
    rendered = self.template.render(self.context)
  File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in render
    return self.environment.handle_exception(exc_info, True)
  File "<template>", line 3, in top-level template code
  File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 144, in get_content
    rendered = self.template.render(self.context)
  File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in render
    return self.environment.handle_exception(exc_info, True)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__
    raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!")
resource_management.core.exceptions.Fail: Configuration parameter 'availableServices' was not found in configurations dictionary!

stdout (I truncated all but the last few lines of stdout):


2017-11-02 13:11:16,171 - Directory['/var/lib/nifi/state/local'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-11-02 13:11:16,171 - Directory['/usr/hdf/current/nifi/lib'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-11-02 13:11:16,173 - Directory['{{nifi_content_repo_dir_default}}'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True}
2017-11-02 13:11:16,174 - Directory['/var/lib/nifi/content_repository'] {'owner': 'nifi', 'group': 'nifi', 'create_parents': True, 'recursive_ownership': True}
2017-11-02 13:11:16,174 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'}
2017-11-02 13:11:16,180 - File['/etc/security/limits.d/nifi.conf'] {'content': Template('nifi.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2017-11-02 13:11:16,181 - PropertiesFile['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'group': 'nifi', 'mode': 0600, 'properties': ...}
2017-11-02 13:11:16,188 - Generating properties file: /usr/hdf/current/nifi/conf/nifi.properties
2017-11-02 13:11:16,188 - File['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600}
2017-11-02 13:11:16,290 - The repository with version 2.6.3.0-235 for this command has been marked as resolved. It will be used to report the version of the component which was installed
2017-11-02 13:11:16,291 - Skipping stack-select on NIFI because it does not exist in the stack-select package structure.

Command failed after 1 tries

[#2]

Not an answer, but another comment. I'm not sure that this is a cause or effect, but I've found that /var/lib/ambari-agent/cache/common-services/NIFI/1.0.0 has no configuration directory nor a metainfo.xml file, just a package directory (even on the ambari server ndoe.) Nifi is installed on the node and can be run as a service. It just isn't able to be controlled by ambari.

[#3]

Hi Raymond, what version of Ambari is your cluster running?

[#4]

Should be the version that's in 2.6.3, but this is the result of the rpm query:

ambari-server-2.6.0.0-267.x86_64

[#5]

The problem you experienced is because the current version of HDF 3.0.1.1 is only compatible with Ambari 2.5.1 (see the Component Availability Matrix here https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.1.1/bk_release-notes/content/ch_hdf_relnotes..... There is an upcoming release of HDF that will address this problem for 2.6.0.

[#6]

Okay, interesting as SAM and Registry work fine. Thanks.

[#7]

Yes the NiFi service has a reliance on an Ambari configuration that is no longer available in Ambari 2.6.0. The upcoming release removes that dependency.

[#8]

@Yolanda, I Ran into the same issue just now :). Seems like have to downgrade to Ambari 2.5.1 and re-install is the only option available?

[#9]

Hi @Muki Soomar,

First make sure that you uninstall the previous mpack (HDF 3.0.1.1), next install the new mpack 3.0.2.

It should work after that.

Regards,

[#10]

@Soomar, as I said, you can just install the deployed nifi software and run it as a service. The only thing ATM that you miss out on is starting/stoping from Ambari and changing the configuration from it. Clustering is _much_ easier if you have it running from Ambari also.

[#11]

Yes that is correct, I apologize for not being clear. The Ambari code in the HDF management pack that manages the NiFi service has a reliance on an Ambari configuration that is no longer available in Ambari 2.6.0.

[#12]

@Raymond Pfaff and @Yolanda M. Davis. Yes. Thanks, I just saw that.

[#13]

Hi @Raymond Pfaff

Ambari MPACKs are only compatible with the version of Ambari they were written against. You can then only upgrade to the next release of each product that coincides Ambari support ( ex. HDP 2.6.1 and HDF 3.0 against Ambari 2.5.1.0 )

HDF 3.0.2 has just been released and supports Ambari 2.6.0 :
https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.2/bk_support-matrices/content/ch_matrices-hdf...

That should take care of the prb.

Regards,

[#14]

Hi @Raymond Pfaff

Ambari MPACKs are only compatible with the version of Ambari they were written against.

You can only upgrade to the next release of each product that coincides with Ambari support ( ex. HDP 2.6.1 and HDF 3.0 against Ambari 2.5.1.0 )

HDF 3.0.2 has just been released and supports Ambari 2.6.0 :
https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.2/bk_support-matrices/content/ch_matrices-hdf...

That should take care of the prb.

Regards,

[#15]

I am using HDP-2.6.5.0 and HDF-ambari-mpack-3.1.2.0-7 still NIFI is not working showing error

Skipping stack-select on NIFI because it does not exist in the stack-select package structure.

[#16]

I am also using HDP-2.6.5.0 and HDF-ambari-mpack-3.1.2.0-7 and am facing the same issue

1. Skipping stack-select on NIFI because it does not exist in the stack-selectpackage structure.

[#17]

@Ashish Kumar ,@Misbah Rehman : Were you able to resolve the issue?