Created on 04-29-2018 12:50 PM
Issue:
This issue has been observed on an HDP cluster running NiFi service with HDF 3.0.1 managed by Ambari 2.6.1.5.
HDF 3.0.1 is supported only with Ambari 2.5.1 version. Therefore, in situation where you upgrade Ambari from 2.5.x to 2.6.y HDF services like NiFi won't start & complain the error shared in the below stack trace.
To handle this, you have to upgrade HDF mpack from 3.0.1 to 3.0.2 after the Ambari upgrade is complete. This additional activity is required to get the HDF service working with Ambari 2.6.y. Also, this step is independent of the HDF upgrade that you would have planned separately
stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 309, in <module> Master().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 375, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 177, in start self.configure(env, is_starting = True) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 120, in locking_configure original_configure(obj, *args, **kw) File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 112, in configure PropertiesFile(params.nifi_config_dir + '/nifi.properties', properties = params.nifi_properties, mode = 0600, owner = params.nifi_user, group = params.nifi_group) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/properties_file.py", line 54, in action_create mode = self.resource.mode File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 166, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 123, in action_create content = self._get_content() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 160, in _get_content return content() File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 52, in __call__ return self.get_content() File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 144, in get_content rendered = self.template.render(self.context) File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in render return self.environment.handle_exception(exc_info, True) File "<template>", line 3, in top-level template code File "/usr/lib/python2.6/site-packages/resource_management/core/source.py", line 144, in get_content rendered = self.template.render(self.context) File "/usr/lib/python2.6/site-packages/ambari_jinja2/environment.py", line 891, in render return self.environment.handle_exception(exc_info, True) File "<template>", line 1, in top-level template code File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/config_dictionary.py", line 73, in __getattr__ raise Fail("Configuration parameter '" + self.name + "' was not found in configurations dictionary!") resource_management.core.exceptions.Fail: Configuration parameter 'availableServices' was not found in configurations dictionary! stdout: 2018-04-27 11:49:37,580 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.1.0-129 -> 2.6.1.0-129 2018-04-27 11:49:37,592 - Using hadoop conf dir: /usr/hdp/2.6.1.0-129/hadoop/conf 2018-04-27 11:49:37,722 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.1.0-129 -> 2.6.1.0-129 2018-04-27 11:49:37,726 - Using hadoop conf dir: /usr/hdp/2.6.1.0-129/hadoop/conf 2018-04-27 11:49:37,726 - Group['kms'] {} 2018-04-27 11:49:37,727 - Group['livy'] {} 2018-04-27 11:49:37,728 - Group['spark'] {} 2018-04-27 11:49:37,728 - Group['ranger'] {} 2018-04-27 11:49:37,728 - Group['sg_hdp_hdfsadmins'] {} 2018-04-27 11:49:37,730 - Group['zeppelin'] {} 2018-04-27 11:49:37,730 - Group['hadoop'] {} 2018-04-27 11:49:37,730 - Group['nifi'] {} 2018-04-27 11:49:37,730 - Group['users'] {} 2018-04-27 11:49:37,730 - Group['knox'] {} 2018-04-27 11:49:37,731 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,732 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,733 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,734 - User['atlas'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,735 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None} 2018-04-27 11:49:37,735 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,736 - User['falcon'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None} 2018-04-27 11:49:37,737 - User['ranger'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'ranger'], 'uid': None} 2018-04-27 11:49:37,738 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None} 2018-04-27 11:49:37,738 - User['zeppelin'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'zeppelin', u'hadoop'], 'uid': None} 2018-04-27 11:49:37,739 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,740 - User['kms'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,741 - User['livy'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,742 - User['spark'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,742 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'users'], 'uid': None} 2018-04-27 11:49:37,743 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,744 - User['hdfs'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['sg_hdp_hdfsadmins'], 'uid': None} 2018-04-27 11:49:37,744 - Modifying user hdfs 2018-04-27 11:49:37,756 - User['sqoop'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,757 - User['yarn'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,758 - User['hbase'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,759 - User['hcat'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,759 - User['mapred'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,760 - User['knox'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': [u'hadoop'], 'uid': None} 2018-04-27 11:49:37,761 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-04-27 11:49:37,762 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2018-04-27 11:49:37,770 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if 2018-04-27 11:49:37,770 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'} 2018-04-27 11:49:37,771 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-04-27 11:49:37,772 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2018-04-27 11:49:37,772 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {} 2018-04-27 11:49:37,782 - call returned (0, '1018') 2018-04-27 11:49:37,782 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1018'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'} 2018-04-27 11:49:37,790 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1018'] due to not_if 2018-04-27 11:49:37,790 - Group['hdfs'] {} 2018-04-27 11:49:37,791 - User['hdfs'] {'fetch_nonlocal_groups': True, 'groups': ['sg_hdp_hdfsadmins', 'hadoop', 'hdfs', u'hdfs']} 2018-04-27 11:49:37,791 - Modifying user hdfs 2018-04-27 11:49:37,802 - FS Type: 2018-04-27 11:49:37,802 - Directory['/etc/hadoop'] {'mode': 0755} 2018-04-27 11:49:37,812 - File['/usr/hdp/2.6.1.0-129/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'root', 'group': 'hadoop'} 2018-04-27 11:49:37,812 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777} 2018-04-27 11:49:37,823 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'} 2018-04-27 11:49:37,832 - Skipping Execute[('setenforce', '0')] due to not_if 2018-04-27 11:49:37,832 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'} 2018-04-27 11:49:37,834 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'} 2018-04-27 11:49:37,834 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'} 2018-04-27 11:49:37,837 - File['/usr/hdp/2.6.1.0-129/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'root'} 2018-04-27 11:49:37,838 - File['/usr/hdp/2.6.1.0-129/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'root'} 2018-04-27 11:49:37,842 - File['/usr/hdp/2.6.1.0-129/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644} 2018-04-27 11:49:37,849 - File['/usr/hdp/2.6.1.0-129/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'} 2018-04-27 11:49:37,849 - File['/usr/hdp/2.6.1.0-129/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755} 2018-04-27 11:49:37,850 - File['/usr/hdp/2.6.1.0-129/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'} 2018-04-27 11:49:37,853 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644} 2018-04-27 11:49:37,859 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755} 2018-04-27 11:49:37,873 - Skipping stack-select on NIFI because it does not exist in the stack-select package structure. 2018-04-27 11:49:38,078 - Stack Feature Version Info: Cluster Stack=2.6, Command Stack=None, Command Version=2.6.1.0-129 -> 2.6.1.0-129 2018-04-27 11:49:38,118 - Using hadoop conf dir: /usr/hdp/2.6.1.0-129/hadoop/conf 2018-04-27 11:49:38,119 - Directory['/var/run/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2018-04-27 11:49:38,121 - Directory['/var/log/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2018-04-27 11:49:38,122 - Directory['/var/lib/nifi'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2018-04-27 11:49:38,189 - Directory['/var/lib/nifi/database_repository'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2018-04-27 11:49:38,189 - Directory['/flowfile_repo'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2018-04-27 11:49:38,197 - Directory['/prov_repo1'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2018-04-27 11:49:38,198 - Directory['/usr/hdf/current/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2018-04-27 11:49:38,199 - Directory['/var/lib/nifi/conf'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2018-04-27 11:49:38,199 - Directory['/var/lib/nifi/state/local'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2018-04-27 11:49:38,200 - Directory['/usr/hdf/current/nifi/lib'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2018-04-27 11:49:38,201 - Directory['{{nifi_content_repo_dir_default}}'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2018-04-27 11:49:38,201 - Directory['/cont_repo2'] {'owner': 'nifi', 'create_parents': True, 'group': 'nifi', 'recursive_ownership': True} 2018-04-27 11:49:38,230 - Directory['/cont_repo1'] {'owner': 'nifi', 'group': 'nifi', 'create_parents': True, 'recursive_ownership': True} 2018-04-27 11:49:38,258 - Directory['/etc/security/limits.d'] {'owner': 'root', 'create_parents': True, 'group': 'root'} 2018-04-27 11:49:38,261 - File['/etc/security/limits.d/nifi.conf'] {'content': Template('nifi.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644} 2018-04-27 11:49:38,271 - PropertiesFile['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'group': 'nifi', 'mode': 0600, 'properties': ...} 2018-04-27 11:49:38,274 - Generating properties file: /usr/hdf/current/nifi/conf/nifi.properties 2018-04-27 11:49:38,274 - File['/usr/hdf/current/nifi/conf/nifi.properties'] {'owner': 'nifi', 'content': InlineTemplate(...), 'group': 'nifi', 'mode': 0600} 2018-04-27 11:49:38,350 - Skipping stack-select on NIFI because it does not exist in the stack-select package structure. Command failed after 1 tries
Resolution:
ssh into Ambari node and follow the steps described on this official doc
For support matrices on HDF 3.0.2 please visit this link
For support matrices on HDF 3.0.1 please visit this link
For any other info, please visit docs.hortonworks.com