Support Questions

Find answers, ask questions, and share your expertise

Nifi-toolkit not found while upgrading to HDF 3.2

avatar
Rising Star

Hey all, I am having issues getting Nifi and Nifi Registry to start so that they may be updated to 1.7 with the release of HDF 3.2. I'm not seeing many other topics covering this issue. I have checked that the file exists that is being referenced here, there is a directory for both Nifi 1.5 and 1.7 toolkit. I have tried moving the 1.7 directory out of the tmp folder in case that was creating confusion but that didn't work either. Any ideas?

stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 280, in <module>
    Master().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 353, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 128, in start
    self.configure(env, is_starting = True)
  File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 98, in configure
    self.write_configurations(params, is_starting)
  File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_registry.py", line 200, in write_configurations
    is_starting, params.toolkit_tmp_dir, params.stack_version_buildnum)
  File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_toolkit_util.py", line 342, in encrypt_sensitive_properties
    encrypt_config_script = get_toolkit_script('encrypt-config.sh',toolkit_tmp_dir, stack_version_buildnum)
  File "/var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/scripts/nifi_toolkit_util.py", line 61, in get_toolkit_script
    raise Exception("Couldn't find nifi toolkit directory in " + scriptDir)
Exception: Couldn't find nifi toolkit directory in /var/lib/ambari-agent/tmp
 stdout:
2018-08-20 16:11:12,671 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.1.0-35 -> 3.1.1.0-35
2018-08-20 16:11:12,849 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.1.0-35 -> 3.1.1.0-35
2018-08-20 16:11:12,852 - Group['nifiregistry'] {}
2018-08-20 16:11:12,853 - Group['hadoop'] {}
2018-08-20 16:11:12,853 - Group['nifi'] {}
2018-08-20 16:11:12,854 - User['streamline'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,855 - User['logsearch'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,855 - User['registry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,856 - User['storm'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,857 - User['infra-solr'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,857 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,858 - User['ams'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,859 - User['ambari-qa'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,860 - User['kafka'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2018-08-20 16:11:12,860 - User['nifiregistry'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifiregistry'], 'uid': None}
2018-08-20 16:11:12,861 - User['nifi'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['nifi'], 'uid': None}
2018-08-20 16:11:12,862 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2018-08-20 16:11:12,863 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2018-08-20 16:11:12,868 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa 0'] due to not_if
2018-08-20 16:11:12,883 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2018-08-20 16:11:12,889 - Skipping Execute[('setenforce', '0')] due to not_if
2018-08-20 16:11:12,889 - Testing the JVM's JCE policy to see it if supports an unlimited key length.
2018-08-20 16:11:13,068 - The unlimited key JCE policy is required, and appears to have been installed.
2018-08-20 16:11:13,313 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.1.0-35 -> 3.1.1.0-35
2018-08-20 16:11:13,325 - Toolkit files dir is /var/lib/ambari-agent/cache/common-services/NIFI_REGISTRY/0.1.0/package/files
2018-08-20 16:11:13,325 - Toolkit tmp dir is /var/lib/ambari-agent/tmp
2018-08-20 16:11:13,376 - Directory['/var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520'] {'group': 'nifiregistry', 'cd_access': 'a', 'create_parents': False, 'recursive_ownership': True, 'owner': 'nifiregistry', 'mode': 0755}
2018-08-20 16:11:13,378 - Changing owner for /var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520 from 0 to nifiregistry
2018-08-20 16:11:13,378 - Changing group for /var/lib/ambari-agent/tmp/nifi-toolkit-1.7.0.3.2.0.0-520 from 0 to nifiregistry
2018-08-20 16:11:13,388 - Directory['/var/run/nifi-registry'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,388 - Directory['/var/lib/nifi-registry'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,389 - Directory['/var/lib/nifi-registry/conf'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,390 - Directory['/var/lib/nifi-registry/database'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,391 - Directory['/usr/hdf/current/nifi-registry/conf'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,391 - Directory['/usr/hdf/current/nifi-registry/bin'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,392 - Directory['/usr/hdf/current/nifi-registry/lib'] {'owner': 'nifiregistry', 'create_parents': True, 'group': 'nifiregistry', 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,394 - Directory['/usr/hdf/current/nifi-registry/docs'] {'owner': 'nifiregistry', 'group': 'nifiregistry', 'create_parents': True, 'recursive_ownership': True, 'cd_access': 'a'}
2018-08-20 16:11:13,396 - PropertiesFile['/usr/hdf/current/nifi-registry/conf/nifi-registry.properties'] {'owner': 'nifiregistry', 'group': 'nifiregistry', 'mode': 0600, 'properties': ...}
2018-08-20 16:11:13,402 - Generating properties file: /usr/hdf/current/nifi-registry/conf/nifi-registry.properties
2018-08-20 16:11:13,402 - File['/usr/hdf/current/nifi-registry/conf/nifi-registry.properties'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600, 'encoding': 'UTF-8'}
2018-08-20 16:11:13,422 - Writing File['/usr/hdf/current/nifi-registry/conf/nifi-registry.properties'] because contents don't match
2018-08-20 16:11:13,425 - File['/usr/hdf/current/nifi-registry/conf/bootstrap.conf'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600}
2018-08-20 16:11:13,428 - File['/usr/hdf/current/nifi-registry/conf/logback.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0400}
2018-08-20 16:11:13,436 - File['/usr/hdf/current/nifi-registry/conf/authorizers.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600}
2018-08-20 16:11:13,440 - File['/usr/hdf/current/nifi-registry/conf/identity-providers.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0600}
2018-08-20 16:11:13,442 - File['/usr/hdf/current/nifi-registry/conf/providers.xml'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0400}
2018-08-20 16:11:13,444 - File['/usr/hdf/current/nifi-registry/bin/nifi-registry-env.sh'] {'owner': 'nifiregistry', 'content': InlineTemplate(...), 'group': 'nifiregistry', 'mode': 0755}
2018-08-20 16:11:13,445 - Encrypting NiFi Registry sensitive configuration properties


Command failed after 1 tries


1 ACCEPTED SOLUTION

avatar

@Nick Lewis looks like it is failing with "Exception: Couldn't find nifi toolkit directory in /var/lib/ambari-agent/tmp stdou" error.

Can you verify with https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.2.0/ambari-managed-hdf-upgrade/content/hdf-veri...

View solution in original post

2 REPLIES 2

avatar

@Nick Lewis looks like it is failing with "Exception: Couldn't find nifi toolkit directory in /var/lib/ambari-agent/tmp stdou" error.

Can you verify with https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.2.0/ambari-managed-hdf-upgrade/content/hdf-veri...

avatar
Rising Star

That solved my problem! I went to Hortonworks downloads and grabbed the correct versions of Nifi toolkit. I am now onto the next error:

resource_management.core.exceptions.ExecutionFailed: Execution of '/var/lib/ambari-agent/tmp/nifi-toolkit-1.5.0.3.1.1.0-35/bin/encrypt-config.sh -v -b /usr/hdf/current/nifi/conf/bootstrap.conf -n /usr/hdf/current/nifi/conf/nifi.properties -f /var/lib/nifi/conf/flow.xml.gz -s '[PROTECTED]' -a /usr/hdf/current/nifi/conf/authorizers.xml -p '[PROTECTED]'' returned 1. Error: Could not find or load main class org.apache.nifi.toolkit.encryptconfig.EncryptConfigMain

But I'll save that for another post once I've investigated some more, thanks!