Member since
07-25-2016
15
Posts
0
Kudos Received
0
Solutions
06-25-2020
10:44 PM
Issue
Unable to install NiFi 3.5.1 from an HDP 3.1.5 cluster.
Background
In a twelve node HDP 3.1.5 cluster, all services are installed and run successfully. The cluster has Kerberos and full SSL installed. The correct mpack for HDF NiFi has been installed. Ambari-server and agents are restarted. However, NiFi fails to install.
Error
The following is the error displayed during the installation:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 312, in <module>
Master().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 66, in install
self.install_packages(env)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 853, in install_packages
retry_count=agent_stack_retry_count)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install
self._pkg_manager.install_package(package_name, self.__create_context())
File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 219, in install_package
shell.repository_manager_executor(cmd, self.properties, context)
File "/usr/lib/ambari-agent/lib/ambari_commons/shell.py", line 753, in repository_manager_executor
raise RuntimeError(message)
RuntimeError: Failed to execute command '/usr/bin/yum -y install nifi_3_5_*', exited with code '1', message: 'https://<username>:<password>@archive.cloudera.com/p/HDF/centos7/3.x/updates/3.5.1.0/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden
Troubleshooting
In this case, a version.xml file was used when installing the HDP cluster. When checking Managed Versions > Versions > 3.1.5, the listing for HDF, HDP, and HDP-Utils was the same. They were are displaying a field like the following:
https://****:****@archive.cloudera.com/p/HDF/centos7/3.x/updates/3.5.1.0
This Cloudera documentation states to edit the file on the NiFi nodes for ambari-hdp-1.repo. When performing this step, the install overwrites this file every time during the installation. However, when the <username>:<password> is replaced with license ID and password, when running the installation again, it overwrites the changes and the file reverts to the original configuration.
File ambari-hdp-1.repo
[HDF-3.5-repo-1]
name=HDF-3.5-repo-1
baseurl=https://<username>:<password>@archive.cloudera.com/p/HDF/centos7/3.x/updates/3.5.1.0
Resolution
In the original version.xml file that was used to install HDP, there were no entries for HDF. However, the new listing in Ambari clearly showed the services for NiFi, NiFi Registery, etc being added. This may be been an issue because the <username>:<license> was not updated during the installation of the mpack.
To resolve this issue, manually insert the username:license into the Managed Version > HDP 3.1.5 > HDF 3.5.0.
Essentially, manually building a combined HDP / HDF version.xml should help you install NiFi without any issues.
... View more
Labels:
11-06-2019
06:20 AM
1 Kudo
@wdailey Use "ldapRealm=org.apache.zeppelin.realm.LdapRealm" Below is the template for your reference ldapRealm=org.apache.zeppelin.realm.LdapRealm
ldapRealm.contextFactory.systemUsername =cn=manager,dc=charan,dc=com
ldapRealm.contextFactory.systemPassword =admin
ldapRealm.contextFactory.authenticationMechanism=simple
ldapRealm.contextFactory.url=ldap://test1:389
ldapRealm.authorizationEnabled=true
#ldapRealm.pagingSize = 20000
ldapRealm.searchBase=
ldapRealm.userSearchBase=
ldapRealm.groupSearchBase=
ldapRealm.userObjectClass=
ldapRealm.userSearchAttributeName = uid
ldapRealm.userSearchScope = subtree
ldapRealm.groupSearchScope = subtree
ldapRealm.userSearchFilter=
ldapRealm.memberAttribute = member
ldapRealm.memberAttributeValueTemplate=(name={0})
... View more
11-09-2017
01:41 PM
I have a question. To use this function, should I install hadoop client on ambari-sever host?
... View more
09-28-2017
02:51 AM
1 Kudo
Hi @Will Dailey, It is possible to configure the secondary LDAP server to handle the AD Failure scenario. --ldap-secondary-url is the parameter(optional) which will take additional AD host in the ldap-setup command line arguments. --ldap-url=LDAP_URL Primary url for LDAP
--ldap-secondary-url=LDAP_SECONDARY_URL Secondary url for LDAP
... View more
02-28-2017
03:05 AM
Here's the latest Ambari doc with the same http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-security/content/optional_ambari_web_inactivity_timeout.html
... View more
03-03-2017
11:36 PM
4 Kudos
@Will Dailey, The JVM heap can be preconfigured for memory boundaries – the initial heap size (defined by the –Xms option) and the maximum heap size (defined by the –Xmx option). Used memory from the perspective of the JVM is Working set + Garbage. The Committed memory is a measure of how much memory the JVM heap is really consuming. If ( –Xms < -Xmx ) and (Used memory == Current heap size), the JVM is likely to grow its heap after a full garbage collection. However if ( –Xms or Current heap size == -Xmx ), either through heap growth or configuration, the heap cannot grow any further. Both of these Values are important to debug datanode failure related Out of Memory, less PermGen Errors, GC errors. Reference link : https://pubs.vmware.com/vfabric52/index.jsp?topic=/com.vmware.vfabric.em4j.1.2/em4j/conf-heap-management.html
... View more
10-09-2016
02:14 AM
1 Kudo
How about yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill ? The default is 15 sec (15000 ms), see here and here for details.
... View more