HDF 3.5.1 installation fails because the paywall credentials is missing in the Ambari-hdp-1.repo file

[#1]

Issue

Unable to install NiFi 3.5.1 from an HDP 3.1.5 cluster.

Background

In a twelve node HDP 3.1.5 cluster, all services are installed and run successfully. The cluster has Kerberos and full SSL installed. The correct mpack for HDF NiFi has been installed. Ambari-server and agents are restarted. However, NiFi fails to install. 

Error

The following is the error displayed during the installation: 

Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 312, in <module>
Master().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 66, in install
self.install_packages(env)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 853, in install_packages
retry_count=agent_stack_retry_count)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install
self._pkg_manager.install_package(package_name, self.__create_context())
File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 219, in install_package
shell.repository_manager_executor(cmd, self.properties, context)
File "/usr/lib/ambari-agent/lib/ambari_commons/shell.py", line 753, in repository_manager_executor
raise RuntimeError(message)
RuntimeError: Failed to execute command '/usr/bin/yum -y install nifi_3_5_*', exited with code '1', message: 'https://<username>:<password>@archive.cloudera.com/p/HDF/centos7/3.x/updates/3.5.1.0/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden

Troubleshooting

In this case, a version.xml file was used when installing the HDP cluster. When checking Managed Versions > Versions >  3.1.5, the listing for HDF, HDP, and HDP-Utils was the same. They were are displaying a field like the following:

https://****:****@archive.cloudera.com/p/HDF/centos7/3.x/updates/3.5.1.0

This Cloudera documentation states to edit the file on the NiFi nodes for ambari-hdp-1.repo. When performing this step, the install overwrites this file every time during the installation. However, when the <username>:<password> is replaced with license ID and password, when running the installation again, it overwrites the changes and the file reverts to the original configuration.

File ambari-hdp-1.repo 

[HDF-3.5-repo-1]
name=HDF-3.5-repo-1
baseurl=https://<username>:<password>@archive.cloudera.com/p/HDF/centos7/3.x/updates/3.5.1.0

Resolution

In the original version.xml file that was used to install HDP, there were no entries for HDF. However, the new listing in Ambari clearly showed the services for NiFi, NiFi Registery, etc being added. This may be been an issue because the <username>:<license> was not updated during the installation of the mpack.

To resolve this issue, manually insert the username:license into the Managed Version > HDP 3.1.5 > HDF 3.5.0. 

Essentially, manually building a combined HDP / HDF version.xml should help you install NiFi without any issues. 

[#2]

Issue

Unable to install NiFi 3.5.1 from an HDP 3.1.5 cluster.

Background

In a twelve node HDP 3.1.5 cluster, all services are installed and run successfully. The cluster has Kerberos and full SSL installed. The correct mpack for HDF NiFi has been installed. Ambari-server and agents are restarted. However, NiFi fails to install. 

Error

The following is the error displayed during the installation: 

Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 312, in <module>
Master().execute()
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/NIFI/1.0.0/package/scripts/nifi.py", line 66, in install
self.install_packages(env)
File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 853, in install_packages
retry_count=agent_stack_retry_count)
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install
self._pkg_manager.install_package(package_name, self.__create_context())
File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 219, in install_package
shell.repository_manager_executor(cmd, self.properties, context)
File "/usr/lib/ambari-agent/lib/ambari_commons/shell.py", line 753, in repository_manager_executor
raise RuntimeError(message)
RuntimeError: Failed to execute command '/usr/bin/yum -y install nifi_3_5_*', exited with code '1', message: 'https://<username>:<password>@archive.cloudera.com/p/HDF/centos7/3.x/updates/3.5.1.0/repodata/repomd.xml: [Errno 14] HTTPS Error 403 - Forbidden

Troubleshooting

In this case, a version.xml file was used when installing the HDP cluster. When checking Managed Versions > Versions >  3.1.5, the listing for HDF, HDP, and HDP-Utils was the same. They were are displaying a field like the following:

https://****:****@archive.cloudera.com/p/HDF/centos7/3.x/updates/3.5.1.0

This Cloudera documentation states to edit the file on the NiFi nodes for ambari-hdp-1.repo. When performing this step, the install overwrites this file every time during the installation. However, when the <username>:<password> is replaced with license ID and password, when running the installation again, it overwrites the changes and the file reverts to the original configuration.

File ambari-hdp-1.repo 

[HDF-3.5-repo-1]
name=HDF-3.5-repo-1
baseurl=https://<username>:<password>@archive.cloudera.com/p/HDF/centos7/3.x/updates/3.5.1.0

Resolution

In the original version.xml file that was used to install HDP, there were no entries for HDF. However, the new listing in Ambari clearly showed the services for NiFi, NiFi Registery, etc being added. This may be been an issue because the <username>:<license> was not updated during the installation of the mpack.

To resolve this issue, manually insert the username:license into the Managed Version > HDP 3.1.5 > HDF 3.5.0. 

Essentially, manually building a combined HDP / HDF version.xml should help you install NiFi without any issues.