Error while upgrading from HDP 2.3.4 to HDP 2.6.5

[#1]

Hi ,

We are currently upgrading from HDP 2.3.4 to HDP 2.6.5 following the following reference document as suggested in the following link.

https://docs.cloudera.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-upgrade/content/ambari_upgrade_guide...

Ours is an environment without internet connection so we downloaded and created the offline repository.

We already upgraded the ambari from 2.4.2 to Ambari 2.6.2.2 successfully and performed all the prerequisites for HDP upgrade. We are currently got stuck in the "Register and Install Target Version" phase. We have registered the local repository for HDP 2.6.5 and from ambari>stacks and versions>manage version we are trying to install the new softwares which is failing with the following error message.This issue is only happening for only one of the hosts in cluster.Rest all other hosts the installation was successful.

"Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do"

Please help. Somehow it is detecting a blank package.

I have attached the output and error logs.

Thanks and Regards,
Ananya

[#2]

Hi Experts,

Any idea on this issue please. We are currently stuck with the upgrade and can not proceed further.

Thanks and Regards,

Ananya

[#3]

Hello @Ananya_Misra ,

thank you for reaching out to the Community!

Please check on the host where the installation fails if there is any package left from the old version. Please remove all packages that are for the 2.3.4 version and attempt the installation on this node again.

Hope this helps!

Kind regards:

Ferenc

[#4]

Hi @Bender Frenec,

I am on the installation binary phase and the upgrade is not yet started.

If I check on the host where the installation is failing as well as where installation is successful. Both servers still got the packages for HDP 2.3.4.

Is it okay to remove them before the upgrade is completed and we finalize the upgrade.

Please advise.

Thanks and Regards,

Ananya

[#5]

Hello @Ananya_Misra ,

I was researching further your issue.

Found a case in which the user did the below workaround after hitting the same exception:

"unregister the version, register again the version to fix the issue"

Also found AMBARI-23058 jira that seems to be related and it described:

After cleaning up with 'yum-complete-transaction --cleanup-only' still Ambari could not able to proceed further with same error.

When user tried to do yum install it was going through successfully.

Finally had to delete /var/lib/yum/transaction* manually for ambari to proceed further with installation. These transactions files were very old and nothing to do with any latest installation but still ambari does not proceed further.

You can check with the below command if you have old transaction files on your node:

ls -lart /var/lib/yum/transaction*

Let us know, please if it helped you to overcome the obstackle!

Kind regards:

Ferenc

[#6]

Hi @Bender Frenec,

Really appreciate your prompt response. I have tried your suggestion.

1. I tried to deregister the 2.6.5 version but the deregister button is disabled and it is saying can not deregister as the version already installed on some nodes.

2. I checked and found no old transaction by running following command on my problematic node.

ls -lart /var/lib/yum/transaction*

Anything else I can check.

Thanks and Regards,

Ananya

[#7]

Please check  ambari data :

select * from execution_command where task_id=13296\G;
select * from host_role_command where task_id=13296\G;
get the 'role' column value

select * from stage where request_id=13296;

 maybe you will find some clues～

[#8 · Accepted solution]

Hi Experts,

My issue is resolved now. I had raised a ticket with cloudera and they provided my with a modified yumrpm.py file which I replaced with my existing file. This new python had option to skip in case it encounters a blank package.

Following are the steps to resolve my issue.

1) verify ambari-agent version is 2.6.2.2 by `ambari-agent --version`  on problematic node

2) stop ambari-agent on problematic node

3)go to /usr/lib/ambari-agent/lib/resource_management/core/providers/package/

~> cd /usr/lib/ambari-agent/lib/resource_management/core/providers/package/

4) take bacup of yumrpm.py

~> mv yumrpm.py /tmp

5) download the attached yumrpm.py and keep it in /user/home/ananya

6) do an diff of

~> diff /tmp/yumrpm.py  /user/home/ananya/yumrpm.py

>>>(diff loooks similar too this)

263,267c263,264

<       if len(name) == 0:

<         Logger.info("Skipping installation of empty package")

<       else:

<         Logger.info("Installing package %s ('%s')" % (name, string_cmd_from_args_list(cmd)))

<         self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput())

---

>       Logger.info("Installing package %s ('%s')" % (name, string_cmd_from_args_list(cmd)))

>       self.checked_call_with_retries(cmd, sudo=True,

> logoutput=self.get_logoutput())

<<<< 

7) move the yumrpm.py  to /usr/lib/ambari-agent/lib/resource_management/core/providers/package/

 mv /user/home/ananya/yumrpm.py  /usr/lib/ambari-agent/lib/resource_management/core/providers/package/

😎 start ambari-agent

Thanks and Regards,

Ananya