question Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail in Support Questions

Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

ilyal — Sun, 18 Aug 2019 02:58:45 GMT

Hi All,

I try run rolling upgrade with subject versions.

And it fails with the very strange message. I added the image with current state

and in ambari-server.log I see this lines:

WARN [Server Action Executor Worker 2015] ServerActionExecutor:471 - Task #2015 failed to complete execution due to thrown exception: java.lang.NullPointerException:null
java.lang.NullPointerException
        at org.apache.ambari.server.serveraction.upgrades.ConfigureAction.execute(ConfigureAction.java:200)
        at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:550)
        at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:466)
        at java.lang.Thread.run(Thread.java:745)

Please help.

Thanks,

Ilya

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

apappu — Thu, 09 Aug 2018 02:39:56 GMT

@Ilya Li

Are you using Ambari 2.6.2 version?

For some reason your source repository is coming as null - you can refer at https://github.com/apache/ambari/blob/release-2.6.2-rc0/ambari-server/src/main/java/org/apache/ambari/server/serveraction/upgrades/ConfigureAction.java#L200

I would recommend you to downgrade and then attempt the upgrade again as upgrade is just started (3% completed)

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

ilyal — Thu, 09 Aug 2018 02:50:10 GMT

My Ambari 2.6.2.2. I tried to downgrade and start an upgrade again. Same issue all the time.

I had before this issue:https://community.hortonworks.com/questions/211829/upgrade-hdf-310-to-hdf-312.html . And maid manual changes in DB (only stack_id update) Maybe I need change more things?

Thanks

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

ilyal — Thu, 09 Aug 2018 03:11:00 GMT

May be I need use ambari-server set-current cli command and not update in database?

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

apappu — Thu, 09 Aug 2018 03:35:29 GMT

@Ilya Li looks like you have null reference in upgrade_history table.

Can you review the entries from below query and see if from_repo_version_id, target_repo_version_id are NULL.

select * from upgrade_history; //Look for zookeeper entry and correct that to valid repo version id.

After correcting that you would need to restart ambari server before attempting the upgrade again.

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

apappu — Thu, 09 Aug 2018 03:36:36 GMT

@Ilya Li looks like you have null reference in upgrade_history table.

Can you review the entries from below query and see if from_repo_version_id, target_repo_version_id are NULL.

select * from upgrade_history; //Look for zookeeper entry and correct that to valid repo version id.

After correcting that you would need to restart ambari server before attempting the upgrade again.

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

ilyal — Thu, 09 Aug 2018 03:44:56 GMT

In this table, all values were set correctly. No nulls.

Maybe this table work as log?

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

apappu — Thu, 09 Aug 2018 04:01:05 GMT

@Ilya Li

code is reading from that table - check whether it has valid/correct "from_repo_version_id" and "target_repo_version_id"

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

akhilsnaik — Thu, 09 Aug 2018 10:54:26 GMT

Hi @Ilya Li ,

Can you please have a look at this table

select * from repo_version;
select * from stack;

I hope you will get some clue.

probably the stack_id of newly registered 3.1.2 will be pointing towards the wrong stack in stack table.

As Amar suggested I would suggest you downgrade the cluster

do the service checks on the zookeeper. change some configs in zookeeper and make sure it works. and then proceed with upgrade.

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

ilyal — Sun, 18 Aug 2019 02:58:37 GMT

I made downgrade, service check on the zookeeper. Nothing helped. Which configs do I need to change in zookeeper?

stack:

repo_version:

cluster:

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

akhilsnaik — Thu, 09 Aug 2018 12:58:06 GMT

Hi @Ilya Li,

Oh this commands everything seems to be ok.

Can you investigate further with this commands :

1) SELECT id, component_id, repo_version_id, state, user_name
FROM ambari.servicecomponent_version;
2) select * from servicecomponentdesiredstate where service_name='ZOOKEEPER';
3) select * from repo_version where repo_version_id in ( select desired_repo_version_id from servicecomponentdesiredstate where service_name='ZOOKEEPER');
4)  select * from repo_version where repo_version_id in ( select desired_repo_version_id from servicecomponentdesiredstate);
5)  select * from servicecomponentdesiredstate where desired_repo_version_id not in ( select repo_version_id from repo_version);

If you happened to see two repo_versions in command 3 and 4 or get some output for command 5 ( which is empty in my env ) you can suspect that's the wrong entry in database and act accordingly.

Hope this helps.

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

ilyal — Sun, 18 Aug 2019 02:58:18 GMT

This is the last upgrade_history

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

ilyal — Sun, 18 Aug 2019 02:58:10 GMT

3) -4)

5) empty
Any ideas?

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

ilyal — Thu, 09 Aug 2018 16:28:12 GMT

Very strange issue. When I downgrade in cluster table I see desired_stack_id=51, but need to be 102.

How ambari decide to which stack to downgrade? in update history all good

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

akhilsnaik — Fri, 10 Aug 2018 00:29:05 GMT

Hi @Ilya Li ,

Looks like thats the issue.

can you make it 102 and try again ?

Hope this helps.

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

ilyal — Fri, 10 Aug 2018 12:11:18 GMT

After each downgrade I changed it to 102. but this is not help.

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

akhilsnaik — Fri, 10 Aug 2018 13:33:51 GMT

you might need to analyze your tables one by one in ambari where it fetchs the HDF-3.0 info.

some useful tables are metainfo ,hostcomponentstate + some tables i mentioned in my previous comment

Re: Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

ilyal — Tue, 09 Oct 2018 20:24:57 GMT

In the end, I created the new cluster with HDF 3.2.0.

With NiFi Registry made it very fast.

Thanks to all which tried to help.