Upgrade HDF 3.1.1.0 to HDF 3.1.2 Rolling Upgrade fail

[#1]

Hi All,

I try run rolling upgrade with subject versions.

And it fails with the very strange message. I added the image with current state

and in ambari-server.log I see this lines:

WARN [Server Action Executor Worker 2015] ServerActionExecutor:471 - Task #2015 failed to complete execution due to thrown exception: java.lang.NullPointerException:null
java.lang.NullPointerException
        at org.apache.ambari.server.serveraction.upgrades.ConfigureAction.execute(ConfigureAction.java:200)
        at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.execute(ServerActionExecutor.java:550)
        at org.apache.ambari.server.serveraction.ServerActionExecutor$Worker.run(ServerActionExecutor.java:466)
        at java.lang.Thread.run(Thread.java:745)

Please help.

Thanks,

Ilya

[#2]

@Ilya Li

Are you using Ambari 2.6.2 version?

For some reason your source repository is coming as null - you can refer at https://github.com/apache/ambari/blob/release-2.6.2-rc0/ambari-server/src/main/java/org/apache/ambar...

I would recommend you to downgrade and then attempt the upgrade again as upgrade is just started (3% completed)

[#3]

My Ambari 2.6.2.2. I tried to downgrade and start an upgrade again. Same issue all the time.

I had before this issue:https://community.hortonworks.com/questions/211829/upgrade-hdf-310-to-hdf-312.html . And maid manual changes in DB (only stack_id update) Maybe I need change more things?

Thanks

[#4]

May be I need use ambari-server set-current cli command and not update in database?

[#5]

@Ilya Li looks like you have null reference in upgrade_history table.

Can you review the entries from below query and see if from_repo_version_id, target_repo_version_id are NULL.

select * from upgrade_history; //Look for zookeeper entry and correct that to valid repo version id.

After correcting that you would need to restart ambari server before attempting the upgrade again.

[#6]

In this table, all values were set correctly. No nulls.

Maybe this table work as log?

[#7]

@Ilya Li

code is reading from that table - check whether it has valid/correct "from_repo_version_id" and "target_repo_version_id"

[#8]

Very strange issue. When I downgrade in cluster table I see desired_stack_id=51, but need to be 102.

How ambari decide to which stack to downgrade? in update history all good

[#9]

Hi @Ilya Li ,

Looks like thats the issue.

can you make it 102 and try again ?

Hope this helps.

[#10]

After each downgrade I changed it to 102. but this is not help.

[#11]

you might need to analyze your tables one by one in ambari where it fetchs the HDF-3.0 info.

some useful tables are metainfo ,hostcomponentstate + some tables i mentioned in my previous comment

[#12]

@Ilya Li looks like you have null reference in upgrade_history table.

Can you review the entries from below query and see if from_repo_version_id, target_repo_version_id are NULL.

select * from upgrade_history; //Look for zookeeper entry and correct that to valid repo version id.

After correcting that you would need to restart ambari server before attempting the upgrade again.

[#13]

This is the last upgrade_history

[#14]

Hi @Ilya Li ,

Can you please have a look at this table

select * from repo_version;
select * from stack;

I hope you will get some clue.

probably the stack_id of newly registered 3.1.2 will be pointing towards the wrong stack in stack table.

As Amar suggested I would suggest you downgrade the cluster

do the service checks on the zookeeper. change some configs in zookeeper and make sure it works. and then proceed with upgrade.

[#15]

I made downgrade, service check on the zookeeper. Nothing helped. Which configs do I need to change in zookeeper?

stack:

repo_version:

cluster:

[#16]

Hi @Ilya Li,

Oh this commands everything seems to be ok.

Can you investigate further with this commands :

1) SELECT id, component_id, repo_version_id, state, user_name
FROM ambari.servicecomponent_version;
2) select * from servicecomponentdesiredstate where service_name='ZOOKEEPER';
3) select * from repo_version where repo_version_id in ( select desired_repo_version_id from servicecomponentdesiredstate where service_name='ZOOKEEPER');
4)  select * from repo_version where repo_version_id in ( select desired_repo_version_id from servicecomponentdesiredstate);
5)  select * from servicecomponentdesiredstate where desired_repo_version_id not in ( select repo_version_id from repo_version);

If you happened to see two repo_versions in command 3 and 4 or get some output for command 5 ( which is empty in my env ) you can suspect that's the wrong entry in database and act accordingly.

Hope this helps.

[#17]

1)

2)

3) -4)

5) empty
Any ideas?

[#18 · Accepted solution]

In the end, I created the new cluster with HDF 3.2.0.

With NiFi Registry made it very fast.

Thanks to all which tried to help.