Upgrade to HDP 3.0. it show 'Waiting to execute the next task'

[#1]

HDP upgrade ui always show 'Waiting to execute the next task'. When I do upgradtion, it failed on 'Move hive tables' step, so I tried to remove row in ambari db for this task. And after roll back the database, the upgrade can't continue, anyone can help me?

[#2]

HI @XM, It seems you are stuck at upgrade .

Can you please pause the upgrade and resume the upgrade , see if this works ?

If it doesnt you might need to see which task is ambari waiting for you can use this command in DB to see which Task is state wich is not in completed,Aborted state:

select distinct(status) from host_role_command; 

If some tasks are instatus= HOLDING or HOLDING_TIMEOUT stage you might need to make it to completed or Aborted.

Additionally can you please tell which is the operation you performed to remove a row from ambari_db. it's not suggested usually to remove a ROW in database instead you can change its status to completed or Aborted.

Hope this helps.

[#3]

Thanks for your reply. I removed 'Move Hive Tables' since it failed when I upgrade the HDP 3.0. I manually did run the command to finish this task.

Another question, can we manually finalize the cluster and make upgrade as complete. Since I have upgrade all componets and most of components are working now. I want to mark upgrade as complete and fix components one bye one if it has issue.

[#4]

HI @XM ,

No there is no way you can manually mark the upgrade as complete, unless you are editing the database.

you can explore the Tables host_role_command and see if there is any task pending stage as I mentioned in the previous comment.

and carefully change those to completed stage might help.

Take ambari backup before the mannual editings. and pause upgrade and stop ambari-server while operation.

[#5]

Thanks akhil. I will try to let task continue per your suggestion. Can you tell me which table/row need to mark as complete to let upgrade complete?

[#6]

There are four 'ABORTED', 'COMPLETED', 'FAILED', 'SKIPPED_FAILED' status in ambari db. But not see HOLDING or HOLDING_TIMEOUT.

When I try to trigger resume upgrade, the log show those message. Any suggestion, Thanks.

ERROR [ambari-client-thread-481] TaskStatusListener:138 - Received update for a task 20232 which is not being tracked as running task 
ERROR [ambari-client-thread-481] TaskStatusListener:138 - Received update for a task 20233 which is not being tracked as running task 
ERROR [ambari-client-thread-481] TaskStatusListener:138 - Received update for a task 20234 which is not being tracked as running task

[#7 · Accepted solution]

hi @XM ,

It seems you might have removed some tasks manually from the database.

can you please execute following commands and see if it helps you.

select * from host_role_command where stage_id in (select stage_id from upgrade_item) order by task_id;
select * from upgrade_item where stage_id not in (select distinct(stage_id) from host_role_command);
select * from upgrade_group;
select * from upgrade_item;

the upgrade_item table contains all the subtasks of upgrade and host_role_command contains all the tasks. both are mapped with stage_id

Please use more specific scripts than '*' as i gave if you are using the command line to inspect queries.

I use PGAdmin 4 a UI tool which gives a better idea of the tables.

There is another simple way which I can suggest downgrade and upgrade again and don't modify Database unless you take backup.

hope this Comment helps you.

[#8]

If I mark upgrade request/stage/command as 'COMPLETED', the upgrade will be as completed automatically? Or I must set to complete somewhere? If yes, where can I set to 'completed' in database.