Upgrade to HDP 3.0 hangs on ”Move Hive Table”

[#1]

I’m trying to migrate to HDP 3.0, but the upgrade hangs on “Move Hive Table”. When I look at the log, the actual move command just exit without any error messages and with exit code 255. This happens even when I try to run the command manually. So it’s kind of hard to understand what the real problem is as I get no output at all from the command. Only time I get something in return is when I add the -h and I get the help output.

/usr/hdp/3.0.0.0-1634/hive/bin/hive --config /etc/hive/conf --service strictmanagedmigration --hiveconf hive.strict.managed.tables=true -m automatic --modifyManagedTables --oldWarehouseRoot /apps/hive/warehouse

Can anybody help me to better understand why it’s exiting with code 255 and if possible how to solve it.

[#2]

Hi @Berry Österlund ,

I think you will get a good idea about the issue from the error logs itself.

exit code 255 is just to inform the task which ambari is performing as part of upgrade is failed.

Seeing the command it seems you have have upgraded to /usr/hdp/3.0.0.0-1634 .

Can you try to mannually run the command on the node which demands to run and see if you get some clue from it ?

[#3]

Are you using MySQL or MariaDB as db for hive? Can you see any error in /var/log/hive/hivemetastore.log?
If you see something like this in the metastore log;

java.lang.NullPointerException
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent

try to source the following manually from sql cli (the metastore log should indicate the from/to versions) and restart the metastore server

source /usr/hdp/3.0.0.0-1634/hive/scripts/metastore/upgrade/upgrade-<version you're upgrading from>-to-<version you're upgrading to>.mysql.sql

[#4]

I'm using MySQL for the metastore. No errors in the log

[#5]

Update: It looks like it's related to one or more tables. I have around 50 databases in Hive. I selected each and every one of them with the -d flag (regexp of databasename) and I only get the 255 exit code on 4 of the databases. All the others are working fine.

I will now try to pinpoint exactly what tables in those databases are causing the error and see if I can find anything strange with them.

[#6 · Accepted solution]

Problem solved

If you have created a hbase or phoenix table through hive as an “internal” table, it will be created as a managed table with the storage handler towards hbase/phoenix. This is what’s causing the problem. Managed tables with the hbase/phoenix storagehandler won’t work in the “Move Hive Tables” part of the upgrade (external tables works ofc). I had to manually remove those tables from the Hive metadatabase and then the “Move Hive Tables” part of the upgrade works fine.

[#7]

@Berry Osterlund : was this issue resolved ? Do you have a solution if it's resolved. Even I am getting the error code 255 , however there is no errors thrown