Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Upgrade to HDP 3.0 hangs on ”Move Hive Table”

Solved Go to solution

Upgrade to HDP 3.0 hangs on ”Move Hive Table”

Contributor

I’m trying to migrate to HDP 3.0, but the upgrade hangs on “Move Hive Table”. When I look at the log, the actual move command just exit without any error messages and with exit code 255. This happens even when I try to run the command manually. So it’s kind of hard to understand what the real problem is as I get no output at all from the command. Only time I get something in return is when I add the -h and I get the help output.

/usr/hdp/3.0.0.0-1634/hive/bin/hive --config /etc/hive/conf --service strictmanagedmigration --hiveconf hive.strict.managed.tables=true -m automatic --modifyManagedTables --oldWarehouseRoot /apps/hive/warehouse

Can anybody help me to better understand why it’s exiting with code 255 and if possible how to solve it.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Upgrade to HDP 3.0 hangs on ”Move Hive Table”

Contributor

Problem solved

If you have created a hbase or phoenix table through hive as an “internal” table, it will be created as a managed table with the storage handler towards hbase/phoenix. This is what’s causing the problem. Managed tables with the hbase/phoenix storagehandler won’t work in the “Move Hive Tables” part of the upgrade (external tables works ofc). I had to manually remove those tables from the Hive metadatabase and then the “Move Hive Tables” part of the upgrade works fine.

6 REPLIES 6

Re: Upgrade to HDP 3.0 hangs on ”Move Hive Table”

Hi @Berry Österlund ,

I think you will get a good idea about the issue from the error logs itself.

exit code 255 is just to inform the task which ambari is performing as part of upgrade is failed.

Seeing the command it seems you have have upgraded to /usr/hdp/3.0.0.0-1634 .

Can you try to mannually run the command on the node which demands to run and see if you get some clue from it ?

Re: Upgrade to HDP 3.0 hangs on ”Move Hive Table”

Are you using MySQL or MariaDB as db for hive? Can you see any error in /var/log/hive/hivemetastore.log?
If you see something like this in the metastore log;

java.lang.NullPointerException
org.apache.hadoop.hive.metastore.HiveMetaException: Upgrade FAILED! Metastore state would be inconsistent

try to source the following manually from sql cli (the metastore log should indicate the from/to versions) and restart the metastore server

source /usr/hdp/3.0.0.0-1634/hive/scripts/metastore/upgrade/upgrade-<version you're upgrading from>-to-<version you're upgrading to>.mysql.sql

Re: Upgrade to HDP 3.0 hangs on ”Move Hive Table”

Contributor

I'm using MySQL for the metastore. No errors in the log

Re: Upgrade to HDP 3.0 hangs on ”Move Hive Table”

Contributor

Update: It looks like it's related to one or more tables. I have around 50 databases in Hive. I selected each and every one of them with the -d flag (regexp of databasename) and I only get the 255 exit code on 4 of the databases. All the others are working fine.

I will now try to pinpoint exactly what tables in those databases are causing the error and see if I can find anything strange with them.

Re: Upgrade to HDP 3.0 hangs on ”Move Hive Table”

Contributor

Problem solved

If you have created a hbase or phoenix table through hive as an “internal” table, it will be created as a managed table with the storage handler towards hbase/phoenix. This is what’s causing the problem. Managed tables with the hbase/phoenix storagehandler won’t work in the “Move Hive Tables” part of the upgrade (external tables works ofc). I had to manually remove those tables from the Hive metadatabase and then the “Move Hive Tables” part of the upgrade works fine.

Re: Upgrade to HDP 3.0 hangs on ”Move Hive Table”

New Contributor

@Berry Osterlund : was this issue resolved ? Do you have a solution if it's resolved. Even I am getting the error code 255 , however there is no errors thrown

Don't have an account?
Coming from Hortonworks? Activate your account here