question Re: [CDH 5.4 Upgrade] cannot select from Hive tables in Archives of Support Questions (Read Only)

[CDH 5.4 Upgrade] cannot select from Hive tables

TS — Fri, 16 Sep 2022 09:28:04 GMT

I have upgraded from CDH 5.3 to CDH 5.4.

While executing a simple select statement in Hive it is erroring out:

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

hive> select * from employees;
FAILED: SemanticException Unable to determine if hdfs://<hive-master>:8020:8020/user/hive/warehouse/employees is encrypted:
java.lang.IllegalArgumentException: Wrong FS: hdfs://<hive-master>:8020:8020/user/hive/warehouse/employees, expected: hdfs://<hive-master>:8020

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

TS — Wed, 06 May 2015 14:29:04 GMT

Port 8020 is used by NameNode.

I am not sure why the HDFS path has it twice!!!

Does it pick up the "fs.defaultFS" property from HDFS service?

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

jxiang — Wed, 06 May 2015 14:41:12 GMT

What's your setting "hive.metastore.warehouse.dir"?

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

TS — Wed, 06 May 2015 14:49:39 GMT

/user/hive/warehouse

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

jxiang — Wed, 06 May 2015 16:57:12 GMT

What's your setting "fs.defaultFS"? What's the output of "describe
formatted employees"?

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

TS — Wed, 06 May 2015 17:12:27 GMT

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://<FQN-host-name>:8020</value>
</property>

.................................

hive> describe formatted employees;
OK
# col_name data_type comment

emp_id                  int
name                    string
salary                  double

# Detailed Table Information
Database:               default
Owner:                  dast
CreateTime:             Thu Apr 09 14:57:46 EDT 2015
LastAccessTime:         UNKNOWN
Protect Mode:           None
Retention:              0
Location:               hdfs://<host-name>:8020:8020/user/hive/warehouse/employees
Table Type:             MANAGED_TABLE
Table Parameters:
        COLUMN_STATS_ACCURATE   true
        comment                 This is the employees table
        numFiles                1
        numRows                 0
        rawDataSize             0
        totalSize               142
        transient_lastDdlTime   1428607184

# Storage Information
SerDe Library:          org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat:            org.apache.hadoop.mapred.TextInputFormat
OutputFormat:           org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
Compressed:             No
Num Buckets:            -1
Bucket Columns:         []
Sort Columns:           []
Storage Desc Params:
        field.delim             ,
        serialization.format    ,
Time taken: 0.405 seconds, Fetched: 35 row(s)

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

jxiang — Wed, 06 May 2015 17:22:12 GMT

For this table, you can use alter table employees set location '
hdfs://:8020/user/hive/warehouse/employees' to fix it. Do you
see the problem for other tables? If you create a new table, does it have
this problem too?

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

TS — Wed, 06 May 2015 17:35:37 GMT

I have just created a brand new table.

The HDFS location/path has the port 8020 ONE time!!!!

How can I revert the existing tables back to report port 8020 one time and not twice????

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

hive> select * from users;
OK
100      User1   passwd1
200      User2   passwd2
300      User3   passwd3
400      User4   passwd4
500      User5   passwd5
600      User6   passwd6
Time taken: 0.073 seconds, Fetched: 6 row(s)
hive> describe formatted users;
OK
# col_name              data_type               comment

user_id                 int
username                string
passwd                  string

# Detailed Table Information
Database:               default
Owner:                  dast
CreateTime:             Wed May 06 13:32:05 EDT 2015
LastAccessTime:         UNKNOWN
Protect Mode:           None
Retention:              0
Location:               hdfs://<host-name>:8020/user/hive/warehouse/users
Table Type:             MANAGED_TABLE
Table Parameters:
        COLUMN_STATS_ACCURATE   true
        comment                 This is the users table
        numFiles                1
        totalSize               121
        transient_lastDdlTime   1430933556

<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

TS — Wed, 06 May 2015 17:40:23 GMT

All the existing Hive tables showing up the 8020 port twice in their HDFS Location!!!

What might have caused this, during the CDH 5.4 upgrade???

Thanks for your assistance!

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

jxiang — Wed, 06 May 2015 17:44:11 GMT

If you don't see the problem with tables created after the upgrade, it
could be because something went wrong during upgrade.

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

TS — Wed, 06 May 2015 17:51:02 GMT

Thank you!

So , the only way to overcome the issue is to execute the 'alter table...' command!

I have to do that to ALL existing tables!

Any other way?

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

jxiang — Wed, 06 May 2015 17:57:11 GMT

You are welcome. This is the safest method. You may want to come up a
script to do so.

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

jxiang — Mon, 11 May 2015 16:13:16 GMT

BTW, if you can let us know how you did the upgrade, it should help us to find out the root cause of the problem. Thanks.

Re: [CDH 5.4 Upgrade] cannot select from Hive tables

albertozgz — Fri, 13 Nov 2015 10:41:41 GMT

We have the same problem, we upgraded from Cloudera 5.3.8 to 5.4.8 usign parcels

<!-- fs.default.name, fs.defaultFS = 8020 -->
<!-- hive.metastore.warehouse.dir = /user/hive/warehouse -->
 
<!-- hdfs://hd01:8020:8020/user/hive/warehouse/ -->