Support Questions

Find answers, ask questions, and share your expertise

Cannot select * from tables in hive

Explorer

After upgrading to a new version of CM, I am unable to list my tables in hive. 

 

select * from prod_83.std_clnt_sku limit 5;

Error: Error while compiling statement: FAILED: SemanticException Unable to determine if hdfs://nameservice1/user/hive/warehouse/prod_83.db/std_clnt_sku is encrypted: java.lang.IllegalArgumentException: Wrong FS: hdfs://nameservice1/user/hive/warehouse/prod_83.db/std_clnt_sku, expected: hdfs://zeus.corpdom.com:8020 (state=42000,code=40000)

 

The Location of the database is listed as:

hdfs://nameservice1/user/hive/warehouse/prod_83.db/std_clnt_sku

 

I can see it inside hdfs as 

drwxrwx--x+ - analytics sentry_rwx 0 2016-10-17 09:49 /user/hive/warehouse/prod_83.db/std_clnt_sku

 

Can you help me solve this problem?

Thank You

Jamie

1 ACCEPTED SOLUTION

Champion

Did you enable High Availability by any chance?

Did you chance your dfs.nameservices by any chance?

View solution in original post

19 REPLIES 19

Champion

@cpuengr  Pls check with your admin that any security implemented as part of upgrade. I can see the keyword sentry in your hdfs path. so you might need a proper role level access to query your table. Also + denotes ACL implemention. Are you using same login for your hive and hdfs? In case if you are using Hue for Hive

 

You will get very highlevel information about security in this path

https://community.cloudera.com/t5/Security-Apache-Sentry/Hadoop-Security-for-beginners/m-p/48576

 

Thanks

Kumar

Explorer

It looks to me like all security is fine and I should have access to these databases. 

 

hdfs dfs -getfacl /user/hive/warehouse
# file: /user/hive/warehouse
# owner: hive
# group: sentry_rwx
user::rwx
group::rwx
group:admin:rwx
group:sentry_rwx:rwx

 

id uname
uid=1009(uname) gid=1011(uname) groups=1006(admin),1011(uname)

 

Is there somewhere else I need to look?

 

Thank You

Champion

@cpuengr

 

Did you rename your cluster as part of upgrade? By default the name is nameservice1. Did you modify it as zeus.corpdom.com by any chance? 

 

Login to CM -> Left Top green button name

 

Also check your table location from the below command. 
describe formatted tablename;

 

Thanks

Kumar

Explorer

The name of my cluster has always been gods. I did not change it, and it is not nameservice1.

 

Champion

Did you enable High Availability by any chance?

Did you chance your dfs.nameservices by any chance?

Champion

To make it more easier


CM -> HDFS -> Configuration -> NameNode Nameservice
I assume it was "nameservice1" before and "zeus.corpdom.com now".

 

Also did you check the location from "describe formatted tablename" in Hive. If the location refers to nameservice1 then change either NameNode Nameservice name or Hive table location. If you have very less tables, then I would recommand to change the hive table location to match with NameService.

 

Hope this helps!

Explorer

Before the upgrade the cluster was HA, but after it will not let me set up the HA anymore. It broke it apart, and I am trying to figure that out as well. 

 

I looked at the name of the dfs.nameservices and they were empty so I now have nameservice1 in there and it still is not working. 

 

Thank you

Jamie

Champion

I still didn't get answer for describe formatted tablename!!

Explorer

I posted that above in another comment. The database is listed here..

hdfs://nameservice1/user/hive/warehouse/prod_83.db/std_clnt_sku

Champion

Since you update nameservice just now, it may require a restart! not sure... pls query after cluster restart! it may help

Explorer

I have restarted the cluster and I am still seeing 

select * from nz_test;
Error: Error while compiling statement: FAILED: SemanticException Unable to determine if hdfs://nameservice1/user/hive/warehouse/nz_test is encrypted: java.lang.IllegalArgumentException: Wrong FS: hdfs://nameservice1/user/hive/warehouse/nz_test, expected: hdfs://zeus.corpdom.com:8020 (state=42000,code=40000)

 

I dont understand why the expected result is not changing.

 

THank You

Jamie

Explorer

Would the HA have anything to do with this? Before it had a master and secondary master and now it has a namenode and secondary name node. I am going to try and set up HA again and see what happens.

 

Thank You

Champion

Champion

If not aware already, do not get confuse with /apps/hive/warehouse/ 

 

in Cloudera Hive meta store databases are usually created under /user/hive/warehouse where as in Hortonworks distribution it is usually under /apps/hive/warehouse/

 

bottom line of that link is to make sure the Hive Warehouse Directory should be match in all possible areas

Explorer

I do not have that metatool on the metastore anywhere. Is it installed with Cloudera Manager?

Thank You

Champion

 

From your examples, i hope you are getting this issue for almost all the tables....

 

1. Can you try to access the table from Impala?

2. Pls do not forget to execute "invalidate metadata table"

3. Are you getting this issue for only existing tables? Can you create a new table in hive and test it? if new table works, compare the configuration difference between old & new tables

Explorer

Yes that is correct all tables show same results. 

I can create a new table and the location is set to zeus.corpdom.com and I can view that table fine. It is looking like this is coming from the cluster not in HA anymore. I believe the namespace1 is available only when you have 2 masters. I will be putting this cluster into HA mode tomorrow to try that out. 

I have not tried Impala yet, since I do not have that installed. I can install that as well and try it out.

 

Thank you so much

Jamie

Explorer

I re-enabled HA and these queries started working again.

Thank You so much

Jamie

Champion

@cpuengr happy to hear it is working now!!