Potential misconfiguration detected - Hue - Hive Editor - HiveServer2

[#1]

Dear Friends

We have just upgraded our CM and CDH from 5.2 to 5.3 successfully. Hue was also working fine.

• Today, when I login to Hue, I got the following message:

Potential misconfiguration detected. Fix and restart Hue. Hive Editor: The application won't work without a running HiveServer2.

• Then I clicked on the Hive editor link in Hue, the database area just try to refresh the sample table list and running a simple select statement generates the error below. Would you please help?

Your query has the following error(s):java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

The strange thing is, if I login with another user to Hue, it do not show me this message above and also I can run the sample query in Hive error fine.

I can see other people have the same issue as below:

http://community.cloudera.com/t5/Apache-Hadoop-Concepts-and/Problem-with-CDH-5-VM-quot-Potential-mis...

P.S. Manually refreshing the table in Hive editor did not work and even reinstalling the same tables in Hue generated the following info:

Could not install table: Error creating table sample_07: Bad status for request TExecuteStatementReq(confOverlay={}, sessionHandle=TSessionHandle(sessionId=THandleIdentifier(secret='e\(replaced by me for security reason J)', guid='\x(again replaced by me for security reason J)')), runAsync=True, statement="CREATE TABLE `sample_07` (\n `code` string ,\n `description` string ,\n `total_emp` int ,\n `salary` int )\nROW FORMAT DELIMITED\n FIELDS TERMINATED BY '\t'\nSTORED AS TextFile"): TExecuteStatementResp(status=TStatus(errorCode=10072, errorMessage='Error while compiling statement: FAILED: SemanticException [Error 10072]: Database does not exist: default', sqlState='42000', infoMessages=None, statusCode=3), operationHandle=None).

Thanks much for your support and please let me know if you have any question.

Kind regards

Andy

[#2]

This is weird. Are you using Sentry maybe? (and the user with error does
not have access to the 'default' database?)

Romain

[#3]

Thanks Romain 

We do not have Sentry service added in CM but I can see "sentry table'' item under "hadoop security" in Hue and when I select it, it generates the following error: Could not connect to localhost:8038 (code THRIFTTRANSPORT): TTransportException('Could not connect to localhost:8038',)

Can you help me verify if my user have access to default database as you mentioned? I use my network AD user to login to Hue using Kerberos/LDAP and we set Hue up for this purpose. 

P.S. The strange thing is, yesterday, when I logged in to Hue, I did not see the error message but today, I got the same Hive error message again. and as mentioned before, this error is new, right after we have upgraded both CDH and CM from 5.2 to 5.3 last week. 

Appreciate your support and case. Please let me know if you have any question. 

Kind regards

Andy

[#4]

This is typical of a misconfigured HiveServer2:

Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

It can be that the remote metastore DB is readonly, or is not found

I usually recommend to debug it by using beeline:
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_hiveserver2_star...


Above comments about Sentry are not related, as this is just the Sentry UI
in Hue.

Romain

[#5]

Thanks much Romain

From the URL you kindly shared with me, I just ran beeline and I was able to get into it command line: 

then I ran the following command all in one line (not sure I need to do that) by replacing my AD username and password below:

!connect jdbc:hive2://localhost:10000 <ADUserName> <Password> org.apache.hive.jdbc.HiveDriver 0: jdbc:hive2://localhost:10000> SHOW TABLES;

but I got this error message: 

Connecting to jdbc:hive2://localhost:10000
Error: Could not open connection to jdbc:hive2://localhost:10000: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
0: jdbc:hive2://localhost:10000 (closed)>

I also try the following: 

sudo service hive-server2 start

but got the error: 
hive-server2: unrecognized service

Can you tell me if I missed something and how to run the above commands correctly or is there any way I can test to see if HiveServer2 is running correctly? 

Thanks much Romain. 

P.S.  Originally, running beeline from the default location  ((/usr/lib/hive/bin/beeline) did not work so I just ran it by typing beeline and it worked 🙂

Cheers

Andy

[#6]

I am not a Hive expert but auth should be LDAP and not PLAIN if you use
LDAP?

https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2#SettingUpHiveServer2-Authent...

Romain

[#7 · Accepted solution]

Thanks Romain for your time and your answer.

I have restarted CDH and the isused has fixed! This is strange but still a very good news 🙂

I have not changed any configuration and have had restarted the service many times before. Anyway, I will wait couple of more days and then update the case. Thanks again. 

Kind regards

Andy

[#8]

hi,

i have exactly the same error and don't know what to do 

any help please ?

i'm using CDH 5.3 with a cluster of 4nodes 

[#9]

You have restarted CDH.

Do you mean by restarting CDH cluster or only Hive Metastore/Hive and Hue.

We are facing the same issue but that is limited to two users.

The same query can be run by other users.

Please confirm.

Below is the stack trace of error.

But, I don't see a bad configuration error in Hue. I am using CDH 5.3.2

2015-03-31 13:57:32,949 ERROR org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message.

java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: DIGEST-MD5: IO error acquiring password

        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)

        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:724)

        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:721)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:356)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)

        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:721)

        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:227)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:724)

[#10]

There is an issue with hive where HS2 has problems renewing tokens after the renewal period and it throws the error you are seeing.  If you restart the Hive service, Hue will probably start working again for a day or two.  You can alleviate this by adding the following to the HMS and HS2 safety valves:

<property>
<name>hive.cluster.delegation.key.update-interval</name>
<value>31536000000</value>
</property>
<property>
<name>hive.cluster.delegation.token.renew-interval</name>
<value>31536000000</value>
</property>
<property>
<name>hive.cluster.delegation.token.max-lifetime</name>
<value>31536000000</value>
</property>

This will set the token renewal to 1 year, so this will only happen once a year.

Hope this helps.