Support Questions

Find answers, ask questions, and share your expertise

Potential misconfiguration detected - Hue - Hive Editor - HiveServer2

Explorer

Dear Friends

 

We have just upgraded our CM and CDH from 5.2 to 5.3 successfully. Hue was also working fine.

 

  • Today, when I login to Hue, I got the following message:

Potential misconfiguration detected. Fix and restart Hue. Hive Editor: The application won't work without a running HiveServer2.

 

  • Then I clicked on the Hive editor link in Hue, the database area just try to refresh the sample table list and running a simple select statement generates the error below. Would you please help?

Your query has the following error(s):java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

The strange thing is, if I login with another user to Hue, it do not show me this message above and also I can run the sample query in Hive error fine.

 

I can see other people have the same issue as below:

http://community.cloudera.com/t5/Apache-Hadoop-Concepts-and/Problem-with-CDH-5-VM-quot-Potential-mis...

 

 

P.S. Manually refreshing the table in Hive editor did not work and even reinstalling the same tables in Hue generated the following info:

Could not install table: Error creating table sample_07: Bad status for request TExecuteStatementReq(confOverlay={}, sessionHandle=TSessionHandle(sessionId=THandleIdentifier(secret='e\(replaced by me for security reason J)', guid='\x(again replaced by me for security reason J)')), runAsync=True, statement="CREATE TABLE `sample_07` (\n `code` string ,\n `description` string ,\n `total_emp` int ,\n `salary` int )\nROW FORMAT DELIMITED\n FIELDS TERMINATED BY '\t'\nSTORED AS TextFile"): TExecuteStatementResp(status=TStatus(errorCode=10072, errorMessage='Error while compiling statement: FAILED: SemanticException [Error 10072]: Database does not exist: default', sqlState='42000', infoMessages=None, statusCode=3), operationHandle=None).

 

Thanks much for your support and please let me know if you have any question.

 

Kind regards

Andy

1 ACCEPTED SOLUTION

Explorer

Thanks Romain for your time and your answer.

 

I have restarted CDH and the isused has fixed! This is strange but still a very good news 🙂

 

I have not changed any configuration and have had restarted the service many times before. Anyway, I will wait couple of more days and then update the case. Thanks again. 

 

Kind regards

Andy

View solution in original post

16 REPLIES 16

This is weird. Are you using Sentry maybe? (and the user with error does
not have access to the 'default' database?)

Romain

Explorer

Thanks Romain 

 

We do not have Sentry service added in CM but I can see "sentry table'' item under "hadoop security" in Hue and when I select it, it generates the following error: Could not connect to localhost:8038 (code THRIFTTRANSPORT): TTransportException('Could not connect to localhost:8038',)

 

Can you help me verify if my user have access to default database as you mentioned? I use my network AD user to login to Hue using Kerberos/LDAP and we set Hue up for this purpose. 

 

P.S. The strange thing is, yesterday, when I logged in to Hue, I did not see the error message but today, I got the same Hive error message again. and as mentioned before, this error is new, right after we have upgraded both CDH and CM from 5.2 to 5.3 last week. 

 

Appreciate your support and case. Please let me know if you have any question. 

 

Kind regards

Andy

This is typical of a misconfigured HiveServer2:

Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

It can be that the remote metastore DB is readonly, or is not found

I usually recommend to debug it by using beeline:
http://www.cloudera.com/content/cloudera/en/documentation/core/latest/topics/cdh_ig_hiveserver2_star...


Above comments about Sentry are not related, as this is just the Sentry UI
in Hue.

Romain

Explorer

Thanks much Romain

 

From the URL you kindly shared with me, I just ran beeline and I was able to get into it command line: 

then I ran the following command all in one line (not sure I need to do that) by replacing my AD username and password below:

!connect jdbc:hive2://localhost:10000 <ADUserName> <Password> org.apache.hive.jdbc.HiveDriver 0: jdbc:hive2://localhost:10000> SHOW TABLES;

but I got this error message: 

 

Connecting to jdbc:hive2://localhost:10000
Error: Could not open connection to jdbc:hive2://localhost:10000: Peer indicated failure: Unsupported mechanism type PLAIN (state=08S01,code=0)
0: jdbc:hive2://localhost:10000 (closed)>


I also try the following: 

sudo service hive-server2 start

but got the error: 
hive-server2: unrecognized service

 

Can you tell me if I missed something and how to run the above commands correctly or is there any way I can test to see if HiveServer2 is running correctly? 

Thanks much Romain. 

 

P.S.  Originally, running beeline from the default location  ((/usr/lib/hive/bin/beeline) did not work so I just ran it by typing beeline and it worked 🙂

 

Cheers

Andy

I am not a Hive expert but auth should be LDAP and not PLAIN if you use
LDAP?

https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2#SettingUpHiveServer2-Authent...

Romain

Explorer

Thanks Romain for your time and your answer.

 

I have restarted CDH and the isused has fixed! This is strange but still a very good news 🙂

 

I have not changed any configuration and have had restarted the service many times before. Anyway, I will wait couple of more days and then update the case. Thanks again. 

 

Kind regards

Andy

Explorer

hi,

i have exactly the same error and don't know what to do 

any help please ?

i'm using CDH 5.3 with a cluster of 4nodes 

Explorer

You have restarted CDH.

Do you mean by restarting CDH cluster or only Hive Metastore/Hive and Hue.

 

We are facing the same issue but that is limited to two users.

The same query can be run by other users.

 

Please confirm.

Below is the stack trace of error.

 

But, I don't see a bad configuration error in Hue. I am using CDH 5.3.2

 

2015-03-31 13:57:32,949 ERROR org.apache.thrift.server.TThreadPoolServer: Error occurred during processing of message.

java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: DIGEST-MD5: IO error acquiring password

        at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219)

        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:724)

        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge20S.java:721)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:356)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1622)

        at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge20S$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge20S.java:721)

        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:227)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:724)

Expert Contributor

There is an issue with hive where HS2 has problems renewing tokens after the renewal period and it throws the error you are seeing.  If you restart the Hive service, Hue will probably start working again for a day or two.  You can alleviate this by adding the following to the HMS and HS2 safety valves:

 

<property>
<name>hive.cluster.delegation.key.update-interval</name>
<value>31536000000</value>
</property>
<property>
<name>hive.cluster.delegation.token.renew-interval</name>
<value>31536000000</value>
</property>
<property>
<name>hive.cluster.delegation.token.max-lifetime</name>
<value>31536000000</value>
</property>

 

This will set the token renewal to 1 year, so this will only happen once a year.

 

Hope this helps.

Explorer

Thank you for the reply. I have added the setting in Test and will update Prod tomorrow then in a time span of two days. we will know the throughput.

Explorer

This worked and the issue has been fixed now.

Only I am looking for an option to set an alert for 365 days for this configuration.

Started another thread already.

Explorer

Allright, we will come to know in 2-3 days. We already have restarted the HiveServer2, hence that has renewed the tickets/toekn. So we will know on 6th of April as soon as our users start completing 2-3 days of token creation.

Explorer

Had the settings go to HiveServer2 only or Metastore safety valve as well?

 

We are getting the issues again.

Guru

Looks like this thread is old, but I would like to add that this was a HiveMetaStore issue, so it should be going into HiveMetaStore safety valve.

 

Cheers

Contributor

@cconnerI'm sorry for digging up this issue after that long time, but we hit exactly the same problem. Following your suggestion we tried set delegation tokens for HS2 and HM, as well as for the Hive service entirely, but in the logs we found:

 

WARN org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.cluster.delegation.key.update-interval does not exist

WARN org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.cluster.delegation.token.max-lifetime does not exist

WARN org.apache.hadoop.hive.conf.HiveConf: [main]: HiveConf of name hive.cluster.delegation.token.renew-interval does not exist

 

We are using CDH 5.9.3. Is this parameter still supported in this version? I see these constants exist in the code with default codes. However, searching for solution in the Internet I found that there is another parameter hive.conf.validation that checks if 'hive.' variables are accepted: https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties

Default Value: true

Added In: Hive 0.10.0 with HIVE-2848

Enables type checking for registered Hive configurations. As of Hive 0.14.0 (HIVE-7211), a configuration name that starts with "hive." is regarded as a Hive system property. With hive.conf.validation true (default), any attempts to set a configuration property that starts with "hive." which is not registered to the Hive system will throw an exception.

 

Is it our case? Or maybe warnings we see in the logs are misleading and actually hive.cluster.delegation.key.* are updated? But we still see in HMS logs messages about new tokens being created/cancelled. I'll be thankful for any clarifications.

Guru
Looks like you already have another thread opened:

http://community.cloudera.com/t5/Batch-SQL-Apache-Hive/Hive-Safety-Valve-configuration-is-not-applie...

Will follow up there.