The Hive Metastore canary failed to create a database.

[#1]

Hive is being run on Name Node 1.

When I start Hive service everything is green and up and running:

Hive Metastore Server
HiveServer2

WebHCat Server

After few minutes Hive Metastore Server becomes unavailable with bad health:

Hive Metastore Canary 

"The Hive Metastore canary failed to create a database."

Few more information:

• we do not have any MySQL database, we are using OOTB configuration for Hive, so it is probably Postgre
• I already run command Update Hive Metastore Database Schema and Update Hive Metastore Name Nodes after installation. They are grey now.

Please advise for further steps.

Thank you

[#2]

Can someone please provide me steps how to resolve this or what should be the following debugging steps? Thank you

[#3]

@VeljkoC,

Cloudera Manager performs health tests to ensure that CDH servers are running properly.  One of these tests is creating and dropping a database in the Hive Metastore.  The health alert you are seeing indicates that that process failed.

Places to look for more information about the failure:

- Service Monitor (which issues the health check):

/var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR*

you might search for the word "canary" or "hive" in that file.

- Hive Metastore log file

/var/log/hive/hadoop-cmf-HIVE-1-HIVEMETASTORE*

Check for the word "canary" or look for WARN and ERROR messages pertaining to database names that include the string "canary".

[#4]

@bgooley  thank you for reply.

This is the output from my analysis.

/var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR

2018-10-18 10:48:15,050 WARN com.cloudera.cmf.cdh5client.hive.MetastoreClientImpl: Could not create a hive database cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_ba135b5c35747b251098753b881a186b
MetaException(message:Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=hue, access=EXECUTE, inode="/user":cloudera-scm:supergroup:drwxrwx---
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:279)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:260)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:201)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:154)

2018-10-18 10:48:15,051 WARN com.cloudera.cmon.firehose.polling.hive.HiveMetastoreCanary: Metastore hive-HIVEMETASTORE-ba135b5c35747b251098753b881a186b:Failed to create database
com.cloudera.cmf.cdhclient.common.hive.MetaException: Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=hue, access=EXECUTE, inode="/user":cloudera-scm:supergroup:drwxrwx---

I am not sure for what this permission is denied? Can you please assist what should be corrected and how?

/var/log/hive/hadoop-cmf-HIVE-1-HIVEMETASTORE

2018-10-18 10:53:32,573 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-5-thread-3]: 3: source:10.0.9.4 create_database: Database(name:cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_ba135b5c35747b251098753b881a186b, description:Cloudera Manager Metastore Canary Test Database, locationUri:/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_ba135b5c35747b251098753b881a186b, parameters:null)
2018-10-18 10:53:32,573 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-3]: ugi=hue ip=10.0.9.4 cmd=source:10.0.9.4 create_database: Database(name:cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_ba135b5c35747b251098753b881a186b, description:Cloudera Manager Metastore Canary Test Database, locationUri:/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_ba135b5c35747b251098753b881a186b, parameters:null)
2018-10-18 10:53:32,694 ERROR hive.log: [pool-5-thread-3]: Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=hue, access=EXECUTE, inode="/user":cloudera-scm:supergroup:drwxrwx---

I think that maybe there is some issue with folder? But not sure exactly with what folder?

Thank you

[#5]

@bgooley

just to add that I have following folders (if the issue is related to folders on HDFS)

[cloudera-scm@cv-nn2 ~]$ hdfs dfs -ls /user
Found 3 items
drwxr-xr-x - cloudera-scm supergroup 0 2018-10-09 08:53 /user/cloudera-scm
drwxrwx--- - cloudera-scm supergroup 0 2018-10-05 06:56 /user/history
drwxrwxr-x - hue hue 0 2018-10-05 06:57 /user/hue

[#6 · Accepted solution]

@VeljkoC,

The following shows the cause of the problem:

2018-10-18 10:53:32,694 ERROR hive.log: [pool-5-thread-3]: Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=hue, access=EXECUTE, inode="/user":cloudera-scm:supergroup:drwxrwx---

The bold text above shows that an attempt was made by the Metastore to access HDFS but the required permission for the user was not available.  In this case, hue user (used by Service Monitor) needs EXECUTE privilege on /user

We can see that this does not exist:

cloudera-scm:supergroup:drwxrwx---

The default permission on /user is drwxr-xr-x I believe so in your cluster it may have been changed for some reason.  In order to allow for listing of files in /user you will need to add EXUCUTE for other.

For example the result should look like this:

drwxr-xr-x   - hdfs  supergroup          0 2018-08-27 10:19 /user

That should help.

[#7]

Thanky you, this resolved an issue

[#8]

Hi @bgooley why user hue is writing in my case when I am using the custom username for hue.?