Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

The Hive Metastore canary failed to create a database.

Solved Go to solution
Highlighted

The Hive Metastore canary failed to create a database.

Explorer

Hive is being run on Name Node 1.

When I start Hive service everything is green and up and running:

Hive Metastore Server
HiveServer2

WebHCat Server

After few minutes Hive Metastore Server becomes unavailable with bad health:

Hive Metastore Canary 

"The Hive Metastore canary failed to create a database."

 

Few more information:

  • we do not have any MySQL database, we are using OOTB configuration for Hive, so it is probably Postgre
  • I already run command Update Hive Metastore Database Schema and Update Hive Metastore Name Nodes after installation. They are grey now.

 

Please advise for further steps.

Thank you

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: The Hive Metastore canary failed to create a database.

Super Guru

@VeljkoC,

 

The following shows the cause of the problem:

 

2018-10-18 10:53:32,694 ERROR hive.log: [pool-5-thread-3]: Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=hue, access=EXECUTE, inode="/user":cloudera-scm:supergroup:drwxrwx---

 

The bold text above shows that an attempt was made by the Metastore to access HDFS but the required permission for the user was not available.  In this case, hue user (used by Service Monitor) needs EXECUTE privilege on /user

 

We can see that this does not exist:

 

cloudera-scm:supergroup:drwxrwx---

 

The default permission on /user is drwxr-xr-x I believe so in your cluster it may have been changed for some reason.  In order to allow for listing of files in /user you will need to add EXUCUTE for other.

 

For example the result should look like this:

drwxr-xr-x   - hdfs  supergroup          0 2018-08-27 10:19 /user

 

 

That should help.

7 REPLIES 7

Re: The Hive Metastore canary failed to create a database.

Explorer
Can someone please provide me steps how to resolve this or what should be the following debugging steps? Thank you

Re: The Hive Metastore canary failed to create a database.

Super Guru

@VeljkoC,

 

Cloudera Manager performs health tests to ensure that CDH servers are running properly.  One of these tests is creating and dropping a database in the Hive Metastore.  The health alert you are seeing indicates that that process failed.

 

Places to look for more information about the failure:

 

- Service Monitor (which issues the health check):

/var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR*

 

you might search for the word "canary" or "hive" in that file.

 

- Hive Metastore log file

/var/log/hive/hadoop-cmf-HIVE-1-HIVEMETASTORE*

 

Check for the word "canary" or look for WARN and ERROR messages pertaining to database names that include the string "canary".

 

 

Re: The Hive Metastore canary failed to create a database.

Explorer

@bgooley  thank you for reply.

This is the output from my analysis.

 

/var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR

 

 

2018-10-18 10:48:15,050 WARN com.cloudera.cmf.cdh5client.hive.MetastoreClientImpl: Could not create a hive database cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_ba135b5c35747b251098753b881a186b
MetaException(message:Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=hue, access=EXECUTE, inode="/user":cloudera-scm:supergroup:drwxrwx---
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthorizationProvider.java:279)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.check(DefaultAuthorizationProvider.java:260)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkTraverse(DefaultAuthorizationProvider.java:201)
at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkPermission(DefaultAuthorizationProvider.java:154)

 

2018-10-18 10:48:15,051 WARN com.cloudera.cmon.firehose.polling.hive.HiveMetastoreCanary: Metastore hive-HIVEMETASTORE-ba135b5c35747b251098753b881a186b:Failed to create database
com.cloudera.cmf.cdhclient.common.hive.MetaException: Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=hue, access=EXECUTE, inode="/user":cloudera-scm:supergroup:drwxrwx---

 

 

I am not sure for what this permission is denied? Can you please assist what should be corrected and how?

 

/var/log/hive/hadoop-cmf-HIVE-1-HIVEMETASTORE

 

 

2018-10-18 10:53:32,573 INFO org.apache.hadoop.hive.metastore.HiveMetaStore: [pool-5-thread-3]: 3: source:10.0.9.4 create_database: Database(name:cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_ba135b5c35747b251098753b881a186b, description:Cloudera Manager Metastore Canary Test Database, locationUri:/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_ba135b5c35747b251098753b881a186b, parameters:null)
2018-10-18 10:53:32,573 INFO org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-3]: ugi=hue ip=10.0.9.4 cmd=source:10.0.9.4 create_database: Database(name:cloudera_manager_metastore_canary_test_db_hive_HIVEMETASTORE_ba135b5c35747b251098753b881a186b, description:Cloudera Manager Metastore Canary Test Database, locationUri:/user/hue/.cloudera_manager_hive_metastore_canary/hive_HIVEMETASTORE_ba135b5c35747b251098753b881a186b, parameters:null)
2018-10-18 10:53:32,694 ERROR hive.log: [pool-5-thread-3]: Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=hue, access=EXECUTE, inode="/user":cloudera-scm:supergroup:drwxrwx---

 

I think that maybe there is some issue with folder? But not sure exactly with what folder?

Thank you

Re: The Hive Metastore canary failed to create a database.

Explorer

@bgooley

just to add that I have following folders (if the issue is related to folders on HDFS)

 

[cloudera-scm@cv-nn2 ~]$ hdfs dfs -ls /user
Found 3 items
drwxr-xr-x - cloudera-scm supergroup 0 2018-10-09 08:53 /user/cloudera-scm
drwxrwx--- - cloudera-scm supergroup 0 2018-10-05 06:56 /user/history
drwxrwxr-x - hue hue 0 2018-10-05 06:57 /user/hue

 

Re: The Hive Metastore canary failed to create a database.

Super Guru

@VeljkoC,

 

The following shows the cause of the problem:

 

2018-10-18 10:53:32,694 ERROR hive.log: [pool-5-thread-3]: Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=hue, access=EXECUTE, inode="/user":cloudera-scm:supergroup:drwxrwx---

 

The bold text above shows that an attempt was made by the Metastore to access HDFS but the required permission for the user was not available.  In this case, hue user (used by Service Monitor) needs EXECUTE privilege on /user

 

We can see that this does not exist:

 

cloudera-scm:supergroup:drwxrwx---

 

The default permission on /user is drwxr-xr-x I believe so in your cluster it may have been changed for some reason.  In order to allow for listing of files in /user you will need to add EXUCUTE for other.

 

For example the result should look like this:

drwxr-xr-x   - hdfs  supergroup          0 2018-08-27 10:19 /user

 

 

That should help.

Re: The Hive Metastore canary failed to create a database.

Explorer
Thanky you, this resolved an issue

Re: The Hive Metastore canary failed to create a database.

Explorer

Hi @bgooley why user hue is writing in my case when I am using the custom username for hue.?

Don't have an account?
Coming from Hortonworks? Activate your account here