Support Questions
Find answers, ask questions, and share your expertise

Hive Metastore Process Unknown Alert Status Unknown

Explorer

Hello,

I see an alert on hive metastore in the alerts page on one of the instances of metastore. The services are running fine, we have HA configuration for HiveServer2 and HiveMetastore. I have tried restarting the hivemetastore instance which is reporting alert and also toggled the ambari alert, but it comes back with same unknown status for this metastore instance.

Attached screenshot and error details below

93374-alert-status.png

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_metastore.py", line 185, in execute
    cmd = format("export HIVE_CONF_DIR='{conf_dir}' ; "
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py", line 95, in format
    return ConfigurationFormatter().format(format_string, args, **result)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/format.py", line 59, in format
    result_protected = self.vformat(format_string, args, all_params)
  File "/usr/lib64/python2.6/string.py", line 549, in vformat
    result = self._vformat(format_string, args, kwargs, used_args, 2)
  File "/usr/lib64/python2.6/string.py", line 571, in _vformat
    obj, arg_used = self.get_field(field_name, args, kwargs)
  File "/usr/lib64/python2.6/string.py", line 632, in get_field
    obj = self.get_value(first, args, kwargs)
  File "/usr/lib64/python2.6/string.py", line 591, in get_value
    return kwargs[key]
  File "/usr/lib/python2.6/site-packages/resource_management/core/utils.py", line 54, in __getitem__
    return self._convert_value(self._dict[name])
KeyError: 'metastore_uri'

Below is the hivemetastore.log on the instance which has alert on ambari

2018-11-23 10:50:21,556 INFO  [pool-1-thread-1]: metastore.HiveMetaStore (HiveMetaStore.java:run(6165)) - Shutting down hive metastore.
2018-11-23 10:50:38,870 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:main(6149)) - Starting hive metastore on port 9083
2018-11-23 10:50:39,096 INFO  [main]: impl.MetricsConfig (MetricsConfig.java:loadFirst(112)) - loaded properties from hadoop-metrics2-hivemetastore.properties
2018-11-23 10:50:39,201 INFO  [main]: timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(82)) - Initializing Timeline metrics sink.
2018-11-23 10:50:39,201 INFO  [main]: timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(100)) - Identified hostname = myhost2.gca, serviceName = hivemetastore
2018-11-23 10:50:39,224 INFO  [main]: timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(118)) - Collector Uri: http://myhost1:6188/ws/v1/timeline/metrics
2018-11-23 10:50:39,235 INFO  [main]: impl.MetricsSinkAdapter (MetricsSinkAdapter.java:start(206)) - Sink timeline started
2018-11-23 10:50:39,256 INFO  [main]: impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(376)) - Scheduled snapshot period at 10 second(s).
2018-11-23 10:50:39,256 INFO  [main]: impl.MetricsSystemImpl (MetricsSystemImpl.java:start(192)) - hivemetastore metrics system started
2018-11-23 10:50:39,287 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:newRawStore(667)) - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
2018-11-23 10:50:39,308 INFO  [main]: metastore.ObjectStore (ObjectStore.java:initializeHelper(370)) - ObjectStore, initialize called
2018-11-23 10:50:39,894 INFO  [main]: metastore.ObjectStore (ObjectStore.java:getPMF(474)) - Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,Database,Type,FieldSchema,Order"
2018-11-23 10:50:40,869 INFO  [main]: metastore.MetaStoreDirectSql (MetaStoreDirectSql.java:<init>(138)) - Using direct SQL, underlying DB is OTHER
2018-11-23 10:50:40,870 INFO  [main]: metastore.ObjectStore (ObjectStore.java:setConf(284)) - Initialized ObjectStore
2018-11-23 10:50:40,966 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles_core(741)) - Added admin role in metastore
2018-11-23 10:50:40,968 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:createDefaultRoles_core(750)) - Added public role in metastore
2018-11-23 10:50:40,989 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:addAdminUsers_core(790)) - No user is added in admin role, since config is empty
2018-11-23 10:50:40,990 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:init(525)) - Begin calculating metadata count metrics.
2018-11-23 10:50:40,999 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:init(527)) - Finished metadata count metrics: 6 databases, 504 tables, 0 partitions.
2018-11-23 10:50:41,106 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6298)) - Starting DB backed MetaStore Server with SetUGI enabled
2018-11-23 10:50:41,107 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6352)) - Started the new metaserver on port [9083]...
2018-11-23 10:50:41,107 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6354)) - Options.minWorkerThreads = 200
2018-11-23 10:50:41,108 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6356)) - Options.maxWorkerThreads = 100000
2018-11-23 10:50:41,108 INFO  [main]: metastore.HiveMetaStore (HiveMetaStore.java:startMetaStore(6358)) - TCP keepalive = true


0 REPLIES 0