Support Questions
Find answers, ask questions, and share your expertise

HDP 2.6 issues with Hive metastore HA

Highlighted

HDP 2.6 issues with Hive metastore HA

Rising Star

Metastore on $SERVER failed (Traceback (most recent call last):

  File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_metastore.py", line 200, in execute
    timeout_kill_strategy=TerminateStrategy.KILL_PROCESS_TREE,
  File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
    self.env.run()
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
ExecutionFailed: Execution of 'export HIVE_CONF_DIR='/usr/hdp/current/hive-metastore/conf' ; hive --hiveconf hive.metastore.uris=thrift://xxxxxxxxx:9083                 --hiveconf hive.metastore.client.connect.retry.delay=1                 --hiveconf hive.metastore.failure.retries=1                 --hiveconf hive.metastore.connect.retries=1                 --hiveconf hive.metastore.client.socket.timeout=14                 --hiveconf hive.execution.engine=mr -e 'show databases;'' returned 1. log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.
Logging initialized using configuration in file:/etc/hive/2.6.0.3-8/0/hive-log4j.properties
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null)
)
4 REPLIES 4
Highlighted

Re: HDP 2.6 issues with Hive metastore HA

Super Mentor

@Amine Hallam

I see the following message. If the logging initialization fails due to any reason then it can cause that component fail to start.

log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.Logging initialized using configuration in file:/etc/hive/2.6.0.3-8/0/hive-log4j.properties

.

- In the above log we see that you might have altered your hive log4j settings in such a way that it is trying to use the "maxFileSize" attribute with DailyRollingFileAppender where as that property is part of "RollingFileAppender"

.

So make sure that you are not using maxFileSize property with DFRA in your logging configuration. There may be other reasons of failure as well but it is good to fix the logging issue.

Please see:

https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/RollingFileAppender.html

https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/DailyRollingFileAppender.html

Highlighted

Re: HDP 2.6 issues with Hive metastore HA

Master Collaborator
@Jay SenSharma

is right about the logging issue. But that is not the cause of your alert. Basically the metastore alert is failing while running the "show databases" command against the Hive Metastore. You should check whether the server mentioned in the thrift url of hive.metastore.uris is running. Also check the metastore log on that specific host for detailed error. Also on that node check the hive cli log under /tmp/<executing user>/hive.log, where <executing user> might be ambari-qa.

Highlighted

Re: HDP 2.6 issues with Hive metastore HA

I'm also getting the same error in hive metastore alerts. Has anyone faced this issue and fixed ?

Highlighted

Re: HDP 2.6 issues with Hive metastore HA

This issue got resolved for me after installing tez_client, hive_client and hcat on hive metastore servers.

Don't have an account?