Hive Metastore Process

[#1]

Hive Metastore Process

Critical alert for Hive Metastore Process

Metastore on p-ods-admin-02 failed (Traceback (most recent call last):

File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_metastore.py", line 203, in execute
timeout_kill_strategy=TerminateStrategy.KILL_PROCESS_TREE,
File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
self.env.run()
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 262, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
result = function(command, **kwargs)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 303, in _call
raise ExecutionFailed(err_msg, code, out, err)

ExecutionFailed: Execution of 'export HIVE_CONF_DIR='/usr/hdp/current/hive-metastore/conf' ; hive --hiveconf

hive.metastore.uris=thrift://p-ods-admin-02.transbank.local:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e 'show databases;'' returned 1. 20/01/09 09:27:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.

Logging initialized using configuration in file:/etc/hive/2.6.5.0-292/0/hive-log4j.properties

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: org.apache.thrift.transport.TTransportException null)
)

[#2]

@Anibal_Linares 

Can you please execute below command manually see what exact issue it's facing

'export HIVE_CONF_DIR='/usr/hdp/current/hive-metastore/conf' ; hive --hiveconf

hive.metastore.uris=thrift://p-ods-admin-02.transbank.local:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e 'show databases;

I think is issue with Hive Client, try to reinstall on the node where this command is executing

[#3]

@Anibal_Linares 

Go to hive metastore host and type ps -ef|grep hive.

Make sure hive metastore is up and running fine. In case if it's found be running fine then go to ambari alert Disable the alert->Click ->Edit->Increase->Connection Timeout->60 seconds.

Perform this step and let me know whether it helps or not.

[#4]

@Prakashcit 

thanks execute the command ps -ef | grep hive and the operation of hive is perfect

What works for me Disable the alert Connection timeout-> 60 seconds.

how will i know if the hive is wrong

[#5]

@Anibal_Linares 

Screenshot you have attached showing hiveserver2 process details.check for the metastore process.

ps -ef|grep hive-metastore

how will i know if the hive is wrong?

To answer your query, you will still get an alert in case if anything goes wrong but after 60 secs, in case you feel the value is higher for your use case then you consider 30 secs. 

[#6]

@Prakashcit 

I send you a Screenshot of the command ps -ef|grep hive-metastore  

[#7]

@Anibal_Linares 

It looks like your hivemetastore process is not running, try to restart the Hive metastore service from ambari and check the process details again.

In case if you see the process details disable and enable back the alert

[#8]

 @Prakashcit 

I think that it is an alternative to restart the metastore because at the moment of restarting it it is observed that there is connectivity I think it is not necessary to disable the alarm because as I knew when you have connectivity the metastore