HDP Upgrade: Hive Metastore restart fails with "ValueError: time data '2017-05-10 19:08:30' does not match format '%Y-%m-%d %H:%M:%S.%f'"

[#1]

Description

During HDP Upgrade, Hive Metastore restart step fails with message - "ValueError: time data '2017-05-10 19:08:30' does not match format '%Y-%m-%d %H:%M:%S.%f'"

Following is the stack trace:

Traceback (most recent call last): 
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 211, in <module> HiveMetastore().execute() 
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) 
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 841, in restart self.pre_upgrade_restart(env, upgrade_type=upgrade_type) 
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 118, in pre_upgrade_restart self.upgrade_schema(env) 
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 150, in upgrade_schema status_params.tmp_dir) 
File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/security_commons.py", line 242, in cached_kinit_executor if (now - datetime.strptime(last_run_time, "%Y-%m-%d %H:%M:%S.%f") > timedelta(minutes=expiration_time)): 
File "/usr/lib64/python2.6/_strptime.py", line 325, in _strptime (data_string, format))

ValueError: time data '2017-05-10 19:08:30' does not match format '%Y-%m-%d %H:%M:%S.%f'

Root cause

During the upgrade, the data will be read from a file, such as *_tmp.txt,

under the /var/lib/ambari-agent/tmp/kinit_executor_cache directory.

This issue occurs if this file is not updated and points to an older date.

Solution

1. Login to Hive Metastore host

2. Move *_tmp.txt files

mv /var/lib/ambari-agent/tmp/kinit_executor_cache/*_tmp.txt /tmp

3. Retry Restart Hive Metastore step from Ambari Upgrade screen