Description During HDP Upgrade, Hive Metastore restart step fails with message - "ValueError: time data '2017-05-10 19:08:30' does not match format '%Y-%m-%d %H:%M:%S.%f'" Following is the stack trace: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 211, in <module> HiveMetastore().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 329, in execute method(env) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 841, in restart self.pre_upgrade_restart(env, upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 118, in pre_upgrade_restart self.upgrade_schema(env) File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/scripts/hive_metastore.py", line 150, in upgrade_schema status_params.tmp_dir) File "/usr/lib/python2.6/site-packages/resource_management/libraries/functions/security_commons.py", line 242, in cached_kinit_executor if (now - datetime.strptime(last_run_time, "%Y-%m-%d %H:%M:%S.%f") > timedelta(minutes=expiration_time)): File "/usr/lib64/python2.6/_strptime.py", line 325, in _strptime (data_string, format)) ValueError: time data '2017-05-10 19:08:30' does not match format '%Y-%m-%d %H:%M:%S.%f' Root cause During the upgrade, the data will be read from a file, such as *_tmp.txt, under the /var/lib/ambari-agent/tmp/kinit_executor_cache directory. This issue occurs if this file is not updated and points to an older date. Solution 1. Login to Hive Metastore host 2. Move *_tmp.txt files mv /var/lib/ambari-agent/tmp/kinit_executor_cache/*_tmp.txt /tmp 3. Retry Restart Hive Metastore step from Ambari Upgrade screen ... View more

During upgrade, if Namenode restarts timeout, it may not appear to be a problem as the request times out from the Ambari UI but the restart process continue to run in the background. However, this can lead to inconsistencies in Ambari database and cause further issues at Finalize upgrade step. Note: This article is only useful upto Ambari-2.5.x version and must be performed before starting the upgrade process. Ambari-2.6 onwards it is a one step change where you only need to update the ambari.properties file instead of all the xml changes listed below. For Ambari-2.6 onwards: refer this document https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-upgrade/content/preparing_to_upgrade_ambari_and_hdp.html Caution: The steps described below are a hack! It is not recommended that we go about making changes to upgrade XML files based on our needs. Please exercise caution and measure your risks before following the steps. You can increase the timeouts for namenode restart using the following steps before you start the upgrade process: Step 1: Locate the upgrade file on Ambari server host. If we are upgrading from HDP-2.5 to HDP-2.6 then : /var/lib/ambari-server/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml [Express Upgrade] /var/lib/ambari-server/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml [Rolling Upgrade] Step 2: Change <service name="HDFS"> <component name="NAMENODE"> <upgrade> <task xsi:type="restart-task"/> </upgrade> </component> to <service name="HDFS"> <component name="NAMENODE"> <upgrade> <task xsi:type="restart-task" timeout-config="upgrade.parameter.nn-restart.timeout"/> </upgrade> </component> Step 3: Add this to ambari.properties upgrade.parameter.nn-restart.timeout=XXXXXX where XXXXXX is the time in seconds Step 4: Restart Ambari Server Step 5: Now you can move on to your upgrade process ... View more

For this article, I am using Ambari-2.5.2.0 and trying to upgrade HDP-2.5.3 to HDP-2.6.2. If you have a large HBase cluster, it can take a long time to do a HBase snapshot. As part of upgrade, this is one of the steps which Ambari will perform for you. However, for a large cluster, this step can actually lead to a timeout from Ambari UI and may result in further inconsistencies just before the Finalize Upgrade step. To overcome this, a lot of people have started performing a manual HBase Snapshot before the upgrade. However, not many have found a way to force Ambari to skip this step and save some time instead of waiting for it to timeout in order to proceed to next step during the upgrade. Here is how you can skip HBase Snapshot step altogether (in case you want to perform it manually before the upgrade): Caution: The steps described below are a hack! It is not recommended that we go about making changes to upgrade XML files based on our needs. Please exercise caution and measure your risks before following the steps. The following steps must be performed before starting the upgrade. Step 1: Locate the upgrade XML file on ambari-server host /var/lib/ambari-server/resources/stacks/HDP/2.5/upgrades/upgrade-2.6.xml (for Rolling Upgrade) /var/lib/ambari-server/resources/stacks/HDP/2.5/upgrades/nonrolling-upgrade-2.6.xml (for Express Upgrade) Step 2: Comment out the following piece of code in the upgrade XML file and save it <execute-stage service="HBASE" component="HBASE_MASTER" title="Snapshot HBASE"> <task xsi:type="execute" hosts="master"> <script>scripts/hbase_upgrade.py</script> <function>take_snapshot</function> </task> </execute-stage> Step 3: Restart Ambari Server for it to pick up the changes. Step 4: Now you can start your upgrade. ... View more

HDFS per-user Metrics aren't emitted by default. Kindly exercise caution before enabling them and make sure to refer to the details of client and service port numbers. To be able to use the HDFS - Users dashboard in your Grafana instance as well as to view metrics for HDFS per user, you will need to add these custom properties to your configuration. Step-by-step guide Presumption for this guide: This is a HA environment with dfs.internal.nameservices=nnha and dfs.ha.namenodes.nnha=nn1,nn2 in Ambari, HDFS > Configs > Advanced > Custom hdfs-site 1. In Ambari, HDFS > Configs > Advanced > Custom hdfs-site - Add the following properties. dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn1=<namenodehost1>:8050 dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nn2=<namenodehost2>:8050 ipc.8020.callqueue.impl=org.apache.hadoop.ipc.FairCallQueue ipc.8020.backoff.enable=true ipc.8020.scheduler.impl=org.apache.hadoop.ipc.DecayRpcScheduler ipc.8020.scheduler.priority.levels=3 ipc.8020.decay-scheduler.backoff.responsetime.enable=true ipc.8020.decay-scheduler.backoff.responsetime.thresholds=10,20,30 If you have already enabled Service RPC port, then you can avoid adding the first two lines about servicerpc-address. Replace 8020 with your Namenode RPC port if it is different. DO NOT replace it with Service RPC Port or DataNode Lifeline Port 2. After this change you may see issues like both namenodes as Active or both as Standby in Ambari. To avoid this issue: a. Stop the ZKFC on both NameNodes b. Run the following command from one of the Namenode host as hdfs user su - hdfs hdfs zkfc -formatZK c. Restart all ZKFC 3: Restart HDFS & you should see the metrics being emitted. 4: After a few minutes, you should also be able to use the HDFS - Users Dashboard in Grafana. Things to ensure: Client port : 8020 (if different, replace it with appropriate port in all keys) Service port: 8021 (if different, replace it with appropriate port in first value) namenodehost1 and namenodehost2: needs to be replaced with actual values from the cluster and must be FQDN. dfs.internal.nameservices: needs to be replaced with acutal vallues from the cluster Example: dfs.namenode.servicerpc-address.nnha.nn1=<namenodehost1>:8050 dfs.namenode.servicerpc-address.nnha.nn2=<namenodehost2>:8050 * For more than 2 namenodes in your HA environment, please add one additional line for each additional namenode: dfs.namenode.servicerpc-address.<dfs.internal.nameservices>.nnX=<namenodehostX>:8021 Adapted from this wiki which describes how to enable per user HDFS metrics for a non-HA environment. Note : This article has been validated against Ambari-2.5.2 and HDP-2.6.2 It will not work in older versions of Ambari due to this BUG https://issues.apache.org/jira/browse/AMBARI-21640 ... View more