@Samant Thakur The JMX response typically indicates 3 things why the datanode was not accessible. Network Issue DataNode is down Excessively long Garbage Collection This message comes from "/usr/lib/python2.6/site-packages/ambari_agent/alerts/metric_alert.py" script and following is the logic: if isinstance(self.metric_info, JmxMetric):
jmx_property_values, http_code = self._load_jmx(alert_uri.is_ssl_enabled, host, port, self.metric_info)
if not jmx_property_values and http_code in [200, 307]:
collect_result = self.RESULT_UNKNOWN
value_list.append('HTTP {0} response (metrics unavailable)'.format(str(http_code)))
elif not jmx_property_values and http_code not in [200, 307]:
raise Exception("[Alert][{0}] Unable to extract JSON from JMX response".format(self.get_name()))
else:
value_list.extend(jmx_property_values)
check_value = self.metric_info.calculate(value_list)
value_list.append(check_value) Network i MTU (Maximum Transmission Unit) is related to TCP/IP networking in Linux. It refers to the size (in bytes) of the largest datagram that a given layer of a communications protocol can pass at a time. It should be identical on all the nodes.MTU is set in /etc/sysconfig/network-scripts/ifcfg-ethx You can see current MTU setting with ifconfig command under Linux: $ netstat -i check the second row or $ ip link list - Check the host file on those failing nodes - Check if DNS server is having problems in name resolution. Run TestDFSIO performance tests yarn jar /usr/hdp/2.x.x.x.x/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-*tests.jar TestDFSIO -write -nrFiles 100 -fileSize 100 TestDFSIO Read Test hadoop jar Iperf Is a widely used tool for network performance measurement and tuning See Typical HDP Cluster Network Configuration Best Practices Datanode is down Restart the datanode using Ambari or manually Garbage Collection Running the GCViewer Enable GC logging for Datanode service
Open hadoop-env.sh and look for the following line export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote -Xms2048m -Xmx2048m -Dhadoop.security.logger=ERROR,DRFAS $HADOOP_DATANODE_OPTS"
Insert the following into HADOOP_DATANODE_OPTS param -verbose:gc
-XX:+PrintGCDetails
-Xloggc:${HADOOP_LOG_DIR}/hadoop-hdfs-datanode-`date +'%Y%m%d%H%M'`.gclog
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=20
After adding the GC log pram the HADOOP_DATANODE_OPTS should look like this export HADOOP_DATANODE_OPTS="-Dcom.sun.management.jmxremote -Xms2048m -Xmx2048m -Dhadoop.security.logger=ERROR,DRFAS -verbose:gc -XX:+PrintGCDetails -Xloggc:${HADOOP_LOG_DIR}/hadoop-hdfs-datanode-`date +'%Y%m%d%H%M'`.gclog -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 $HADOOP_DATANODE_OPTS" The log should give you a detailed info. Hope that helps
... View more
