About suhel_khan

suhel_khan · ‎09-04-2017

Hi .. @Rajesh , @Jay SenSharma .. thanks for getting back on this as quickly as you guys did. And apologies for late response as restoring business continuity and back log processing took a while. We did not update any packages or any core component of HDP suite. The following package check return the same component versions in all nodes. [dataops@dh07 ~]$ rpm -qa | grep ambari ambari-metrics-hadoop-sink-2.2.1.0-161.x86_64 ambari-agent-2.2.1.0-161.x86_64 ambari-metrics-monitor-2.2.1.0-161.x86_64 [dataops@dh08 ~]$ rpm -qa | grep ambari ambari-metrics-collector-2.2.1.0-161.x86_64 ambari-metrics-monitor-2.2.1.0-161.x86_64 ambari-agent-2.2.1.0-161.x86_64 ambari-metrics-hadoop-sink-2.2.1.0-161.x86_64 [dataops@dh01 ~]$ rpm -qa | grep ambari ambari-metrics-monitor-2.2.1.0-161.x86_64 ambari-server-2.2.1.0-161.x86_64 ambari-agent-2.2.1.0-161.x86_64 ambari-metrics-hadoop-sink-2.2.1.0-161.x86_64 Even now with all processes running fine we have the HDFS Service Check issue, but now the Service check errors out in DH01 (stand by NN). PS : By the way , we had to take out DH04 and DH03 out of the cluster to restore sanity , we were able to stabilize the cluster then onwards. We are working to re balance the HDFS and then reformat DH04 to bring it back as a fresh Data nodes.

suhel_khan · ‎08-31-2017

Hi ALL, Environment: Cluster details: (All are on internal Datacentre and all nodes are VM servers) DH01 - Active Name Node DH02 - Secondary Name Node Data Nodes -DH03,DH04,DH05,DH07,DH08,DH09 We had a disk failure in DH03 yesterday (it shared a partition with all other datanodes) and since then we have had cluster issues. We were able to restore functioning of the cluster by restarting services and servers itself by end of the day. But today we have an issue with DH04 one of the datanodes , There is no error in the Ambari Dashboard for this node. Error while running HDFS Service Check in Ambari stderr: Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/service_check.py", line 146, in <module> HdfsServiceCheck().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/service_check.py", line 85, in service_check journalnode_port = params.journalnode_port AttributeError: 'module' object has no attribute 'journalnode_port' stdout: 2017-08-31 12:16:58,770 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-08-31 12:16:58,773 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf 2017-08-31 12:16:58,786 - ExecuteHadoop['dfsadmin -fs hdfs://belongcluster1 -safemode get | grep OFF'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'logoutput': True, 'try_sleep': 3, 'tries': 20, 'user': 'hdfs'} 2017-08-31 12:16:58,816 - Execute['hadoop --config /usr/hdp/current/hadoop-client/conf dfsadmin -fs hdfs://belongcluster1 -safemode get | grep OFF'] {'logoutput': True, 'try_sleep': 3, 'environment': {}, 'tries': 20, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']} DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. Safe mode is OFF in dh01.int.belong.com.au/58.162.144.211:8020 Safe mode is OFF in dh02.int.belong.com.au/58.162.144.163:8020 2017-08-31 12:17:01,324 - HdfsResource['/tmp'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://belongcluster1', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['create_on_execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'mode': 0777} 2017-08-31 12:17:01,328 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh01.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpiDhVPF 2>/tmp/tmpZ6ub1k''] {'quiet': False} 2017-08-31 12:17:01,364 - call returned (0, '') 2017-08-31 12:17:01,366 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh02.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpInP_PS 2>/tmp/tmpLTWE4J''] {'quiet': False} 2017-08-31 12:17:01,401 - call returned (0, '') 2017-08-31 12:17:01,401 - NameNode HA states: active_namenodes = [('nn1', 'dh01.int.belong.com.au:50070')], standby_namenodes = [('nn2', 'dh02.int.belong.com.au:50070')], unknown_namenodes = [] 2017-08-31 12:17:01,403 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh01.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmponZBis 2>/tmp/tmplQEMFQ''] {'quiet': False} 2017-08-31 12:17:01,437 - call returned (0, '') 2017-08-31 12:17:01,439 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh02.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpcIIQH0 2>/tmp/tmpCV7JdS''] {'quiet': False} 2017-08-31 12:17:01,475 - call returned (0, '') 2017-08-31 12:17:01,476 - NameNode HA states: active_namenodes = [('nn1', 'dh01.int.belong.com.au:50070')], standby_namenodes = [('nn2', 'dh02.int.belong.com.au:50070')], unknown_namenodes = [] 2017-08-31 12:17:01,479 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://dh01.int.belong.com.au:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpr3Qr8R 2>/tmp/tmpOC6vO1''] {'logoutput': None, 'quiet': False} 2017-08-31 12:17:01,521 - call returned (0, '') 2017-08-31 12:17:01,523 - HdfsResource['/tmp/ida23a1791_date163117'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://belongcluster1', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['delete_on_execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'file'} 2017-08-31 12:17:01,524 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh01.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpfXuEJY 2>/tmp/tmp1S6ZvL''] {'quiet': False} 2017-08-31 12:17:01,561 - call returned (0, '') 2017-08-31 12:17:01,563 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh02.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp7IJkiJ 2>/tmp/tmpfBFxn0''] {'quiet': False} 2017-08-31 12:17:01,600 - call returned (0, '') 2017-08-31 12:17:01,601 - NameNode HA states: active_namenodes = [('nn1', 'dh01.int.belong.com.au:50070')], standby_namenodes = [('nn2', 'dh02.int.belong.com.au:50070')], unknown_namenodes = [] 2017-08-31 12:17:01,602 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh01.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp2Eq8T1 2>/tmp/tmpHQxjDx''] {'quiet': False} 2017-08-31 12:17:01,636 - call returned (0, '') 2017-08-31 12:17:01,638 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh02.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpf1URL3 2>/tmp/tmpvhaZVZ''] {'quiet': False} 2017-08-31 12:17:01,676 - call returned (0, '') 2017-08-31 12:17:01,677 - NameNode HA states: active_namenodes = [('nn1', 'dh01.int.belong.com.au:50070')], standby_namenodes = [('nn2', 'dh02.int.belong.com.au:50070')], unknown_namenodes = [] 2017-08-31 12:17:01,679 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://dh01.int.belong.com.au:50070/webhdfs/v1/tmp/ida23a1791_date163117?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpr_fu1H 2>/tmp/tmpgoPH6t''] {'logoutput': None, 'quiet': False} 2017-08-31 12:17:01,721 - call returned (0, '') 2017-08-31 12:17:01,723 - HdfsResource['/tmp/ida23a1791_date163117'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'source': '/etc/passwd', 'default_fs': 'hdfs://belongcluster1', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['create_on_execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'file'} 2017-08-31 12:17:01,724 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh01.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp3lwQ6c 2>/tmp/tmptAtVx4''] {'quiet': False} 2017-08-31 12:17:01,762 - call returned (0, '') 2017-08-31 12:17:01,764 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh02.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp0LqJrZ 2>/tmp/tmp2UX9mD''] {'quiet': False} 2017-08-31 12:17:01,802 - call returned (0, '') 2017-08-31 12:17:01,804 - NameNode HA states: active_namenodes = [('nn1', 'dh01.int.belong.com.au:50070')], standby_namenodes = [('nn2', 'dh02.int.belong.com.au:50070')], unknown_namenodes = [] 2017-08-31 12:17:01,805 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh01.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpVSsujH 2>/tmp/tmpvH1da9''] {'quiet': False} 2017-08-31 12:17:01,840 - call returned (0, '') 2017-08-31 12:17:01,842 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh02.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpSS2wIv 2>/tmp/tmpjscftE''] {'quiet': False} 2017-08-31 12:17:01,880 - call returned (0, '') 2017-08-31 12:17:01,882 - NameNode HA states: active_namenodes = [('nn1', 'dh01.int.belong.com.au:50070')], standby_namenodes = [('nn2', 'dh02.int.belong.com.au:50070')], unknown_namenodes = [] 2017-08-31 12:17:01,884 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://dh01.int.belong.com.au:50070/webhdfs/v1/tmp/ida23a1791_date163117?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp6LBGYJ 2>/tmp/tmpSryJ0C''] {'logoutput': None, 'quiet': False} 2017-08-31 12:17:01,921 - call returned (0, '') 2017-08-31 12:17:01,922 - Creating new file /tmp/ida23a1791_date163117 in DFS 2017-08-31 12:17:01,924 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT -T /etc/passwd '"'"'http://dh01.int.belong.com.au:50070/webhdfs/v1/tmp/ida23a1791_date163117?op=CREATE&user.name=hdfs&overwrite=True'"'"' 1>/tmp/tmpnWZTOm 2>/tmp/tmpd0EkYg''] {'logoutput': None, 'quiet': False} 2017-08-31 12:17:02,577 - call returned (0, '') 2017-08-31 12:17:02,580 - HdfsResource[None] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://belongcluster1', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf'} Datanode Log: cat hadoop-hdfs-datanode-dh04.log | tail -500 ************************************************************/ 2017-08-31 12:23:49,718 INFO datanode.DataNode (LogAdapter.java:info(45)) - STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting DataNode STARTUP_MSG: host = dh04.int.belong.com.au/XX.XXX.XXX.XX STARTUP_MSG: args = [] STARTUP_MSG: version = 2.7.1.2.4.0.0-169 STARTUP_MSG: classpath = /usr/hdp/current/hadoop-client/conf:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/spark-yarn-shuffle.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/azure-storage-2.2.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/junit-4.11.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ranger-yarn-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/activation-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ojdbc6.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-common-2.7.1.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-azure-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-annotations-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-azure.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-auth.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-common.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-nfs.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-nfs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-aws.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-common-tests.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-aws-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-annotations.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/./:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-2.7.1.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-nfs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/gson-2.2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/javax.inject-1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/guice-3.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jsp-api-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/zookeeper-3.4.6.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jettison-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/fst-2.24.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jsch-0.1.42.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/objenesis-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-net-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/activation-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-digester-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-registry.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-api-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-registry-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-tests-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-client-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-timeline-plugins-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/guice-3.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/junit-4.11.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-datajoin.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jets3t-0.9.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-streaming-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.1.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//gson-2.2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-archives.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//curator-client-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-gridmix.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-httpclient-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//metrics-core-3.0.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-datajoin-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//httpclient-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-extras.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jsp-api-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-archives-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//curator-recipes-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hamcrest-core-1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-lang3-3.3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//httpcore-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-auth.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jettison-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-ant-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-configuration-1.6.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//mockito-all-1.8.5.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//curator-framework-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-beanutils-1.7.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//junit-4.11.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-sls.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-ant.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jsch-0.1.42.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-net-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//stax-api-1.0-2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-openstack-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-openstack.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-extras-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//java-xmlbuilder-0.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-gridmix-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jaxb-api-2.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-xc-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//api-util-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-core-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-distcp-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-sls-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//activation-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//joda-time-2.9.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-rumen-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-digester-1.8.jar::mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.4.0.0-169/tez/tez-dag-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-common-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-tests-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-runtime-internals-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-history-with-acls-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-examples-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-history-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-api-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-history-parser-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-history-with-fs-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-runtime-library-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-mapreduce-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-cache-plugin-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-yarn-server-web-proxy-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-collections4-4.1.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-azure-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-annotations-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.4.0.0-169/tez/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/tez/lib/jettison-1.3.4.jar:/usr/hdp/2.4.0.0-169/tez/lib/jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/tez/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/tez/lib/jsr305-2.0.3.jar:/usr/hdp/2.4.0.0-169/tez/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/tez/lib/jersey-client-1.9.jar:/usr/hdp/2.4.0.0-169/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-aws-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-yarn-server-timeline-plugins-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/conf:mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.4.0.0-169/tez/tez-dag-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-common-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-tests-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-runtime-internals-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-history-with-acls-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-examples-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-history-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-api-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-history-parser-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-history-with-fs-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-runtime-library-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-mapreduce-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-cache-plugin-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-yarn-server-web-proxy-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-collections4-4.1.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-azure-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-annotations-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.4.0.0-169/tez/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/tez/lib/jettison-1.3.4.jar:/usr/hdp/2.4.0.0-169/tez/lib/jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/tez/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/tez/lib/jsr305-2.0.3.jar:/usr/hdp/2.4.0.0-169/tez/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/tez/lib/jersey-client-1.9.jar:/usr/hdp/2.4.0.0-169/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-aws-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-yarn-server-timeline-plugins-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/conf STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 26104d8ac833884c8776473823007f176854f2eb; compiled by 'jenkins' on 2016-02-10T06:18Z STARTUP_MSG: java = 1.8.0_60 ************************************************************/ 2017-08-31 12:23:49,742 INFO datanode.DataNode (LogAdapter.java:info(45)) - registered UNIX signal handlers for [TERM, HUP, INT] 2017-08-31 12:23:50,446 INFO impl.MetricsConfig (MetricsConfig.java:loadFirst(112)) - loaded properties from hadoop-metrics2.properties 2017-08-31 12:23:50,627 INFO timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(61)) - Initializing Timeline metrics sink. 2017-08-31 12:23:50,628 INFO timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(79)) - Identified hostname = dh04.int.belong.com.au, serviceName = datanode 2017-08-31 12:23:50,633 INFO timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(91)) - Collector Uri: http://dh08.int.belong.com.au:6188/ws/v1/timeline/metrics 2017-08-31 12:23:50,646 INFO impl.MetricsSinkAdapter (MetricsSinkAdapter.java:start(206)) - Sink timeline started 2017-08-31 12:23:50,751 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(377)) - Scheduled snapshot period at 60 second(s). 2017-08-31 12:23:50,751 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:start(192)) - DataNode metrics system started 2017-08-31 12:23:50,760 INFO datanode.BlockScanner (BlockScanner.java:<init>(172)) - Initialized block scanner with targetBytesPerSec 1048576 2017-08-31 12:23:50,762 INFO datanode.DataNode (DataNode.java:<init>(418)) - File descriptor passing is enabled. 2017-08-31 12:23:50,762 INFO datanode.DataNode (DataNode.java:<init>(429)) - Configured hostname is dh04.int.belong.com.au 2017-08-31 12:23:50,772 INFO datanode.DataNode (DataNode.java:startDataNode(1127)) - Starting DataNode with maxLockedMemory = 0 2017-08-31 12:23:50,801 INFO datanode.DataNode (DataNode.java:initDataXceiver(921)) - Opened streaming server at /0.0.0.0:50010 2017-08-31 12:23:50,803 INFO datanode.DataNode (DataXceiverServer.java:<init>(76)) - Balancing bandwith is 6250000 bytes/s 2017-08-31 12:23:50,803 INFO datanode.DataNode (DataXceiverServer.java:<init>(77)) - Number threads for balancing is 5 2017-08-31 12:23:50,807 INFO datanode.DataNode (DataXceiverServer.java:<init>(76)) - Balancing bandwith is 6250000 bytes/s 2017-08-31 12:23:50,807 INFO datanode.DataNode (DataXceiverServer.java:<init>(77)) - Number threads for balancing is 5 2017-08-31 12:23:50,808 INFO datanode.DataNode (DataNode.java:initDataXceiver(936)) - Listening on UNIX domain socket: /var/lib/hadoop-hdfs/dn_socket 2017-08-31 12:23:50,908 INFO mortbay.log (Slf4jLog.java:info(67)) - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2017-08-31 12:23:50,920 INFO server.AuthenticationFilter (AuthenticationFilter.java:constructSecretProvider(294)) - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2017-08-31 12:23:50,927 INFO http.HttpRequestLog (HttpRequestLog.java:getRequestLog(80)) - Http request log for http.requests.datanode is not defined 2017-08-31 12:23:50,934 INFO http.HttpServer2 (HttpServer2.java:addGlobalFilter(710)) - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2017-08-31 12:23:50,937 INFO http.HttpServer2 (HttpServer2.java:addFilter(685)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode 2017-08-31 12:23:50,937 INFO http.HttpServer2 (HttpServer2.java:addFilter(693)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2017-08-31 12:23:50,937 INFO http.HttpServer2 (HttpServer2.java:addFilter(693)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2017-08-31 12:23:50,955 INFO http.HttpServer2 (HttpServer2.java:openListeners(915)) - Jetty bound to port 34808 2017-08-31 12:23:50,955 INFO mortbay.log (Slf4jLog.java:info(67)) - jetty-6.1.26.hwx 2017-08-31 12:23:51,179 INFO mortbay.log (Slf4jLog.java:info(67)) - Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34808 2017-08-31 12:23:51,394 INFO web.DatanodeHttpServer (DatanodeHttpServer.java:start(201)) - Listening HTTP traffic on /0.0.0.0:50075 2017-08-31 12:23:51,585 INFO datanode.DataNode (DataNode.java:startDataNode(1144)) - dnUserName = hdfs 2017-08-31 12:23:51,585 INFO datanode.DataNode (DataNode.java:startDataNode(1145)) - supergroup = hdfs 2017-08-31 12:23:51,634 INFO ipc.CallQueueManager (CallQueueManager.java:<init>(56)) - Using callQueue class java.util.concurrent.LinkedBlockingQueue 2017-08-31 12:23:51,654 INFO ipc.Server (Server.java:run(676)) - Starting Socket Reader #1 for port 8010 2017-08-31 12:23:51,687 INFO datanode.DataNode (DataNode.java:initIpcServer(837)) - Opened IPC server at /0.0.0.0:8010 2017-08-31 12:23:51,701 INFO datanode.DataNode (BlockPoolManager.java:refreshNamenodes(152)) - Refresh request received for nameservices: belongcluster1 2017-08-31 12:23:51,731 INFO datanode.DataNode (BlockPoolManager.java:doRefreshNamenodes(197)) - Starting BPOfferServices for nameservices: belongcluster1 2017-08-31 12:23:51,746 INFO datanode.DataNode (BPServiceActor.java:run(814)) - Block pool <registering> (Datanode Uuid unassigned) service to dh01.int.belong.com.au/58.162.144.211:8020 starting to offer service 2017-08-31 12:23:51,746 INFO datanode.DataNode (BPServiceActor.java:run(814)) - Block pool <registering> (Datanode Uuid unassigned) service to dh02.int.belong.com.au/58.162.144.163:8020 starting to offer service 2017-08-31 12:23:51,754 INFO ipc.Server (Server.java:run(906)) - IPC Server Responder: starting 2017-08-31 12:23:51,754 INFO ipc.Server (Server.java:run(746)) - IPC Server listener on 8010: starting 2017-08-31 12:23:52,015 INFO common.Storage (Storage.java:tryLock(715)) - Lock on /data/hadoop/hdfs/data/in_use.lock acquired by nodename 15550@dh04.int.belong.com.au 2017-08-31 12:23:52,063 INFO common.Storage (BlockPoolSliceStorage.java:recoverTransitionRead(241)) - Analyzing storage directories for bpid BP-1930018148-58.162.144.211-1462411884867 2017-08-31 12:23:52,063 INFO common.Storage (Storage.java:lock(675)) - Locking is disabled for /data/hadoop/hdfs/data/current/BP-1930018148-58.162.144.211-1462411884867 2017-08-31 12:23:52,066 INFO datanode.DataNode (DataNode.java:initStorage(1402)) - Setting up storage: nsid=1515412344;bpid=BP-1930018148-58.162.144.211-1462411884867;lv=-56;nsInfo=lv=-63;cid=CID-0019b609-89c6-421f-b98b-21607b8a21c6;nsid=1515412344;c=0;bpid=BP-1930018148-58.162.144.211-1462411884867;dnuuid=fcb7fe98-6504-40aa-be27-4a4f29e2dde9 2017-08-31 12:23:52,114 INFO impl.FsDatasetImpl (FsVolumeList.java:addVolume(304)) - Added new volume: DS-0abd3f8e-d495-4740-9fdf-bf528bec435a 2017-08-31 12:23:52,114 INFO impl.FsDatasetImpl (FsDatasetImpl.java:addVolume(391)) - Added volume - /data/hadoop/hdfs/data/current, StorageType: DISK 2017-08-31 12:23:52,152 INFO impl.FsDatasetImpl (FsDatasetImpl.java:registerMBean(2055)) - Registered FSDatasetState MBean 2017-08-31 12:23:52,159 INFO impl.FsDatasetImpl (FsDatasetImpl.java:addBlockPool(2501)) - Adding block pool BP-1930018148-58.162.144.211-1462411884867 2017-08-31 12:23:52,161 INFO impl.FsDatasetImpl (FsVolumeList.java:run(403)) - Scanning block pool BP-1930018148-58.162.144.211-1462411884867 on volume /data/hadoop/hdfs/data/current... 2017-08-31 12:24:22,319 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43133 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:25:22,303 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43149 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:26:22,307 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43155 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:27:22,304 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43165 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:28:22,308 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43177 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:29:22,303 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43187 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:30:22,313 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43199 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:31:22,310 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43213 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:32:22,306 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43224 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:33:22,301 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43234 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:34:22,305 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43246 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:35:22,303 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43254 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:36:22,312 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43268 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:37:22,303 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43276 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) 2017-08-31 12:38:22,286 INFO impl.FsDatasetImpl (FsVolumeList.java:run(408)) - Time taken to scan block pool BP-1930018148-58.162.144.211-1462411884867 on /data/hadoop/hdfs/data/current: 870125ms 2017-08-31 12:38:22,286 INFO impl.FsDatasetImpl (FsVolumeList.java:addBlockPool(434)) - Total time to scan all replicas for block pool BP-1930018148-58.162.144.211-1462411884867: 870126ms 2017-08-31 12:38:22,287 INFO impl.FsDatasetImpl (FsVolumeList.java:run(190)) - Adding replicas to map for block pool BP-1930018148-58.162.144.211-1462411884867 on volume /data/hadoop/hdfs/data/current... 2017-08-31 12:38:22,288 INFO impl.BlockPoolSlice (BlockPoolSlice.java:readReplicasFromCache(710)) - Replica Cache file: /data/hadoop/hdfs/data/current/BP-1930018148-58.162.144.211-1462411884867/current/replicas doesn't exist 2017-08-31 12:38:22,300 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43290 dst: /127.0.0.1:50010 java.io.EOFException at java.io.DataInputStream.readShort(DataInputStream.java:315) at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58) at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227) at java.lang.Thread.run(Thread.java:745) Any input will be helpfull.

suhel_khan · ‎08-04-2017

@Jay SenSharma Thanks for getting back on this, the details of Ambari Agent as below ]$ ambari-agent --version 2.2.1.0 ]$ rpm -qa|grep ambari-agent ambari-agent-2.2.1.0-161.x86_64 Its does seem like , the issue indicated in the Jira is relevant to the issue that occurred. As of now this issue has occurred only once but it does seem like migrating would be a good option to avoid this issue in future. Also, i had indicated that Namenode CPU WIO was N/A, after a few hours i am able to see the metric on the Dashboard.

suhel_khan · ‎08-04-2017

The issue started with an Alert on Hive Metastore Service: Metastore on dh01.int.belong.com.au failed (Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_metastore.py", line 183, in execute timeout=int(check_command_timeout) ) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__ self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run tries=self.resource.tries, try_sleep=self.resource.try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper result = _call(command, **kwargs_copy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) Fail: Execution of 'export HIVE_CONF_DIR='/usr/hdp/current/hive-metastore/conf/conf.server' ; hive --hiveconf hive.metastore.uris=thrift://dh01.int.belong.com.au:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e 'show databases;'' returned 5. Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000002c0000000, 977797120, 0) failed; error='Cannot allocate memory' (errno=12) Unable to determine Hadoop version information. 'hadoop version' returned: Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000002c0000000, 977797120, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 977797120 bytes for committing reserved memory. # An error report file with more information is saved as: # /home/ambari-qa/hs_err_pid4858.log ) I tried launching hive from command prompt : sudo hive , this error ed out with Java Run time Environment Exception. Then, i looked at memory utilization which indicated that SWAP has run out. ]$ free -m total used free shared buffers cached Mem : 64560 63952 607 0 77 565 -/+ buffers/cache: 63309 1251 Swap : 1023 1023 0 I tried to restart Hive Metastore service from Ambari but that operation Hung for over 30 minutes without printing anything in the stdout and strerror logs. At this point I involved Server Administrator in the investigation and it was revealed that the following process had reserved upto 40 GB. It seemed strange (I am not sure what is the optimal utilization pattern for Ambari Agent/Monitor ?? !! ) root 3424 3404 14 2016 ? 52-22:05:00 /usr/bin/python2 /usr/lib/python2.6/site-packages/ambari_agent/main.py start At this point i tried to restart Ambari Metric service on the name node from Ambari, the operation Timed out and then "Heart Beat" from the node stopped. As can be seen in the image. I was not able to restart Ambari Metric service on the Name Node from Ambari Console, as the option was disabled. I tried to so a rolling restart of all Ambari Monitor Services, but the Monitor Service on Name Node did not start. At this point we decided to 2 things, add more swap space (Admin added 1 more GB ) and then i stopped and started Ambari Services as follows: #Stop operation did not succed at first go and i had to kill the Pid sudo su - ams -c '/usr/sbin/ambari-metrics-monitor --config /etc/ambari-metrics-monitor/conf stop' sudo su - ams -c '/usr/sbin/ambari-metrics-monitor --config /etc/ambari-metrics-monitor/conf start' #I looked at Agent Status sudo ambari-agent status#The agent was not running, Hence i started the agent sudo ambari-agent start After the agent start the monitor from this node was up and reflected in Ambari. The only issue that i have now is that Namenode CPU WIO is N/A on the Ambari Dashboard ? , Will be helpfull to know how to get this back ? Also, what i intend to do is to review HiveServer2 and Metastore heap sizes which current stand at, again would these settings cause this issue were swap runs out. This has not happened before ! HiveServer2 Heap Size = 20480 MB Metastore Heap Size = 12288 MB Environment Information: Hadoop 2.7.1.2.4.0.0-169 hive-meta-store - 2.4.0.0-169 hive-server2 - 2.4.0.0-169 hive-webhcat - 2.4.0.0-169 Ambari 2.2.1.0RAM: 64 GB Helpfull links: https://community.hortonworks.com/questions/15862/how-can-i-start-my-ambari-heartbeat.html https://cwiki.apache.org/confluence/display/AMBARI/Metrics

suhel_khan · ‎07-27-2017

Try: https://cwiki.apache.org/confluence/display/Hive/AvroSerDe#AvroSerDe-Overview–WorkingwithAvrofromHive

suhel_khan · ‎07-27-2017

HI @Priya Parmar , From some amount of research(Not deep dive) that i did during that time , i could not find a reference to what i wanted to achieve hence i abandoned the approach and did not use AVRO schema , as i had limited time at hand. Ref: https://cwiki.apache.org/confluence/display/Hive/AvroSerDe#AvroSerDe-Overview–WorkingwithAvrofromHive Do let me know if you got around this pattern or if a way forward has been developed.

suhel_khan · ‎07-21-2017

Hi All, This is a more capacity planning question. Over the last few months i have observed that YARN RAM utilization of our cluster has gone up considerably , due to new process being on boarded onto the cluster. At this point i am trying to find out the length of time (cumulative) through the day for which RAM usage is peaking. Any information on how to go about this will be helpfull. Currently: Hadoop2.7.1.2.4.0.0-169 Ambari - 2.2.1 HDP 2.4 Master - 2 (63 GB RAM) ,DN -6 (63 GB RAM) RAM 320 GB Total allocation.

suhel_khan · ‎07-21-2017

@mqureshi .. Thanks for getting back. I have reduced the HiveServer2 Heap Size to 20 GB and observing the behavior, i intend to reduce to 12 GB ,step wise over the coming days.

suhel_khan · ‎07-19-2017

I am facing hive errors intermittently, Garbage Collection Issues indicated in the log: hiveserver2: @dh01 hive]$ cat hiveserver2.log | grep 'GC' at org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:118) at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:163) at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471) 2017-07-17 14:00:22,815 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@59fc6d05]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of approximately 1913ms GC pool 'PS Scavenge' had collection(s): count=1 time=1961ms 2017-07-17 14:14:28,531 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@59fc6d05]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of approximately 1452ms GC pool 'PS Scavenge' had collection(s): count=1 time=1701ms 2017-07-17 15:04:32,309 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@59fc6d05]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of approximately 1838ms GC pool 'PS Scavenge' had collection(s): count=1 time=2195ms 2017-07-17 16:08:45,121 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@59fc6d05]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of approximately 1568ms GC pool 'PS Scavenge' had collection(s): count=1 time=1707ms hivemetastore: @dh01 hive]$ cat hivemetastore.log | grep -i "GC pool" GC pool 'PS Scavenge' had collection(s): count=1 time=3521ms GC pool 'PS MarkSweep' had collection(s): count=1 time=11097ms GC pool 'PS Scavenge' had collection(s): count=1 time=37ms @dh01 hive]$ cat hivemetastore.log | grep -i "JvmPauseMonitor" 2017-07-19 04:26:50,008 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@4f85aca0]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of approximately 3050ms 2017-07-19 11:01:32,392 WARN [org.apache.hadoop.util.JvmPauseMonitor$Monitor@4f85aca0]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(191)) - Detected pause in JVM or host machine (eg GC): pause of approximately 10915ms HiveServer2 Heap Size = 24210 MB (had been set already) Metastore Heap Size = 12288 MB (changed from 8 GB previously). Client heap Size= 2 GB (changed from 1 GB previously). I did read the article below and the provided links, which was helpfull: https://community.hortonworks.com/articles/14170/namenode-garbage-collection-configuration-best-pra.html but after having made the changes to indicated heap sizes , i still had instances were Hiveserver2 or Metastore service would go on alert in ambari for a few seconds and come back healthy. The logs , did not have any errors in this instance hive.out hive.log hive-server2.out hive-server2.log hivemetastore.log hiveserver2.log Am i missing something ?, would setting HiveServer2 Heap Size and Metastore Heap Size Same help.. i.e setting (HiveServer2 Heap Size =12288 MB) Environment: Hadoop 2.7.1.2.4.0.0-169 hive-meta-store - 2.4.0.0-169 hive-server2 - 2.4.0.0-169 hive-webhcat - 2.4.0.0-169 Ambari 2.2.1.0

suhel_khan · ‎06-29-2017

Hi @ssathish, I did look at the Link you posted and decided to delete the file. CAUTION: For some reason a few hours later there were inconsistencies in the cluster . One of the data nodes (D5) were clean up was done had corruption in the way containers were processed. Some jobs for which containers were lunched in D5 executed to completion successfully and some other jobs failed due to Vertex failed error. We could not find any errors in RM log/Datanode Log/Node Manager Log We had to remove D5 off the cluster and reinstall node manager to set things right.

suhel_khan · ‎06-26-2017

I have a disk running full on one of my Data node: [ayguha@dh03 hadoop]$ sudo du -h --max-depth=1 674G ./hdfs 243G ./yarn 916G . [xx@dh03 local]$ sudo du -h --max-depth=1 1.4G ./filecache 3.2G ./usercache 68K ./nmPrivate 242G . There are over 1k tmp files accumulating in /data/hadoop/yarn/local [ayguha@dh03 local]$ ls -l *.tmp | wc -l 1055 ./optimized-preview-record-buffer-2808068b-4d54-492e-a31a-385065d25a408826610818023522318.tmp ./preview-record-buffer-24a7477f-01f0-427e-a032-54866df48b197825057363055390034.tmp ./preview-record-buffer-b22020bb-6ec2-4f73-9d65-65dbba50136e527236496621902098.tmp [ayguha@dh03 local]$ find ./*preview-record-buffer* -type f -mtime +90 | wc -l 973 There are near 1k files that are older than 3 months . Is it safe to delete these files ? ENV: Hadoop 2.7.1.2.4.0.0-169 HDP 2.4

suhel_khan · ‎05-29-2017

@mqureshi The cluster currently only has one active name node. Is there a better way to find out the 'Active Node' ? I used the following as well.. but does not distinguish curl --user admin:admin http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/host_components?HostRoles/component_name=NAMENODE&metrics/dfs/FSNamesystem/HAState=active dh01 ~]$ curl --user admin:admin http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/host_components?HostRoles/component_name=NAMENODE&metrics/dfs/FSNamesystem/HAState=active [1] 16533 -bash: metrics/dfs/FSNamesystem/HAState=active: No such file or directory [ayguha@dh01 ~]$ { "href" : "http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/host_components?HostRoles/component_name=NAMENODE", "items" : [ { "href" : "http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/hosts/dh01.int.belong.com.au/host_components/NAMENODE", "HostRoles" : { "cluster_name" : "belong1", "component_name" : "NAMENODE", "host_name" : "dh01.int.belong.com.au" }, "host" : { "href" : "http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/hosts/dh01.int.belong.com.au" } }, { "href" : "http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/hosts/dh02.int.belong.com.au/host_components/NAMENODE", "HostRoles" : { "cluster_name" : "belong1", "component_name" : "NAMENODE", "host_name" : "dh02.int.belong.com.au" }, "host" : { "href" : "http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/hosts/dh02.int.belong.com.au" } } ] } Also hdfs-site.xml does not have the property dfs.namenode.rpc-address.

suhel_khan · ‎05-29-2017

@mqureshi Command: tried it directly without pushing it to background sudo -u hdfs hdfs balancer -fs hdfs://belongcluster1:8020 -threshold 5 [ayguha@dh01 ~]$ sudo -u hdfs hdfs balancer -fs hdfs://belongcluster1:8020 -threshold 5 17/05/29 15:29:39 INFO balancer.Balancer: Using a threshold of 5.0 17/05/29 15:29:39 INFO balancer.Balancer: namenodes = [hdfs://belongcluster1, hdfs://belongcluster1:8020] 17/05/29 15:29:39 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false] 17/05/29 15:29:39 INFO balancer.Balancer: included nodes = [] 17/05/29 15:29:39 INFO balancer.Balancer: excluded nodes = [] 17/05/29 15:29:39 INFO balancer.Balancer: source nodes = [] Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved 17/05/29 15:29:41 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec 17/05/29 15:29:41 INFO block.BlockTokenSecretManager: Setting block keys 17/05/29 15:29:41 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec 17/05/29 15:29:42 INFO block.BlockTokenSecretManager: Setting block keys 17/05/29 15:29:42 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec 17/05/29 15:29:42 INFO block.BlockTokenSecretManager: Setting block keys 17/05/29 15:29:42 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec java.io.IOException: Another Balancer is running.. Exiting ... May 29, 2017 3:29:42 PM Balancing took 3.035 seconds Error: 17/05/29 15:29:42 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec java.io.IOException: Another Balancer is running.. Exiting ... Also checked if balancer process is stuck.. from the output it does not look like anything is hanging from previous tries. dh01 ~]$ ps -ef | grep "balancer" ayguha 4611 2551 0 15:34 pts/0 00:00:00 grep balancer dh01 ~]$hdfs dfs -ls /system/balancer.id ls: `/system/balancer.id': No such file or directory

suhel_khan · ‎05-29-2017

@mqureshi I found another thread with similar issue: https://community.hortonworks.com/questions/22105/hdfs-balancer-is-getting-failed-after-30-mins-in-a.html here they say indicate that if HA is enabled then one would need to remove dfs.namenode.rpc-address . I ran a check on Ambari Server using the configs.sh: /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080 get dh01.int.belong.com.au belong1 hdfs-site and the output does not contain the dfs.namenode.rpc-address property. ########## Performing 'GET' on (Site:hdfs-site, Tag:version1470359698835) "properties" : { "dfs.block.access.token.enable" : "true", "dfs.blockreport.initialDelay" : "120", "dfs.blocksize" : "134217728", "dfs.client.block.write.replace-datanode-on-failure.enable" : "NEVER", "dfs.client.failover.proxy.provider.belongcluster1" : "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider", "dfs.client.read.shortcircuit" : "true", "dfs.client.read.shortcircuit.streams.cache.size" : "4096", "dfs.client.retry.policy.enabled" : "false", "dfs.cluster.administrators" : " hdfs", "dfs.content-summary.limit" : "5000", "dfs.datanode.address" : "0.0.0.0:50010", "dfs.datanode.balance.bandwidthPerSec" : "6250000", "dfs.datanode.data.dir" : "/data/hadoop/hdfs/data", "dfs.datanode.data.dir.perm" : "750", "dfs.datanode.du.reserved" : "1073741824", "dfs.datanode.failed.volumes.tolerated" : "0", "dfs.datanode.http.address" : "0.0.0.0:50075", "dfs.datanode.https.address" : "0.0.0.0:50475", "dfs.datanode.ipc.address" : "0.0.0.0:8010", "dfs.datanode.max.transfer.threads" : "16384", "dfs.domain.socket.path" : "/var/lib/hadoop-hdfs/dn_socket", "dfs.encrypt.data.transfer.cipher.suites" : "AES/CTR/NoPadding", "dfs.encryption.key.provider.uri" : "", "dfs.ha.automatic-failover.enabled" : "true", "dfs.ha.fencing.methods" : "shell(/bin/true)", "dfs.ha.namenodes.belongcluster1" : "nn1,nn2", "dfs.heartbeat.interval" : "3", "dfs.hosts.exclude" : "/etc/hadoop/conf/dfs.exclude", "dfs.http.policy" : "HTTP_ONLY", "dfs.https.port" : "50470", "dfs.journalnode.edits.dir" : "/hadoop/hdfs/journal", "dfs.journalnode.https-address" : "0.0.0.0:8481", "dfs.namenode.accesstime.precision" : "0", "dfs.namenode.acls.enabled" : "true", "dfs.namenode.audit.log.async" : "true", "dfs.namenode.avoid.read.stale.datanode" : "true", "dfs.namenode.avoid.write.stale.datanode" : "true", "dfs.namenode.checkpoint.dir" : "/tmp/hadoop/hdfs/namesecondary", "dfs.namenode.checkpoint.edits.dir" : "${dfs.namenode.checkpoint.dir}", "dfs.namenode.checkpoint.period" : "21600", "dfs.namenode.checkpoint.txns" : "1000000", "dfs.namenode.fslock.fair" : "false", "dfs.namenode.handler.count" : "200", "dfs.namenode.http-address" : "dh01.int.belong.com.au:50070", "dfs.namenode.http-address.belongcluster1.nn1" : "dh01.int.belong.com.au:50070", "dfs.namenode.http-address.belongcluster1.nn2" : "dh02.int.belong.com.au:50070", "dfs.namenode.https-address" : "dh01.int.belong.com.au:50470", "dfs.namenode.https-address.belongcluster1.nn1" : "dh01.int.belong.com.au:50470", "dfs.namenode.https-address.belongcluster1.nn2" : "dh02.int.belong.com.au:50470", "dfs.namenode.name.dir" : "/data/hadoop/hdfs/namenode", "dfs.namenode.name.dir.restore" : "true", "dfs.namenode.rpc-address.belongcluster1.nn1" : "dh01.int.belong.com.au:8020", "dfs.namenode.rpc-address.belongcluster1.nn2" : "dh02.int.belong.com.au:8020", "dfs.namenode.safemode.threshold-pct" : "0.99", "dfs.namenode.shared.edits.dir" : "qjournal://dh03.int.belong.com.au:8485;dh02.int.belong.com.au:8485;dh01.int.belong.com.au:8485/belongcluster1", "dfs.namenode.stale.datanode.interval" : "30000", "dfs.namenode.startup.delay.block.deletion.sec" : "3600", "dfs.namenode.write.stale.datanode.ratio" : "1.0f", "dfs.nameservices" : "belongcluster1", "dfs.permissions.enabled" : "true", "dfs.permissions.superusergroup" : "hdfs", "dfs.replication" : "3", "dfs.replication.max" : "50", "dfs.support.append" : "true", "dfs.webhdfs.enabled" : "true", "fs.permissions.umask-mode" : "022", "nfs.exports.allowed.hosts" : "* rw", "nfs.file.dump.dir" : "/tmp/.hdfs-nfs" } Are you suggesting that i just keep 1 namenode service address and point it to primary name node host:port. Something like the below: <property> <name>dfs.namenode.rpc-address.belongcluster1</name> <value>dh01.int.belong.com.au:8020</value> </property>

suhel_khan · ‎05-29-2017

@mqureshi About : https://community.hortonworks.com/articles/4595/balancer-not-working-in-hdfs-ha.html my hdfs-site.xml has 2 entries .. i am not sure if i need to delete both or NN2 only.. <property> <name>dfs.namenode.rpc-address.belongcluster1.nn1</name> <value>dh01.int.belong.com.au:8020</value> </property> <property> <name>dfs.namenode.rpc-address.belongcluster1.nn2</name> <value>dh02.int.belong.com.au:8020</value> </property>

suhel_khan · ‎05-29-2017

@mqureshi The cluster has a primary and secondary configuration for NN. When i run the balance command as you indicated, i get an error stating "Another Balancer is running". But ps -ef | grep balancer does not show any running balancer process [root@dh01 ~]# sudo -u hdfs hdfs balancer -fs hdfs://dh01.int.belong.com.au:8020 -threshold 5 17/05/29 12:14:53 INFO balancer.Balancer: Using a threshold of 5.0 17/05/29 12:14:53 INFO balancer.Balancer: namenodes = [hdfs://belongcluster1, hdfs://dh01.int.belong.com.au:8020] 17/05/29 12:14:53 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false] 17/05/29 12:14:53 INFO balancer.Balancer: included nodes = [] 17/05/29 12:14:53 INFO balancer.Balancer: excluded nodes = [] 17/05/29 12:14:53 INFO balancer.Balancer: source nodes = [] Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved 17/05/29 12:14:54 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec 17/05/29 12:14:54 INFO block.BlockTokenSecretManager: Setting block keys 17/05/29 12:14:54 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec 17/05/29 12:14:55 INFO block.BlockTokenSecretManager: Setting block keys 17/05/29 12:14:55 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec 17/05/29 12:14:55 INFO block.BlockTokenSecretManager: Setting block keys 17/05/29 12:14:55 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec java.io.IOException: Another Balancer is running.. Exiting ... May 29, 2017 12:14:55 PM Balancing took 2.431 seconds

suhel_khan · ‎05-29-2017

@mqureshi I used "-b" option to push the processing to background. I have also tried the following from server that has NN. Trial 1: (on Command Prompt) nohup sudo -u hdfs hdfs balancer -threshold 5 > /var/log/hadoop/hdfs/balancer.$(date +%F_%H-%M-%S.%N).log 2>&1 & Trial 2: (on Command Prompt) . DH05 needs to be offloaded as its the most unbalanced sudo -u hdfs -b hdfs balancer -threshold 5 -source DH05 > /var/log/hadoop/hdfs/balancer.$(date +%F_%H-%M-%S.%N).log 2>&1 & I get the same output from Balancer as it exists stating that "The cluster is balanced". It's somehow not able to get the current stats on data in the datanodes.

suhel_khan · ‎05-28-2017

Command ran through shell script: ....Logging sudo -u hdfs -b hdfs balancer -threshold 5 .... Log: The Balance exits successfully without balancing. 17/05/26 16:38:51 INFO balancer.Balancer: Using a threshold of 5.0 17/05/26 16:38:51 INFO balancer.Balancer: namenodes = [hdfs://belongcluster1] 17/05/26 16:38:51 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false] 17/05/26 16:38:51 INFO balancer.Balancer: included nodes = [] 17/05/26 16:38:51 INFO balancer.Balancer: excluded nodes = [] 17/05/26 16:38:51 INFO balancer.Balancer: source nodes = [] Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved 17/05/26 16:38:53 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec 17/05/26 16:38:53 INFO block.BlockTokenSecretManager: Setting block keys 17/05/26 16:38:53 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec 17/05/26 16:38:53 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000) 17/05/26 16:38:53 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000) 17/05/26 16:38:53 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200) 17/05/26 16:38:53 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 5 (default=5) 17/05/26 16:38:53 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648) 17/05/26 16:38:53 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760) 17/05/26 16:38:53 INFO block.BlockTokenSecretManager: Setting block keys 17/05/26 16:38:53 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240) 17/05/26 16:38:53 INFO balancer.Balancer: dfs.blocksize = 134217728 (default=134217728) 17/05/26 16:38:53 INFO net.NetworkTopology: Adding a new node: /default-rack/58.XXX.144.YYY:50010 17/05/26 16:38:53 INFO net.NetworkTopology: Adding a new node: /default-rack/58.XXX.144.YYY:50010 17/05/26 16:38:53 INFO net.NetworkTopology: Adding a new node: /default-rack/58.XXX.145.YY:50010 17/05/26 16:38:53 INFO net.NetworkTopology: Adding a new node: /default-rack/58.XXX.145.YY:50010 17/05/26 16:38:53 INFO net.NetworkTopology: Adding a new node: /default-rack/58.XXX.145.YY:50010 17/05/26 16:38:53 INFO net.NetworkTopology: Adding a new node: /default-rack/58.XXX.144.YY:50010 17/05/26 16:38:53 INFO balancer.Balancer: 0 over-utilized: [] 17/05/26 16:38:53 INFO balancer.Balancer: 0 underutilized: [] The cluster is balanced. Exiting... May 26, 2017 4:38:53 PM 0 0 B 0 B -1 B May 26, 2017 4:38:54 PM Balancing took 2.773 seconds The Ambari Host view indicates that the data is still not balanced across the nodes: hostviewambari.jpg ( Updated) The cluster has HA configuration (Primary-Secondary). Ambari : 2.2.1.0 Hadoop : 2.7.1.2.4.0.0-169 Any input will be helpfull.

suhel_khan · ‎04-04-2017

Hi All, Problem Statement: I have a source system that takes website data and sends out a lot of columns about 500+, At this point we only have interest in 100+ columns, we are trying to create a mechanism by which its easy to modify the HIVE table schema as the need to read more column arises. I have previously used AVRO schema files with JSON data and it worked quiet well to help add new columns easily to the schema file without altering data file. But currently the source system can only send flat CSV. I am trying to create a HIVE table and read this file as below CREATE EXTERNAL TABLE players ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION 'hdfs://xxxxxcluster1/apps/hive/warehouse/test.db/players' tblproperties ('avro.schema.url'='hdfs://xxxxxcluster1/tmp/prototype/players.avsc',"skip.header.line.count"="1"); Error: (I have changed all file permissions to 777) FAILED: SemanticException [Error 10043]: Either list of columns or a custom serializer should be specified I have not been able to find references to reading TEXTFILE with avro schema. Is this possible to do ? or is there a better way to make schema update modular with TEXTFILE data rather than having to drop and recreate table every time.

suhel_khan · ‎03-26-2017

Hi @Jay SenSharma, Thanks for your input. I removed the other repo files and have only "ambari.repo", "HDP.repo" and "HDP-UTILS.repo" and ran the following: sudo python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py --silent --skip=users yum clean all ls -ltr /etc/yum.repos.d/ But , i still could not get around the following error that was there previously: File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-monitor' returned 1. ERROR with rpm_check_debug vs depsolve: libkadm5clnt_mit.so.8()(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64 libkadm5clnt_mit.so.8(kadm5clnt_mit_8_MIT)(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64 libkadm5srv_mit.so.8()(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64 libkadm5srv_mit.so.8(kadm5srv_mit_8_MIT)(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64 You could try running: rpm -Va --nofiles --nodigest Your transaction was saved, rerun it with: yum load-transaction /tmp/yum_save_tx-2017-03-25-06-305zHb5j.yumtx After some googling I did the following on all hosts and i tried the CLUSTER INSTALL WIZARD from AMBARI again for Metrics Collector and Zookeeper installation on all EC2 nodes. It worked !! 🙂 [ec2-user@ip-172-31-5-78 ~]$ yum install libkadm5 Loaded plugins: amazon-id, rhui-lb, security Repo rhui-REGION-client-config-server-6 forced skip_if_unavailable=True due to: /etc/pki/rhui/cdn.redhat.com-chain.crt Repo rhui-REGION-client-config-server-6 forced skip_if_unavailable=True due to: /etc/pki/rhui/product/rhui-client-config-server-6.crt Repo rhui-REGION-client-config-server-6 forced skip_if_unavailable=True due to: /etc/pki/rhui/rhui-client-config-server-6.key Repo rhui-REGION-rhel-server-releases forced skip_if_unavailable=True due to: /etc/pki/rhui/cdn.redhat.com-chain.crt Repo rhui-REGION-rhel-server-releases forced skip_if_unavailable=True due to: /etc/pki/rhui/product/content-rhel6.crt Repo rhui-REGION-rhel-server-releases forced skip_if_unavailable=True due to: /etc/pki/rhui/content-rhel6.key Repo rhui-REGION-rhel-server-releases-optional forced skip_if_unavailable=True due to: /etc/pki/rhui/cdn.redhat.com-chain.crt Repo rhui-REGION-rhel-server-releases-optional forced skip_if_unavailable=True due to: /etc/pki/rhui/product/content-rhel6.crt Repo rhui-REGION-rhel-server-releases-optional forced skip_if_unavailable=True due to: /etc/pki/rhui/content-rhel6.key Repo rhui-REGION-rhel-server-rh-common forced skip_if_unavailable=True due to: /etc/pki/rhui/cdn.redhat.com-chain.crt Repo rhui-REGION-rhel-server-rh-common forced skip_if_unavailable=True due to: /etc/pki/rhui/product/content-rhel6.crt Repo rhui-REGION-rhel-server-rh-common forced skip_if_unavailable=True due to: /etc/pki/rhui/content-rhel6.key You need to be root to perform this command. [ec2-user@ip-172-31-5-78 ~]$ sudo yum install libkadm5 Loaded plugins: amazon-id, rhui-lb, security Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package libkadm5.x86_64 0:1.10.3-65.el6 will be installed --> Processing Dependency: krb5-libs(x86-64) = 1.10.3-65.el6 for package: libkadm5-1.10.3-65.el6.x86_64 --> Running transaction check ---> Package krb5-libs.x86_64 0:1.10.3-15.el6_5.1 will be updated --> Processing Dependency: krb5-libs = 1.10.3-15.el6_5.1 for package: krb5-workstation-1.10.3-15.el6_5.1.x86_64 ---> Package krb5-libs.x86_64 0:1.10.3-65.el6 will be an update --> Running transaction check ---> Package krb5-workstation.x86_64 0:1.10.3-15.el6_5.1 will be updated ---> Package krb5-workstation.x86_64 0:1.10.3-65.el6 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================ Package Arch Version Repository Size ============================================================================================================================================================================ Installing: libkadm5 x86_64 1.10.3-65.el6 rhui-REGION-rhel-server-releases 143 k Updating for dependencies: krb5-libs x86_64 1.10.3-65.el6 rhui-REGION-rhel-server-releases 675 k krb5-workstation x86_64 1.10.3-65.el6 rhui-REGION-rhel-server-releases 814 k Transaction Summary ============================================================================================================================================================================ Install 1 Package(s) Upgrade 2 Package(s) Total size: 1.6 M Total download size: 143 k Is this ok [y/N]: y Downloading Packages: libkadm5-1.10.3-65.el6.x86_64.rpm | 143 kB 00:00 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Updating : krb5-libs-1.10.3-65.el6.x86_64 1/5 Installing : libkadm5-1.10.3-65.el6.x86_64 2/5 Updating : krb5-workstation-1.10.3-65.el6.x86_64 3/5 Cleanup : krb5-workstation-1.10.3-15.el6_5.1.x86_64 4/5 Cleanup : krb5-libs-1.10.3-15.el6_5.1.x86_64 5/5 Verifying : krb5-libs-1.10.3-65.el6.x86_64 1/5 Verifying : libkadm5-1.10.3-65.el6.x86_64 2/5 Verifying : krb5-workstation-1.10.3-65.el6.x86_64 3/5 Verifying : krb5-libs-1.10.3-15.el6_5.1.x86_64 4/5 Verifying : krb5-workstation-1.10.3-15.el6_5.1.x86_64 5/5 Installed: libkadm5.x86_64 0:1.10.3-65.el6 Dependency Updated: krb5-libs.x86_64 0:1.10.3-65.el6 krb5-workstation.x86_64 0:1.10.3-65.el6 Complete!

suhel_khan · ‎03-25-2017

I have 6 AWS Ec2-Instances (3 Masters / 3 Slaves). 1 has Public IP (My Main Server) and 5 have EIP. (I added EIPs on other servers in order to try and resolve the issue, assuming that some of the libraries might need to be downloaded from net.) All AWS-EC2 instances are same Instance type-c3.xlarge Availability zone-ap-southeast-2a AMI ID -RHEL-6.5_GA-20140929-x86_64-11-Hourly2-GP2 (ami-eb2a47d1) I downloaded repos onto my Main Sever , changed the BASE URL for ambari.repo and HDP.repo and then pushed these to other servers. I updated the same local URL details in the AMBARI UI -> CLUSTER INSTALL WIZARD. Issue came up when i tried to install Metrics Collector and Zookeeper on to the EC2 nodes. Repo information: [ec2-user@ip-172-31-7-74 ~]$ sudo yum repolistLoaded plugins: amazon-id, rhui-lb, security Repository Updates-ambari-2.1.0 is listed more than once in the configuration Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration Repository HDP-2.3.0.0 is listed more than once in the configuration Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration HDP-UTILS-1.1.0.20 | 2.9 kB 00:00 repo id repo name status HDP-2.3 HDP-2.3 175 HDP-2.3.0.0 HDP Version - HDP-2.3.0.0 175 HDP-UTILS-1.1.0.20 HDP-UTILS-1.1.0.20 43 Updates-ambari-2.1.0 ambari-2.1.0 - Updates 8 rhui-REGION-client-config-server-6 Red Hat Update Infrastructure 2.0 Client Configuration Server 6 6 rhui-REGION-rhel-server-releases Red Hat Enterprise Linux Server 6 (RPMs) 19,417 rhui-REGION-rhel-server-releases-optional Red Hat Enterprise Linux Server 6 Optional (RPMs) 11,057 rhui-REGION-rhel-server-rh-common Red Hat Enterprise Linux Server 6 RH Common (RPMs) 129 repolist: 31,010I executed the command from the error , on my main EC2 instance , Master01 --I tried to run the command that was erroring , from the log [ec2-user@ip-172-31-7-74 ~]$ sudo yum -d 0 -e 0 -y install ambari-metrics-monitor ERROR with rpm_check_debug vs depsolve:libkadm5clnt_mit.so.8()(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64 libkadm5clnt_mit.so.8(kadm5clnt_mit_8_MIT)(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64 libkadm5srv_mit.so.8()(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64 libkadm5srv_mit.so.8(kadm5srv_mit_8_MIT)(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64 You could try running: rpm -Va --nofiles --nodigestYour transaction was saved, rerun it with: yum load-transaction /tmp/yum_save_tx-2017-03-25-07-01vjZxht.yumtx Content of repo directory on my Main Master [ec2-user@ip-172-31-7-74 ~]$ ls -l /etc/yum.repos.d/ total 44 -rw-r--r--. 1 root root 280 Mar 20 06:35 ambari_bkp.repo -rw-r--r-- 1 ec2-user ec2-user 317 Mar 25 06:27 ambari.repo -rw-r--r--. 1 root root 576 Mar 20 06:36 hdp_bkp.repo -rw-r--r--. 1 root root 660 Mar 20 07:20 hdp.repo -rw-r--r-- 1 root root 157 Mar 25 06:30 HDP.repo -rw-r--r-- 1 root root 180 Mar 25 02:11 HDP-UTILS.repo -rw-r--r-- 1 root root 606 Mar 24 23:27 redhat-rhui-client-config.repo -rw-r--r--. 1 root root 6300 Mar 24 23:27 redhat-rhui.repo -rw-r--r--. 1 root root 529 Oct 30 2013 rhel-source.repo -rw-r--r-- 1 root root 90 Mar 24 23:27 rhui-load-balancers.conf Content of ambari.repo and HDP.repo on my Main Master [ec2-user@ip-172-31-7-74 ~]$ cat /etc/yum.repos.d/ambari.repo #VERSION_NUMBER=2.1.0-1470 [Updates-ambari-2.1.0] name=ambari-2.1.0 - Updates baseurl=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/ambari-2.1.0/centos6/ gpgcheck=1 gpgkey=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/ambari-2.1.0/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [ec2-user@ip-172-31-7-74 ~]$ cat /etc/yum.repos.d/HDP.repo [HDP-2.3] name=HDP-2.3 baseurl=http://PublicIP/yum/HDP/centos6/2.x/updates/2.3.0.0 path=/ enabled=1 gpgcheck=0 [ec2-user@ip-172-31-7-74 ~]$ cat /etc/yum.repos.d/hdp.repo #VERSION_NUMBER=2.3.0.0-2557 [HDP-2.3.0.0] name=HDP Version - HDP-2.3.0.0 baseurl=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP/centos6/2.x/updates/2.3.0.0 gpgcheck=1 gpgkey=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP/centos6/2.x/updates/2.3.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [HDP-UTILS-1.1.0.20] name=HDP Utils Version - HDP-UTILS-1.1.0.20 baseurl=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP-UTILS-1.1.0.20/repos/centos6 gpgcheck=1 gpgkey=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP/centos6/2.x/updates/2.3.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 The repos have been copied over to other master and slaves, a view of the contents on those EC2: [ec2-user@ip-172-31-5-78 ~]$ ls -l /etc/yum.repos.d/ total 36 -rw-r--r-- 1 ec2-user ec2-user 317 Mar 25 06:27 ambari.repo -rw-r--r--. 1 ec2-user ec2-user 660 Mar 20 07:41 hdp.repo -rw-r--r-- 1 root root 157 Mar 25 06:30 HDP.repo -rw-r--r-- 1 root root 180 Mar 25 02:11 HDP-UTILS.repo -rw-r--r-- 1 root root 606 Mar 20 04:59 redhat-rhui-client-config.repo -rw-r--r-- 1 root root 6300 Mar 20 04:59 redhat-rhui.repo -rw-r--r-- 1 root root 529 Oct 30 2013 rhel-source.repo -rw-r--r--. 1 root root 90 Mar 20 04:59 rhui-load-balancers.conf [ec2-user@ip-172-31-5-78 ~]$ cat /etc/yum.repos.d/ambari.repo #VERSION_NUMBER=2.1.0-1470 [Updates-ambari-2.1.0] name=ambari-2.1.0 - Updates baseurl=http://ip-172-31-7-74.southeast.compute.internal/yum/ambari-2.1.0/centos6/ gpgcheck=1 gpgkey=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/ambari-2.1.0/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [ec2-user@ip-172-31-5-78 ~]$ cat /etc/yum.repos.d/hdp.repo #VERSION_NUMBER=2.3.0.0-2557 [HDP-2.3.0.0] name=HDP Version - HDP-2.3.0.0 baseurl=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP/centos6/2.x/updates/2.3.0.0 gpgcheck=1 gpgkey=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP/centos6/2.x/updates/2.3.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 [HDP-UTILS-1.1.0.20] name=HDP Utils Version - HDP-UTILS-1.1.0.20 baseurl=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP-UTILS-1.1.0.20/repos/centos6 gpgcheck=1 gpgkey=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP/centos6/2.x/updates/2.3.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins enabled=1 priority=1 I get the following ambari-metrics-collector information from my main server. [ec2-user@ip-172-31-7-74 ~]$ rpm -qa | grep ambari-metrics ambari-metrics-collector-2.1.0-1470.x86_64 I could not really find anything specific on net: https://www.rpmfind.net/linux/rpm2html/search.php?query=libkadm5clnt_mit.so.8()(64bit) https://rhn.redhat.com/errata/RHBA-2017-0643.html Any information on how to proceed will be helpfull. Errors: -------------------------------------------------------------------- /var/lib/ambari-agent/data/errors-107.txt -------------------------------------------------------------------- Repository Updates-ambari-2.1.0 is listed more than once in the configuration Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration Repository HDP-2.3.0.0 is listed more than once in the configuration Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration Repository Updates-ambari-2.1.0 is listed more than once in the configuration Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration Repository HDP-2.3.0.0 is listed more than once in the configuration Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration Repository Updates-ambari-2.1.0 is listed more than once in the configuration Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration Repository HDP-2.3.0.0 is listed more than once in the configuration Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration Repository Updates-ambari-2.1.0 is listed more than once in the configuration Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration Repository HDP-2.3.0.0 is listed more than once in the configuration Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration Traceback (most recent call last): File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_monitor.py", line 58, in AmsMonitor().execute() File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute method(env) File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_monitor.py", line 28, in install self.install_packages(env, exclude_packages = ['ambari-metrics-collector']) File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 376, in install_packages Package(name) File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 157, in init self.env.run() File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run self.run_action(resource, action) File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action provider_action() File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos) File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput()) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner result = function(command, **kwargs) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call tries=tries, try_sleep=try_sleep) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in callwrapper result = call(command, **kwargscopy) File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-monitor' returned 1. ERROR with rpm_check_debug vs depsolve: libkadm5clnt_mit.so.8()(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64 libkadm5clnt_mit.so.8(kadm5clnt_mit_8_MIT)(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64 libkadm5srv_mit.so.8()(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64 libkadm5srv_mit.so.8(kadm5srv_mit_8_MIT)(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64 You could try running: rpm -Va --nofiles --nodigest Your transaction was saved, rerun it with: yum load-transaction /tmp/yum_save_tx-2017-03-25-06-305zHb5j.yumtx -------------------------------------------------------------------- /var/lib/ambari-agent/data/output-107.txt -------------------------------------------------------------------- 2017-03-25 06:30:20,223 - Directory['/var/lib/ambari-agent/data/tmp/AMBARI-artifacts/'] {'recursive': True} 2017-03-25 06:30:20,225 - File['/var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jce_policy-8.zip'] {'content': DownloadSource('http://ip-172-31-7-74.ap-southeast-2.compute.internal:8080/resources//jce_policy-8.zip')} 2017-03-25 06:30:20,225 - Not downloading the file from http://ip-172-31-7-74.ap-southeast-2.compute.internal:8080/resources//jce_policy-8.zip, because /var/lib/ambari-agent/data/tmp/jce_policy-8.zip already exists 2017-03-25 06:30:20,226 - Group['hadoop'] {'ignore_failures': False} 2017-03-25 06:30:20,226 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']} 2017-03-25 06:30:20,227 - User['ams'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']} 2017-03-25 06:30:20,228 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']} 2017-03-25 06:30:20,228 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555} 2017-03-25 06:30:20,229 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'} 2017-03-25 06:30:20,240 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if 2017-03-25 06:30:20,256 - Repository['HDP-2.3'] {'base_url': 'http://PublicIP/yum/HDP/centos6/2.x/updates/2.3.0.0', 'action': ['create'], 'components': ['HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP', 'mirror_list': None} 2017-03-25 06:30:20,265 - File['/etc/yum.repos.d/HDP.repo'] {'content': InlineTemplate(...)} 2017-03-25 06:30:20,266 - Writing File['/etc/yum.repos.d/HDP.repo'] because it doesn't exist 2017-03-25 06:30:20,266 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://PublicIP/yum/HDP-UTILS-1.1.0.20/repos/centos6', 'action': ['create'], 'components': ['HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None} 2017-03-25 06:30:20,269 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': InlineTemplate(...)} 2017-03-25 06:30:20,269 - Package['unzip'] {} 2017-03-25 06:30:20,365 - Skipping installation of existing package unzip 2017-03-25 06:30:20,366 - Package['curl'] {} 2017-03-25 06:30:20,408 - Skipping installation of existing package curl 2017-03-25 06:30:20,408 - Package['hdp-select'] {} 2017-03-25 06:30:20,450 - Installing package hdp-select ('/usr/bin/yum -d 0 -e 0 -y install hdp-select') 2017-03-25 06:30:22,167 - Directory['/var/lib/ambari-agent/data/tmp/AMBARI-artifacts/'] {'recursive': True} 2017-03-25 06:30:22,168 - File['/var/lib/ambari-agent/data/tmp/jdk-8u40-linux-x64.tar.gz'] {'content': DownloadSource('http://ip-172-31-7-74.ap-southeast-2.compute.internal:8080/resources//jdk-8u40-linux-x64.tar.gz'), 'not_if': 'test -f /var/lib/ambari-agent/data/tmp/jdk-8u40-linux-x64.tar.gz'} 2017-03-25 06:30:22,178 - Skipping File['/var/lib/ambari-agent/data/tmp/jdk-8u40-linux-x64.tar.gz'] due to not_if 2017-03-25 06:30:22,179 - Directory['/usr/jdk64'] {} 2017-03-25 06:30:22,180 - Execute['('chmod', 'a+x', '/usr/jdk64')'] {'not_if': 'test -e /usr/jdk64/jdk1.8.0_40/bin/java', 'sudo': True} 2017-03-25 06:30:22,189 - Skipping Execute['('chmod', 'a+x', '/usr/jdk64')'] due to not_if 2017-03-25 06:30:22,190 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/jdk && cd /var/lib/ambari-agent/data/tmp/jdk && tar -xf /var/lib/ambari-agent/data/tmp/jdk-8u40-linux-x64.tar.gz && ambari-sudo.sh cp -rp /var/lib/ambari-agent/data/tmp/jdk/* /usr/jdk64'] {'not_if': 'test -e /usr/jdk64/jdk1.8.0_40/bin/java'} 2017-03-25 06:30:22,199 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/jdk && cd /var/lib/ambari-agent/data/tmp/jdk && tar -xf /var/lib/ambari-agent/data/tmp/jdk-8u40-linux-x64.tar.gz && ambari-sudo.sh cp -rp /var/lib/ambari-agent/data/tmp/jdk/* /usr/jdk64'] due to not_if 2017-03-25 06:30:22,199 - File['/usr/jdk64/jdk1.8.0_40/bin/java'] {'mode': 0755, 'cd_access': 'a'} 2017-03-25 06:30:22,201 - Execute['('chgrp', '-R', 'hadoop', '/usr/jdk64/jdk1.8.0_40')'] {'sudo': True} 2017-03-25 06:30:22,231 - Execute['('chown', '-R', 'root', '/usr/jdk64/jdk1.8.0_40')'] {'sudo': True} 2017-03-25 06:30:22,441 - Package['ambari-metrics-monitor'] {} 2017-03-25 06:30:22,539 - Installing package ambari-metrics-monitor ('/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-monitor')

suhel_khan · ‎12-01-2016

Can't find the cluster size calculator.. , Same issue.. looks like this thread in not being monitored.

Online	Offline
Last Visited	‎03-22-2018 09:43 AM

Date Registered	‎12-01-2016 03:42 AM
Last Visited	‎03-22-2018 09:43 AM
Total Messages Posted	25
Total Kudos Received	1

Re: Wierd Error while installing ambari-metrics - ...

Re: HDFS Service Check failed with (AttributeErro...

HDFS Service Check failed with (AttributeError: '...

Re: Unable to start Metrics Monitor on Name Node :...

Unable to start Metrics Monitor on Name Node : Hea...

Re: Is it possible to read CSV/TEXTFILE with AVRO ...

Re: Is it possible to read CSV/TEXTFILE with AVRO ...

What is the recommended way to find the average ti...

Re: Should the HiveServer2 Heap Size and Metastore...

Should the HiveServer2 Heap Size and Metastore Hea...

Re: Cleaning up preview-record-buffer tmp files f...

Cleaning up preview-record-buffer tmp files from ...

Re: HDFS Balancer exits without balancing

Re: HDFS Balancer exits without balancing

Re: HDFS Balancer exits without balancing

Re: HDFS Balancer exits without balancing

Re: HDFS Balancer exits without balancing

Re: HDFS Balancer exits without balancing

HDFS Balancer exits without balancing

Is it possible to read CSV/TEXTFILE with AVRO Sche...

Re: Wierd Error while installing ambari-metrics - ...

Wierd Error while installing ambari-metrics - rela...

Re: Cluster Sizing calculator