Member since
12-01-2016
25
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
951 | 03-26-2017 03:54 AM |
09-04-2017
03:13 AM
Hi .. @Rajesh , @Jay SenSharma .. thanks for getting back on this as quickly as you guys did. And apologies for late response as restoring business continuity and back log processing took a while. We did not update any packages or any core component of HDP suite. The following package check return the same component versions in all nodes. [dataops@dh07 ~]$ rpm -qa | grep ambari
ambari-metrics-hadoop-sink-2.2.1.0-161.x86_64
ambari-agent-2.2.1.0-161.x86_64
ambari-metrics-monitor-2.2.1.0-161.x86_64 [dataops@dh08 ~]$ rpm -qa | grep ambari
ambari-metrics-collector-2.2.1.0-161.x86_64
ambari-metrics-monitor-2.2.1.0-161.x86_64
ambari-agent-2.2.1.0-161.x86_64
ambari-metrics-hadoop-sink-2.2.1.0-161.x86_64
[dataops@dh01 ~]$ rpm -qa | grep ambari
ambari-metrics-monitor-2.2.1.0-161.x86_64
ambari-server-2.2.1.0-161.x86_64
ambari-agent-2.2.1.0-161.x86_64
ambari-metrics-hadoop-sink-2.2.1.0-161.x86_64
Even now with all processes running fine we have the HDFS Service Check issue, but now the Service check errors out in DH01 (stand by NN). PS : By the way , we had to take out DH04 and DH03 out of the cluster to restore sanity , we were able to stabilize the cluster then onwards. We are working to re balance the HDFS and then reformat DH04 to bring it back as a fresh Data nodes.
... View more
08-31-2017
02:53 AM
Hi ALL, Environment: Cluster details: (All are on internal Datacentre and all nodes are VM servers) DH01 - Active Name Node DH02 - Secondary Name Node Data Nodes -DH03,DH04,DH05,DH07,DH08,DH09 We had a disk failure in DH03 yesterday (it shared a partition with all other datanodes) and since then we have had cluster issues. We were able to restore functioning of the cluster by restarting services and servers itself by end of the day. But today we have an issue with DH04 one of the datanodes , There is no error in the Ambari Dashboard for this node. Error while running HDFS Service Check in Ambari stderr:
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/service_check.py", line 146, in <module>
HdfsServiceCheck().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 219, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/HDFS/2.1.0.2.0/package/scripts/service_check.py", line 85, in service_check
journalnode_port = params.journalnode_port
AttributeError: 'module' object has no attribute 'journalnode_port'
stdout:
2017-08-31 12:16:58,770 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-08-31 12:16:58,773 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2017-08-31 12:16:58,786 - ExecuteHadoop['dfsadmin -fs hdfs://belongcluster1 -safemode get | grep OFF'] {'bin_dir': '/usr/hdp/current/hadoop-client/bin', 'conf_dir': '/usr/hdp/current/hadoop-client/conf', 'logoutput': True, 'try_sleep': 3, 'tries': 20, 'user': 'hdfs'}
2017-08-31 12:16:58,816 - Execute['hadoop --config /usr/hdp/current/hadoop-client/conf dfsadmin -fs hdfs://belongcluster1 -safemode get | grep OFF'] {'logoutput': True, 'try_sleep': 3, 'environment': {}, 'tries': 20, 'user': 'hdfs', 'path': ['/usr/hdp/current/hadoop-client/bin']}
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
Safe mode is OFF in dh01.int.belong.com.au/58.162.144.211:8020
Safe mode is OFF in dh02.int.belong.com.au/58.162.144.163:8020
2017-08-31 12:17:01,324 - HdfsResource['/tmp'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://belongcluster1', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['create_on_execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'directory', 'mode': 0777}
2017-08-31 12:17:01,328 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh01.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpiDhVPF 2>/tmp/tmpZ6ub1k''] {'quiet': False}
2017-08-31 12:17:01,364 - call returned (0, '')
2017-08-31 12:17:01,366 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh02.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpInP_PS 2>/tmp/tmpLTWE4J''] {'quiet': False}
2017-08-31 12:17:01,401 - call returned (0, '')
2017-08-31 12:17:01,401 - NameNode HA states: active_namenodes = [('nn1', 'dh01.int.belong.com.au:50070')], standby_namenodes = [('nn2', 'dh02.int.belong.com.au:50070')], unknown_namenodes = []
2017-08-31 12:17:01,403 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh01.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmponZBis 2>/tmp/tmplQEMFQ''] {'quiet': False}
2017-08-31 12:17:01,437 - call returned (0, '')
2017-08-31 12:17:01,439 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh02.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpcIIQH0 2>/tmp/tmpCV7JdS''] {'quiet': False}
2017-08-31 12:17:01,475 - call returned (0, '')
2017-08-31 12:17:01,476 - NameNode HA states: active_namenodes = [('nn1', 'dh01.int.belong.com.au:50070')], standby_namenodes = [('nn2', 'dh02.int.belong.com.au:50070')], unknown_namenodes = []
2017-08-31 12:17:01,479 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://dh01.int.belong.com.au:50070/webhdfs/v1/tmp?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpr3Qr8R 2>/tmp/tmpOC6vO1''] {'logoutput': None, 'quiet': False}
2017-08-31 12:17:01,521 - call returned (0, '')
2017-08-31 12:17:01,523 - HdfsResource['/tmp/ida23a1791_date163117'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://belongcluster1', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['delete_on_execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'file'}
2017-08-31 12:17:01,524 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh01.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpfXuEJY 2>/tmp/tmp1S6ZvL''] {'quiet': False}
2017-08-31 12:17:01,561 - call returned (0, '')
2017-08-31 12:17:01,563 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh02.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp7IJkiJ 2>/tmp/tmpfBFxn0''] {'quiet': False}
2017-08-31 12:17:01,600 - call returned (0, '')
2017-08-31 12:17:01,601 - NameNode HA states: active_namenodes = [('nn1', 'dh01.int.belong.com.au:50070')], standby_namenodes = [('nn2', 'dh02.int.belong.com.au:50070')], unknown_namenodes = []
2017-08-31 12:17:01,602 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh01.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp2Eq8T1 2>/tmp/tmpHQxjDx''] {'quiet': False}
2017-08-31 12:17:01,636 - call returned (0, '')
2017-08-31 12:17:01,638 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh02.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpf1URL3 2>/tmp/tmpvhaZVZ''] {'quiet': False}
2017-08-31 12:17:01,676 - call returned (0, '')
2017-08-31 12:17:01,677 - NameNode HA states: active_namenodes = [('nn1', 'dh01.int.belong.com.au:50070')], standby_namenodes = [('nn2', 'dh02.int.belong.com.au:50070')], unknown_namenodes = []
2017-08-31 12:17:01,679 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://dh01.int.belong.com.au:50070/webhdfs/v1/tmp/ida23a1791_date163117?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmpr_fu1H 2>/tmp/tmpgoPH6t''] {'logoutput': None, 'quiet': False}
2017-08-31 12:17:01,721 - call returned (0, '')
2017-08-31 12:17:01,723 - HdfsResource['/tmp/ida23a1791_date163117'] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'source': '/etc/passwd', 'default_fs': 'hdfs://belongcluster1', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['create_on_execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf', 'type': 'file'}
2017-08-31 12:17:01,724 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh01.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp3lwQ6c 2>/tmp/tmptAtVx4''] {'quiet': False}
2017-08-31 12:17:01,762 - call returned (0, '')
2017-08-31 12:17:01,764 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh02.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmp0LqJrZ 2>/tmp/tmp2UX9mD''] {'quiet': False}
2017-08-31 12:17:01,802 - call returned (0, '')
2017-08-31 12:17:01,804 - NameNode HA states: active_namenodes = [('nn1', 'dh01.int.belong.com.au:50070')], standby_namenodes = [('nn2', 'dh02.int.belong.com.au:50070')], unknown_namenodes = []
2017-08-31 12:17:01,805 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh01.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpVSsujH 2>/tmp/tmpvH1da9''] {'quiet': False}
2017-08-31 12:17:01,840 - call returned (0, '')
2017-08-31 12:17:01,842 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -s '"'"'http://dh02.int.belong.com.au:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem'"'"' 1>/tmp/tmpSS2wIv 2>/tmp/tmpjscftE''] {'quiet': False}
2017-08-31 12:17:01,880 - call returned (0, '')
2017-08-31 12:17:01,882 - NameNode HA states: active_namenodes = [('nn1', 'dh01.int.belong.com.au:50070')], standby_namenodes = [('nn2', 'dh02.int.belong.com.au:50070')], unknown_namenodes = []
2017-08-31 12:17:01,884 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X GET '"'"'http://dh01.int.belong.com.au:50070/webhdfs/v1/tmp/ida23a1791_date163117?op=GETFILESTATUS&user.name=hdfs'"'"' 1>/tmp/tmp6LBGYJ 2>/tmp/tmpSryJ0C''] {'logoutput': None, 'quiet': False}
2017-08-31 12:17:01,921 - call returned (0, '')
2017-08-31 12:17:01,922 - Creating new file /tmp/ida23a1791_date163117 in DFS
2017-08-31 12:17:01,924 - call['ambari-sudo.sh su hdfs -l -s /bin/bash -c 'curl -sS -L -w '"'"'%{http_code}'"'"' -X PUT -T /etc/passwd '"'"'http://dh01.int.belong.com.au:50070/webhdfs/v1/tmp/ida23a1791_date163117?op=CREATE&user.name=hdfs&overwrite=True'"'"' 1>/tmp/tmpnWZTOm 2>/tmp/tmpd0EkYg''] {'logoutput': None, 'quiet': False}
2017-08-31 12:17:02,577 - call returned (0, '')
2017-08-31 12:17:02,580 - HdfsResource[None] {'security_enabled': False, 'hadoop_bin_dir': '/usr/hdp/current/hadoop-client/bin', 'keytab': [EMPTY], 'default_fs': 'hdfs://belongcluster1', 'hdfs_site': ..., 'kinit_path_local': '/usr/bin/kinit', 'principal_name': None, 'user': 'hdfs', 'action': ['execute'], 'hadoop_conf_dir': '/usr/hdp/current/hadoop-client/conf'}
Datanode Log: cat hadoop-hdfs-datanode-dh04.log | tail -500
************************************************************/
2017-08-31 12:23:49,718 INFO datanode.DataNode (LogAdapter.java:info(45)) - STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting DataNode
STARTUP_MSG: host = dh04.int.belong.com.au/XX.XXX.XXX.XX
STARTUP_MSG: args = []
STARTUP_MSG: version = 2.7.1.2.4.0.0-169
STARTUP_MSG: classpath = /usr/hdp/current/hadoop-client/conf:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/spark-yarn-shuffle.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/azure-storage-2.2.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/junit-4.11.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ranger-yarn-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/activation-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ojdbc6.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-common-2.7.1.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-azure-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-annotations-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-azure.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-auth.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-common.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-nfs.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-nfs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-aws.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-common-tests.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-aws-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-annotations.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/./:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-2.7.1.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-nfs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/gson-2.2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/javax.inject-1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/guice-3.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jsp-api-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/zookeeper-3.4.6.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jettison-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/fst-2.24.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jsch-0.1.42.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/objenesis-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-net-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/activation-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-digester-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-registry.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-api-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-registry-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-tests-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-client-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-timeline-plugins-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/guice-3.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/junit-4.11.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-datajoin.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jets3t-0.9.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-streaming-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.1.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//gson-2.2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-archives.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//curator-client-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-gridmix.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-httpclient-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//metrics-core-3.0.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-datajoin-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//httpclient-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-extras.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jsp-api-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-archives-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//curator-recipes-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hamcrest-core-1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-lang3-3.3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//httpcore-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-auth.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jettison-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-ant-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-configuration-1.6.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//mockito-all-1.8.5.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//curator-framework-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-beanutils-1.7.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//junit-4.11.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-sls.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-ant.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jsch-0.1.42.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-net-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//stax-api-1.0-2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-openstack-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-openstack.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-extras-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//java-xmlbuilder-0.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-gridmix-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jaxb-api-2.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-xc-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//api-util-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-core-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-distcp-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-sls-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//activation-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//joda-time-2.9.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-rumen-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-digester-1.8.jar::mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.4.0.0-169/tez/tez-dag-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-common-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-tests-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-runtime-internals-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-history-with-acls-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-examples-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-history-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-api-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-history-parser-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-history-with-fs-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-runtime-library-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-mapreduce-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-cache-plugin-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-yarn-server-web-proxy-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-collections4-4.1.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-azure-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-annotations-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.4.0.0-169/tez/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/tez/lib/jettison-1.3.4.jar:/usr/hdp/2.4.0.0-169/tez/lib/jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/tez/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/tez/lib/jsr305-2.0.3.jar:/usr/hdp/2.4.0.0-169/tez/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/tez/lib/jersey-client-1.9.jar:/usr/hdp/2.4.0.0-169/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-aws-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-yarn-server-timeline-plugins-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/conf:mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.4.0.0-169/tez/tez-dag-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-common-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-tests-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-runtime-internals-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-history-with-acls-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-examples-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-history-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-api-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-history-parser-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-history-with-fs-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-runtime-library-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-mapreduce-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/tez-yarn-timeline-cache-plugin-0.7.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-yarn-server-web-proxy-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-collections4-4.1.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-azure-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-annotations-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.4.0.0-169/tez/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/tez/lib/jettison-1.3.4.jar:/usr/hdp/2.4.0.0-169/tez/lib/jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/tez/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/tez/lib/jsr305-2.0.3.jar:/usr/hdp/2.4.0.0-169/tez/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/tez/lib/jersey-client-1.9.jar:/usr/hdp/2.4.0.0-169/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-aws-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/tez/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/tez/lib/hadoop-yarn-server-timeline-plugins-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/tez/conf
STARTUP_MSG: build = git@github.com:hortonworks/hadoop.git -r 26104d8ac833884c8776473823007f176854f2eb; compiled by 'jenkins' on 2016-02-10T06:18Z
STARTUP_MSG: java = 1.8.0_60
************************************************************/
2017-08-31 12:23:49,742 INFO datanode.DataNode (LogAdapter.java:info(45)) - registered UNIX signal handlers for [TERM, HUP, INT]
2017-08-31 12:23:50,446 INFO impl.MetricsConfig (MetricsConfig.java:loadFirst(112)) - loaded properties from hadoop-metrics2.properties
2017-08-31 12:23:50,627 INFO timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(61)) - Initializing Timeline metrics sink.
2017-08-31 12:23:50,628 INFO timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(79)) - Identified hostname = dh04.int.belong.com.au, serviceName = datanode
2017-08-31 12:23:50,633 INFO timeline.HadoopTimelineMetricsSink (HadoopTimelineMetricsSink.java:init(91)) - Collector Uri: http://dh08.int.belong.com.au:6188/ws/v1/timeline/metrics
2017-08-31 12:23:50,646 INFO impl.MetricsSinkAdapter (MetricsSinkAdapter.java:start(206)) - Sink timeline started
2017-08-31 12:23:50,751 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:startTimer(377)) - Scheduled snapshot period at 60 second(s).
2017-08-31 12:23:50,751 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:start(192)) - DataNode metrics system started
2017-08-31 12:23:50,760 INFO datanode.BlockScanner (BlockScanner.java:<init>(172)) - Initialized block scanner with targetBytesPerSec 1048576
2017-08-31 12:23:50,762 INFO datanode.DataNode (DataNode.java:<init>(418)) - File descriptor passing is enabled.
2017-08-31 12:23:50,762 INFO datanode.DataNode (DataNode.java:<init>(429)) - Configured hostname is dh04.int.belong.com.au
2017-08-31 12:23:50,772 INFO datanode.DataNode (DataNode.java:startDataNode(1127)) - Starting DataNode with maxLockedMemory = 0
2017-08-31 12:23:50,801 INFO datanode.DataNode (DataNode.java:initDataXceiver(921)) - Opened streaming server at /0.0.0.0:50010
2017-08-31 12:23:50,803 INFO datanode.DataNode (DataXceiverServer.java:<init>(76)) - Balancing bandwith is 6250000 bytes/s
2017-08-31 12:23:50,803 INFO datanode.DataNode (DataXceiverServer.java:<init>(77)) - Number threads for balancing is 5
2017-08-31 12:23:50,807 INFO datanode.DataNode (DataXceiverServer.java:<init>(76)) - Balancing bandwith is 6250000 bytes/s
2017-08-31 12:23:50,807 INFO datanode.DataNode (DataXceiverServer.java:<init>(77)) - Number threads for balancing is 5
2017-08-31 12:23:50,808 INFO datanode.DataNode (DataNode.java:initDataXceiver(936)) - Listening on UNIX domain socket: /var/lib/hadoop-hdfs/dn_socket
2017-08-31 12:23:50,908 INFO mortbay.log (Slf4jLog.java:info(67)) - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2017-08-31 12:23:50,920 INFO server.AuthenticationFilter (AuthenticationFilter.java:constructSecretProvider(294)) - Unable to initialize FileSignerSecretProvider, falling back to use random secrets.
2017-08-31 12:23:50,927 INFO http.HttpRequestLog (HttpRequestLog.java:getRequestLog(80)) - Http request log for http.requests.datanode is not defined
2017-08-31 12:23:50,934 INFO http.HttpServer2 (HttpServer2.java:addGlobalFilter(710)) - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2017-08-31 12:23:50,937 INFO http.HttpServer2 (HttpServer2.java:addFilter(685)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode
2017-08-31 12:23:50,937 INFO http.HttpServer2 (HttpServer2.java:addFilter(693)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2017-08-31 12:23:50,937 INFO http.HttpServer2 (HttpServer2.java:addFilter(693)) - Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2017-08-31 12:23:50,955 INFO http.HttpServer2 (HttpServer2.java:openListeners(915)) - Jetty bound to port 34808
2017-08-31 12:23:50,955 INFO mortbay.log (Slf4jLog.java:info(67)) - jetty-6.1.26.hwx
2017-08-31 12:23:51,179 INFO mortbay.log (Slf4jLog.java:info(67)) - Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34808
2017-08-31 12:23:51,394 INFO web.DatanodeHttpServer (DatanodeHttpServer.java:start(201)) - Listening HTTP traffic on /0.0.0.0:50075
2017-08-31 12:23:51,585 INFO datanode.DataNode (DataNode.java:startDataNode(1144)) - dnUserName = hdfs
2017-08-31 12:23:51,585 INFO datanode.DataNode (DataNode.java:startDataNode(1145)) - supergroup = hdfs
2017-08-31 12:23:51,634 INFO ipc.CallQueueManager (CallQueueManager.java:<init>(56)) - Using callQueue class java.util.concurrent.LinkedBlockingQueue
2017-08-31 12:23:51,654 INFO ipc.Server (Server.java:run(676)) - Starting Socket Reader #1 for port 8010
2017-08-31 12:23:51,687 INFO datanode.DataNode (DataNode.java:initIpcServer(837)) - Opened IPC server at /0.0.0.0:8010
2017-08-31 12:23:51,701 INFO datanode.DataNode (BlockPoolManager.java:refreshNamenodes(152)) - Refresh request received for nameservices: belongcluster1
2017-08-31 12:23:51,731 INFO datanode.DataNode (BlockPoolManager.java:doRefreshNamenodes(197)) - Starting BPOfferServices for nameservices: belongcluster1
2017-08-31 12:23:51,746 INFO datanode.DataNode (BPServiceActor.java:run(814)) - Block pool <registering> (Datanode Uuid unassigned) service to dh01.int.belong.com.au/58.162.144.211:8020 starting to offer service
2017-08-31 12:23:51,746 INFO datanode.DataNode (BPServiceActor.java:run(814)) - Block pool <registering> (Datanode Uuid unassigned) service to dh02.int.belong.com.au/58.162.144.163:8020 starting to offer service
2017-08-31 12:23:51,754 INFO ipc.Server (Server.java:run(906)) - IPC Server Responder: starting
2017-08-31 12:23:51,754 INFO ipc.Server (Server.java:run(746)) - IPC Server listener on 8010: starting
2017-08-31 12:23:52,015 INFO common.Storage (Storage.java:tryLock(715)) - Lock on /data/hadoop/hdfs/data/in_use.lock acquired by nodename 15550@dh04.int.belong.com.au
2017-08-31 12:23:52,063 INFO common.Storage (BlockPoolSliceStorage.java:recoverTransitionRead(241)) - Analyzing storage directories for bpid BP-1930018148-58.162.144.211-1462411884867
2017-08-31 12:23:52,063 INFO common.Storage (Storage.java:lock(675)) - Locking is disabled for /data/hadoop/hdfs/data/current/BP-1930018148-58.162.144.211-1462411884867
2017-08-31 12:23:52,066 INFO datanode.DataNode (DataNode.java:initStorage(1402)) - Setting up storage: nsid=1515412344;bpid=BP-1930018148-58.162.144.211-1462411884867;lv=-56;nsInfo=lv=-63;cid=CID-0019b609-89c6-421f-b98b-21607b8a21c6;nsid=1515412344;c=0;bpid=BP-1930018148-58.162.144.211-1462411884867;dnuuid=fcb7fe98-6504-40aa-be27-4a4f29e2dde9
2017-08-31 12:23:52,114 INFO impl.FsDatasetImpl (FsVolumeList.java:addVolume(304)) - Added new volume: DS-0abd3f8e-d495-4740-9fdf-bf528bec435a
2017-08-31 12:23:52,114 INFO impl.FsDatasetImpl (FsDatasetImpl.java:addVolume(391)) - Added volume - /data/hadoop/hdfs/data/current, StorageType: DISK
2017-08-31 12:23:52,152 INFO impl.FsDatasetImpl (FsDatasetImpl.java:registerMBean(2055)) - Registered FSDatasetState MBean
2017-08-31 12:23:52,159 INFO impl.FsDatasetImpl (FsDatasetImpl.java:addBlockPool(2501)) - Adding block pool BP-1930018148-58.162.144.211-1462411884867
2017-08-31 12:23:52,161 INFO impl.FsDatasetImpl (FsVolumeList.java:run(403)) - Scanning block pool BP-1930018148-58.162.144.211-1462411884867 on volume /data/hadoop/hdfs/data/current...
2017-08-31 12:24:22,319 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43133 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:25:22,303 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43149 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:26:22,307 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43155 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:27:22,304 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43165 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:28:22,308 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43177 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:29:22,303 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43187 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:30:22,313 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43199 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:31:22,310 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43213 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:32:22,306 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43224 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:33:22,301 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43234 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:34:22,305 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43246 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:35:22,303 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43254 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:36:22,312 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43268 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:37:22,303 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43276 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
2017-08-31 12:38:22,286 INFO impl.FsDatasetImpl (FsVolumeList.java:run(408)) - Time taken to scan block pool BP-1930018148-58.162.144.211-1462411884867 on /data/hadoop/hdfs/data/current: 870125ms
2017-08-31 12:38:22,286 INFO impl.FsDatasetImpl (FsVolumeList.java:addBlockPool(434)) - Total time to scan all replicas for block pool BP-1930018148-58.162.144.211-1462411884867: 870126ms
2017-08-31 12:38:22,287 INFO impl.FsDatasetImpl (FsVolumeList.java:run(190)) - Adding replicas to map for block pool BP-1930018148-58.162.144.211-1462411884867 on volume /data/hadoop/hdfs/data/current...
2017-08-31 12:38:22,288 INFO impl.BlockPoolSlice (BlockPoolSlice.java:readReplicasFromCache(710)) - Replica Cache file: /data/hadoop/hdfs/data/current/BP-1930018148-58.162.144.211-1462411884867/current/replicas doesn't exist
2017-08-31 12:38:22,300 ERROR datanode.DataNode (DataXceiver.java:run(278)) - dh04.int.belong.com.au:50010:DataXceiver error processing unknown operation src: /127.0.0.1:43290 dst: /127.0.0.1:50010
java.io.EOFException
at java.io.DataInputStream.readShort(DataInputStream.java:315)
at org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:58)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:227)
at java.lang.Thread.run(Thread.java:745)
Any input will be helpfull.
... View more
Labels:
08-04-2017
12:38 PM
@Jay SenSharma Thanks for getting back on this, the details of Ambari Agent as below ]$ ambari-agent --version
2.2.1.0
]$ rpm -qa|grep ambari-agent
ambari-agent-2.2.1.0-161.x86_64 Its does seem like , the issue indicated in the Jira is relevant to the issue that occurred. As of now this issue has occurred only once but it does seem like migrating would be a good option to avoid this issue in future. Also, i had indicated that Namenode CPU WIO was N/A, after a few hours i am able to see the metric on the Dashboard.
... View more
08-04-2017
10:58 AM
The issue started with an Alert on Hive Metastore Service: Metastore on dh01.int.belong.com.au failed (Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/HIVE/0.12.0.2.0/package/alerts/alert_hive_metastore.py", line 183, in execute
timeout=int(check_command_timeout) )
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 154, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 158, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 121, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 238, in action_run
tries=self.resource.tries, try_sleep=self.resource.try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in _call_wrapper
result = _call(command, **kwargs_copy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
Fail: Execution of 'export HIVE_CONF_DIR='/usr/hdp/current/hive-metastore/conf/conf.server' ; hive --hiveconf hive.metastore.uris=thrift://dh01.int.belong.com.au:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e 'show databases;'' returned 5. Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000002c0000000, 977797120, 0) failed; error='Cannot allocate memory' (errno=12)
Unable to determine Hadoop version information.
'hadoop version' returned:
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000002c0000000, 977797120, 0) failed; error='Cannot allocate memory' (errno=12)
# # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 977797120 bytes for committing reserved memory. # An error report file with more information is saved as: # /home/ambari-qa/hs_err_pid4858.log
)
I tried launching hive from command prompt : sudo hive , this error ed out with Java Run time Environment Exception. Then, i looked at memory utilization which indicated that SWAP has run out. ]$ free -m total used free shared buffers cached
Mem : 64560 63952 607 0 77 565
-/+ buffers/cache: 63309 1251
Swap : 1023 1023 0
I tried to restart Hive Metastore service from Ambari but that operation Hung for over 30 minutes without printing anything in the stdout and strerror logs. At this point I involved Server Administrator in the investigation and it was revealed that the following process had reserved upto 40 GB. It seemed strange (I am not sure what is the optimal utilization pattern for Ambari Agent/Monitor ?? !! ) root 3424 3404 14 2016 ? 52-22:05:00 /usr/bin/python2 /usr/lib/python2.6/site-packages/ambari_agent/main.py start At this point i tried to restart Ambari Metric service on the name node from Ambari, the operation Timed out and then "Heart Beat" from the node stopped. As can be seen in the image. I was not able to restart Ambari Metric service on the Name Node from Ambari Console, as the option was disabled. I tried to so a rolling restart of all Ambari Monitor Services, but the Monitor Service on Name Node did not start. At this point we decided to 2 things, add more swap space (Admin added 1 more GB ) and then i stopped and started Ambari Services as follows: #Stop operation did not succed at first go and i had to kill the Pid
sudo su - ams -c '/usr/sbin/ambari-metrics-monitor --config /etc/ambari-metrics-monitor/conf stop'
sudo su - ams -c '/usr/sbin/ambari-metrics-monitor --config /etc/ambari-metrics-monitor/conf start'
#I looked at Agent Status
sudo ambari-agent status#The agent was not running, Hence i started the agent
sudo ambari-agent start
After the agent start the monitor from this node was up and reflected in Ambari. The only issue that i have now is that Namenode CPU WIO is N/A on the Ambari Dashboard ? , Will be helpfull to know how to get this back ? Also, what i intend to do is to review HiveServer2 and Metastore heap sizes which current stand at, again would these settings cause this issue were swap runs out. This has not happened before ! HiveServer2 Heap Size = 20480 MB Metastore Heap Size = 12288 MB
Environment Information:
Hadoop 2.7.1.2.4.0.0-169
hive-meta-store - 2.4.0.0-169
hive-server2 - 2.4.0.0-169
hive-webhcat - 2.4.0.0-169
Ambari 2.2.1.0RAM: 64 GB Helpfull links: https://community.hortonworks.com/questions/15862/how-can-i-start-my-ambari-heartbeat.html https://cwiki.apache.org/confluence/display/AMBARI/Metrics
... View more
Labels:
- Labels:
-
Apache Ambari
07-27-2017
01:32 PM
Try:
https://cwiki.apache.org/confluence/display/Hive/AvroSerDe#AvroSerDe-Overview–WorkingwithAvrofromHive
... View more
07-27-2017
01:16 PM
HI @Priya Parmar ,
From some amount of research(Not deep dive) that i did during that time , i could not find a reference to what i wanted to achieve hence i abandoned the approach and did not use AVRO schema , as i had limited time at hand.
Ref:
https://cwiki.apache.org/confluence/display/Hive/AvroSerDe#AvroSerDe-Overview–WorkingwithAvrofromHive
Do let me know if you got around this pattern or if a way forward has been developed.
... View more
07-21-2017
07:47 AM
Hi All, This is a more capacity planning question. Over the last few months i have observed that YARN RAM utilization of our cluster has gone up considerably , due to new process being on boarded onto the cluster. At this point i am trying to find out the length of time (cumulative) through the day for which RAM usage is peaking.
Any information on how to go about this will be helpfull. Currently:
Hadoop2.7.1.2.4.0.0-169 Ambari - 2.2.1 HDP 2.4 Master - 2 (63 GB RAM) ,DN -6 (63 GB RAM) RAM 320 GB Total allocation.
... View more
Labels:
- Labels:
-
Apache YARN
07-21-2017
07:32 AM
@mqureshi .. Thanks for getting back. I have reduced the HiveServer2 Heap Size to 20 GB and observing the behavior, i intend to reduce to 12 GB ,step wise over the coming days.
... View more
07-19-2017
07:48 AM
I am facing hive errors intermittently,
Garbage Collection Issues indicated in the log:
hiveserver2: @dh01 hive]$ cat hiveserver2.log | grep 'GC'
at org.apache.tez.dag.api.client.DAGClientHandler.submitDAG(DAGClientHandler.java:118)
at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolBlockingPBServerImpl.submitDAG(DAGClientAMProtocolBlockingPBServerImpl.java:163)
at org.apache.tez.dag.api.client.rpc.DAGClientAMProtocolRPC$DAGClientAMProtocol$2.callBlockingMethod(DAGClientAMProtocolRPC.java:7471)
2017-07-17 14:00:22,815 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@59fc6d05]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of approximately 1913ms
GC pool 'PS Scavenge' had collection(s): count=1 time=1961ms
2017-07-17 14:14:28,531 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@59fc6d05]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of approximately 1452ms
GC pool 'PS Scavenge' had collection(s): count=1 time=1701ms
2017-07-17 15:04:32,309 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@59fc6d05]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of approximately 1838ms
GC pool 'PS Scavenge' had collection(s): count=1 time=2195ms
2017-07-17 16:08:45,121 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@59fc6d05]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of approximately 1568ms
GC pool 'PS Scavenge' had collection(s): count=1 time=1707ms
hivemetastore: @dh01 hive]$ cat hivemetastore.log | grep -i "GC pool"
GC pool 'PS Scavenge' had collection(s): count=1 time=3521ms
GC pool 'PS MarkSweep' had collection(s): count=1 time=11097ms
GC pool 'PS Scavenge' had collection(s): count=1 time=37ms
@dh01 hive]$ cat hivemetastore.log | grep -i "JvmPauseMonitor"
2017-07-19 04:26:50,008 INFO [org.apache.hadoop.util.JvmPauseMonitor$Monitor@4f85aca0]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM or host machine (eg GC): pause of approximately 3050ms
2017-07-19 11:01:32,392 WARN [org.apache.hadoop.util.JvmPauseMonitor$Monitor@4f85aca0]: util.JvmPauseMonitor (JvmPauseMonitor.java:run(191)) - Detected pause in JVM or host machine (eg GC): pause of approximately 10915ms
HiveServer2 Heap Size = 24210 MB (had been set already)
Metastore Heap Size = 12288 MB (changed from 8 GB previously).
Client heap Size= 2 GB (changed from 1 GB previously). I did read the article below and the provided links, which was helpfull: https://community.hortonworks.com/articles/14170/namenode-garbage-collection-configuration-best-pra.html but after having made the changes to indicated heap sizes , i still had instances were Hiveserver2 or Metastore service would go on alert in ambari for a few seconds and come back healthy. The logs , did not have any errors in this instance hive.out hive.log hive-server2.out hive-server2.log hivemetastore.log hiveserver2.log Am i missing something ?, would setting HiveServer2 Heap Size and Metastore Heap Size Same help.. i.e setting (HiveServer2 Heap Size =12288 MB) Environment: Hadoop 2.7.1.2.4.0.0-169
hive-meta-store - 2.4.0.0-169
hive-server2 - 2.4.0.0-169
hive-webhcat - 2.4.0.0-169
Ambari 2.2.1.0
... View more
Labels:
- Labels:
-
Apache Hive
06-29-2017
11:21 PM
Hi @ssathish, I did look at the Link you posted and decided to delete the file.
CAUTION:
For some reason a few hours later there were inconsistencies in the cluster . One of the data nodes (D5) were clean up was done had corruption in the way containers were processed. Some jobs for which containers were lunched in D5 executed to completion successfully and some other jobs failed due to Vertex failed error. We could not find any errors in RM log/Datanode Log/Node Manager Log We had to remove D5 off the cluster and reinstall node manager to set things right.
... View more
06-26-2017
02:29 AM
I have a disk running full on one of my Data node:
[ayguha@dh03 hadoop]$ sudo du -h --max-depth=1
674G ./hdfs
243G ./yarn
916G .
[xx@dh03 local]$ sudo du -h --max-depth=1
1.4G ./filecache
3.2G ./usercache
68K ./nmPrivate
242G .
There are over 1k tmp files accumulating in /data/hadoop/yarn/local [ayguha@dh03 local]$ ls -l *.tmp | wc -l
1055
./optimized-preview-record-buffer-2808068b-4d54-492e-a31a-385065d25a408826610818023522318.tmp
./preview-record-buffer-24a7477f-01f0-427e-a032-54866df48b197825057363055390034.tmp
./preview-record-buffer-b22020bb-6ec2-4f73-9d65-65dbba50136e527236496621902098.tmp
[ayguha@dh03 local]$ find ./*preview-record-buffer* -type f -mtime +90 | wc -l
973 There are near 1k files that are older than 3 months . Is it safe to delete these files ? ENV:
Hadoop 2.7.1.2.4.0.0-169
HDP 2.4
... View more
- Tags:
- Hadoop Core
- logs
- YARN
Labels:
- Labels:
-
Apache YARN
05-29-2017
06:17 AM
@mqureshi
The cluster currently only has one active name node.
Is there a better way to find out the 'Active Node' ?
I used the following as well.. but does not distinguish
curl --user admin:admin http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/host_components?HostRoles/component_name=NAMENODE&metrics/dfs/FSNamesystem/HAState=active dh01 ~]$ curl --user admin:admin http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/host_components?HostRoles/component_name=NAMENODE&metrics/dfs/FSNamesystem/HAState=active
[1] 16533
-bash: metrics/dfs/FSNamesystem/HAState=active: No such file or directory
[ayguha@dh01 ~]$ {
"href" : "http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/host_components?HostRoles/component_name=NAMENODE",
"items" : [
{
"href" : "http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/hosts/dh01.int.belong.com.au/host_components/NAMENODE",
"HostRoles" : {
"cluster_name" : "belong1",
"component_name" : "NAMENODE",
"host_name" : "dh01.int.belong.com.au"
},
"host" : {
"href" : "http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/hosts/dh01.int.belong.com.au"
}
},
{
"href" : "http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/hosts/dh02.int.belong.com.au/host_components/NAMENODE",
"HostRoles" : {
"cluster_name" : "belong1",
"component_name" : "NAMENODE",
"host_name" : "dh02.int.belong.com.au"
},
"host" : {
"href" : "http://dh01.int.belong.com.au:8080/api/v1/clusters/belong1/hosts/dh02.int.belong.com.au"
}
}
]
}
Also hdfs-site.xml does not have the property dfs.namenode.rpc-address.
... View more
05-29-2017
05:36 AM
@mqureshi Command: tried it directly without pushing it to background
sudo -u hdfs hdfs balancer -fs hdfs://belongcluster1:8020 -threshold 5
[ayguha@dh01 ~]$ sudo -u hdfs hdfs balancer -fs hdfs://belongcluster1:8020 -threshold 5
17/05/29 15:29:39 INFO balancer.Balancer: Using a threshold of 5.0
17/05/29 15:29:39 INFO balancer.Balancer: namenodes = [hdfs://belongcluster1, hdfs://belongcluster1:8020]
17/05/29 15:29:39 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]
17/05/29 15:29:39 INFO balancer.Balancer: included nodes = []
17/05/29 15:29:39 INFO balancer.Balancer: excluded nodes = []
17/05/29 15:29:39 INFO balancer.Balancer: source nodes = []
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
17/05/29 15:29:41 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
17/05/29 15:29:41 INFO block.BlockTokenSecretManager: Setting block keys
17/05/29 15:29:41 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
17/05/29 15:29:42 INFO block.BlockTokenSecretManager: Setting block keys
17/05/29 15:29:42 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
17/05/29 15:29:42 INFO block.BlockTokenSecretManager: Setting block keys
17/05/29 15:29:42 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
java.io.IOException: Another Balancer is running.. Exiting ...
May 29, 2017 3:29:42 PM Balancing took 3.035 seconds
Error: 17/05/29 15:29:42 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
java.io.IOException: Another Balancer is running.. Exiting ... Also checked if balancer process is stuck.. from the output it does not look like anything is hanging from previous tries. dh01 ~]$ ps -ef | grep "balancer"
ayguha 4611 2551 0 15:34 pts/0 00:00:00 grep balancer
dh01 ~]$hdfs dfs -ls /system/balancer.id
ls: `/system/balancer.id': No such file or directory
... View more
05-29-2017
03:40 AM
@mqureshi
I found another thread with similar issue:
https://community.hortonworks.com/questions/22105/hdfs-balancer-is-getting-failed-after-30-mins-in-a.html
here they say indicate that if HA is enabled then one would need to remove dfs.namenode.rpc-address .
I ran a check on Ambari Server using the configs.sh:
/var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -port 8080 get dh01.int.belong.com.au belong1 hdfs-site and the output does not contain the dfs.namenode.rpc-address property. ########## Performing 'GET' on (Site:hdfs-site, Tag:version1470359698835)
"properties" : {
"dfs.block.access.token.enable" : "true",
"dfs.blockreport.initialDelay" : "120",
"dfs.blocksize" : "134217728",
"dfs.client.block.write.replace-datanode-on-failure.enable" : "NEVER",
"dfs.client.failover.proxy.provider.belongcluster1" : "org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider",
"dfs.client.read.shortcircuit" : "true",
"dfs.client.read.shortcircuit.streams.cache.size" : "4096",
"dfs.client.retry.policy.enabled" : "false",
"dfs.cluster.administrators" : " hdfs",
"dfs.content-summary.limit" : "5000",
"dfs.datanode.address" : "0.0.0.0:50010",
"dfs.datanode.balance.bandwidthPerSec" : "6250000",
"dfs.datanode.data.dir" : "/data/hadoop/hdfs/data",
"dfs.datanode.data.dir.perm" : "750",
"dfs.datanode.du.reserved" : "1073741824",
"dfs.datanode.failed.volumes.tolerated" : "0",
"dfs.datanode.http.address" : "0.0.0.0:50075",
"dfs.datanode.https.address" : "0.0.0.0:50475",
"dfs.datanode.ipc.address" : "0.0.0.0:8010",
"dfs.datanode.max.transfer.threads" : "16384",
"dfs.domain.socket.path" : "/var/lib/hadoop-hdfs/dn_socket",
"dfs.encrypt.data.transfer.cipher.suites" : "AES/CTR/NoPadding",
"dfs.encryption.key.provider.uri" : "",
"dfs.ha.automatic-failover.enabled" : "true",
"dfs.ha.fencing.methods" : "shell(/bin/true)",
"dfs.ha.namenodes.belongcluster1" : "nn1,nn2",
"dfs.heartbeat.interval" : "3",
"dfs.hosts.exclude" : "/etc/hadoop/conf/dfs.exclude",
"dfs.http.policy" : "HTTP_ONLY",
"dfs.https.port" : "50470",
"dfs.journalnode.edits.dir" : "/hadoop/hdfs/journal",
"dfs.journalnode.https-address" : "0.0.0.0:8481",
"dfs.namenode.accesstime.precision" : "0",
"dfs.namenode.acls.enabled" : "true",
"dfs.namenode.audit.log.async" : "true",
"dfs.namenode.avoid.read.stale.datanode" : "true",
"dfs.namenode.avoid.write.stale.datanode" : "true",
"dfs.namenode.checkpoint.dir" : "/tmp/hadoop/hdfs/namesecondary",
"dfs.namenode.checkpoint.edits.dir" : "${dfs.namenode.checkpoint.dir}",
"dfs.namenode.checkpoint.period" : "21600",
"dfs.namenode.checkpoint.txns" : "1000000",
"dfs.namenode.fslock.fair" : "false",
"dfs.namenode.handler.count" : "200",
"dfs.namenode.http-address" : "dh01.int.belong.com.au:50070",
"dfs.namenode.http-address.belongcluster1.nn1" : "dh01.int.belong.com.au:50070",
"dfs.namenode.http-address.belongcluster1.nn2" : "dh02.int.belong.com.au:50070",
"dfs.namenode.https-address" : "dh01.int.belong.com.au:50470",
"dfs.namenode.https-address.belongcluster1.nn1" : "dh01.int.belong.com.au:50470",
"dfs.namenode.https-address.belongcluster1.nn2" : "dh02.int.belong.com.au:50470",
"dfs.namenode.name.dir" : "/data/hadoop/hdfs/namenode",
"dfs.namenode.name.dir.restore" : "true",
"dfs.namenode.rpc-address.belongcluster1.nn1" : "dh01.int.belong.com.au:8020",
"dfs.namenode.rpc-address.belongcluster1.nn2" : "dh02.int.belong.com.au:8020",
"dfs.namenode.safemode.threshold-pct" : "0.99",
"dfs.namenode.shared.edits.dir" : "qjournal://dh03.int.belong.com.au:8485;dh02.int.belong.com.au:8485;dh01.int.belong.com.au:8485/belongcluster1",
"dfs.namenode.stale.datanode.interval" : "30000",
"dfs.namenode.startup.delay.block.deletion.sec" : "3600",
"dfs.namenode.write.stale.datanode.ratio" : "1.0f",
"dfs.nameservices" : "belongcluster1",
"dfs.permissions.enabled" : "true",
"dfs.permissions.superusergroup" : "hdfs",
"dfs.replication" : "3",
"dfs.replication.max" : "50",
"dfs.support.append" : "true",
"dfs.webhdfs.enabled" : "true",
"fs.permissions.umask-mode" : "022",
"nfs.exports.allowed.hosts" : "* rw",
"nfs.file.dump.dir" : "/tmp/.hdfs-nfs"
}
Are you suggesting that i just keep 1 namenode service address and point it to primary name node host:port. Something like the below: <property>
<name>dfs.namenode.rpc-address.belongcluster1</name>
<value>dh01.int.belong.com.au:8020</value>
</property>
... View more
05-29-2017
02:39 AM
@mqureshi About : https://community.hortonworks.com/articles/4595/balancer-not-working-in-hdfs-ha.html my hdfs-site.xml has 2 entries .. i am not sure if i need to delete both or NN2 only.. <property>
<name>dfs.namenode.rpc-address.belongcluster1.nn1</name>
<value>dh01.int.belong.com.au:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.belongcluster1.nn2</name>
<value>dh02.int.belong.com.au:8020</value>
</property>
... View more
05-29-2017
02:17 AM
@mqureshi
The cluster has a primary and secondary configuration for NN.
When i run the balance command as you indicated, i get an error stating "Another Balancer is running".
But ps -ef | grep balancer does not show any running balancer process
[root@dh01 ~]# sudo -u hdfs hdfs balancer -fs hdfs://dh01.int.belong.com.au:8020 -threshold 5
17/05/29 12:14:53 INFO balancer.Balancer: Using a threshold of 5.0
17/05/29 12:14:53 INFO balancer.Balancer: namenodes = [hdfs://belongcluster1, hdfs://dh01.int.belong.com.au:8020]
17/05/29 12:14:53 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]
17/05/29 12:14:53 INFO balancer.Balancer: included nodes = []
17/05/29 12:14:53 INFO balancer.Balancer: excluded nodes = []
17/05/29 12:14:53 INFO balancer.Balancer: source nodes = []
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
17/05/29 12:14:54 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
17/05/29 12:14:54 INFO block.BlockTokenSecretManager: Setting block keys
17/05/29 12:14:54 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
17/05/29 12:14:55 INFO block.BlockTokenSecretManager: Setting block keys
17/05/29 12:14:55 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
17/05/29 12:14:55 INFO block.BlockTokenSecretManager: Setting block keys
17/05/29 12:14:55 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
java.io.IOException: Another Balancer is running.. Exiting ...
May 29, 2017 12:14:55 PM Balancing took 2.431 seconds
... View more
05-29-2017
12:30 AM
@mqureshi
I used "-b" option to push the processing to background.
I have also tried the following from server that has NN.
Trial 1: (on Command Prompt)
nohup sudo -u hdfs hdfs balancer -threshold 5 > /var/log/hadoop/hdfs/balancer.$(date +%F_%H-%M-%S.%N).log 2>&1 & Trial 2: (on Command Prompt) . DH05 needs to be offloaded as its the most unbalanced sudo -u hdfs -b hdfs balancer -threshold 5 -source DH05 > /var/log/hadoop/hdfs/balancer.$(date +%F_%H-%M-%S.%N).log 2>&1 &
I get the same output from Balancer as it exists stating that "The cluster is balanced". It's somehow not able to get the current stats on data in the datanodes.
... View more
05-28-2017
11:07 PM
Command ran through shell script: ....Logging
sudo -u hdfs -b hdfs balancer -threshold 5
.... Log: The Balance exits successfully without balancing. 17/05/26 16:38:51 INFO balancer.Balancer: Using a threshold of 5.0
17/05/26 16:38:51 INFO balancer.Balancer: namenodes = [hdfs://belongcluster1]
17/05/26 16:38:51 INFO balancer.Balancer: parameters = Balancer.BalancerParameters [BalancingPolicy.Node, threshold = 5.0, max idle iteration = 5, #excluded nodes = 0, #included nodes = 0, #source nodes = 0, #blockpools = 0, run during upgrade = false]
17/05/26 16:38:51 INFO balancer.Balancer: included nodes = []
17/05/26 16:38:51 INFO balancer.Balancer: excluded nodes = []
17/05/26 16:38:51 INFO balancer.Balancer: source nodes = []
Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
17/05/26 16:38:53 INFO balancer.KeyManager: Block token params received from NN: update interval=10hrs, 0sec, token lifetime=10hrs, 0sec
17/05/26 16:38:53 INFO block.BlockTokenSecretManager: Setting block keys
17/05/26 16:38:53 INFO balancer.KeyManager: Update block keys every 2hrs, 30mins, 0sec
17/05/26 16:38:53 INFO balancer.Balancer: dfs.balancer.movedWinWidth = 5400000 (default=5400000)
17/05/26 16:38:53 INFO balancer.Balancer: dfs.balancer.moverThreads = 1000 (default=1000)
17/05/26 16:38:53 INFO balancer.Balancer: dfs.balancer.dispatcherThreads = 200 (default=200)
17/05/26 16:38:53 INFO balancer.Balancer: dfs.datanode.balance.max.concurrent.moves = 5 (default=5)
17/05/26 16:38:53 INFO balancer.Balancer: dfs.balancer.getBlocks.size = 2147483648 (default=2147483648)
17/05/26 16:38:53 INFO balancer.Balancer: dfs.balancer.getBlocks.min-block-size = 10485760 (default=10485760)
17/05/26 16:38:53 INFO block.BlockTokenSecretManager: Setting block keys
17/05/26 16:38:53 INFO balancer.Balancer: dfs.balancer.max-size-to-move = 10737418240 (default=10737418240)
17/05/26 16:38:53 INFO balancer.Balancer: dfs.blocksize = 134217728 (default=134217728)
17/05/26 16:38:53 INFO net.NetworkTopology: Adding a new node: /default-rack/58.XXX.144.YYY:50010
17/05/26 16:38:53 INFO net.NetworkTopology: Adding a new node: /default-rack/58.XXX.144.YYY:50010
17/05/26 16:38:53 INFO net.NetworkTopology: Adding a new node: /default-rack/58.XXX.145.YY:50010
17/05/26 16:38:53 INFO net.NetworkTopology: Adding a new node: /default-rack/58.XXX.145.YY:50010
17/05/26 16:38:53 INFO net.NetworkTopology: Adding a new node: /default-rack/58.XXX.145.YY:50010
17/05/26 16:38:53 INFO net.NetworkTopology: Adding a new node: /default-rack/58.XXX.144.YY:50010
17/05/26 16:38:53 INFO balancer.Balancer: 0 over-utilized: []
17/05/26 16:38:53 INFO balancer.Balancer: 0 underutilized: []
The cluster is balanced. Exiting...
May 26, 2017 4:38:53 PM 0 0 B 0 B -1 B
May 26, 2017 4:38:54 PM Balancing took 2.773 seconds
The Ambari Host view indicates that the data is still not balanced across the nodes: hostviewambari.jpg ( Updated) The cluster has HA configuration (Primary-Secondary). Ambari : 2.2.1.0 Hadoop : 2.7.1.2.4.0.0-169 Any input will be helpfull.
... View more
Labels:
04-04-2017
02:40 AM
Hi All,
Problem Statement:
I have a source system that takes website data and sends out a lot of columns about 500+, At this point we only have interest in 100+ columns, we are trying to create a mechanism by which its easy to modify the HIVE table schema as the need to read more column arises. I have previously used AVRO schema files with JSON data and it worked quiet well to help add new columns easily to the schema file without altering data file. But currently the source system can only send flat CSV.
I am trying to create a HIVE table and read this file as below CREATE EXTERNAL TABLE players
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION 'hdfs://xxxxxcluster1/apps/hive/warehouse/test.db/players'
tblproperties ('avro.schema.url'='hdfs://xxxxxcluster1/tmp/prototype/players.avsc',"skip.header.line.count"="1"); Error: (I have changed all file permissions to 777) FAILED: SemanticException [Error 10043]: Either list of columns or a custom serializer should be specified I have not been able to find references to reading TEXTFILE with avro schema. Is this possible to do ? or is there a better way to make schema update modular with TEXTFILE data rather than having to drop and recreate table every time.
... View more
- Tags:
- Avro
- csv
- Data Processing
- Upgrade to HDP 2.5.3 : ConcurrentModificationException When Executing Insert Overwrite : Hive
Labels:
- Labels:
-
Apache Hive
-
HDFS
03-26-2017
03:54 AM
1 Kudo
Hi @Jay SenSharma,
Thanks for your input.
I removed the other repo files and have only "ambari.repo", "HDP.repo" and "HDP-UTILS.repo" and ran the following: sudo python /usr/lib/python2.6/site-packages/ambari_agent/HostCleanup.py --silent --skip=users
yum clean all
ls -ltr /etc/yum.repos.d/ But , i still could not get around the following error that was there previously: File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call raise Fail(err_msg) resource_management.core.exceptions.Fail:
Execution of '/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-monitor' returned 1. ERROR with rpm_check_debug vs depsolve:
libkadm5clnt_mit.so.8()(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64
libkadm5clnt_mit.so.8(kadm5clnt_mit_8_MIT)(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64
libkadm5srv_mit.so.8()(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64
libkadm5srv_mit.so.8(kadm5srv_mit_8_MIT)(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64
You could try running: rpm -Va --nofiles --nodigest Your transaction was saved, rerun it with:
yum load-transaction /tmp/yum_save_tx-2017-03-25-06-305zHb5j.yumtx After some googling I did the following on all hosts and i tried the CLUSTER INSTALL WIZARD from AMBARI again for Metrics Collector and Zookeeper installation on all EC2 nodes.
It worked !! 🙂
[ec2-user@ip-172-31-5-78 ~]$ yum install libkadm5
Loaded plugins: amazon-id, rhui-lb, security
Repo rhui-REGION-client-config-server-6 forced skip_if_unavailable=True due to: /etc/pki/rhui/cdn.redhat.com-chain.crt
Repo rhui-REGION-client-config-server-6 forced skip_if_unavailable=True due to: /etc/pki/rhui/product/rhui-client-config-server-6.crt
Repo rhui-REGION-client-config-server-6 forced skip_if_unavailable=True due to: /etc/pki/rhui/rhui-client-config-server-6.key
Repo rhui-REGION-rhel-server-releases forced skip_if_unavailable=True due to: /etc/pki/rhui/cdn.redhat.com-chain.crt
Repo rhui-REGION-rhel-server-releases forced skip_if_unavailable=True due to: /etc/pki/rhui/product/content-rhel6.crt
Repo rhui-REGION-rhel-server-releases forced skip_if_unavailable=True due to: /etc/pki/rhui/content-rhel6.key
Repo rhui-REGION-rhel-server-releases-optional forced skip_if_unavailable=True due to: /etc/pki/rhui/cdn.redhat.com-chain.crt
Repo rhui-REGION-rhel-server-releases-optional forced skip_if_unavailable=True due to: /etc/pki/rhui/product/content-rhel6.crt
Repo rhui-REGION-rhel-server-releases-optional forced skip_if_unavailable=True due to: /etc/pki/rhui/content-rhel6.key
Repo rhui-REGION-rhel-server-rh-common forced skip_if_unavailable=True due to: /etc/pki/rhui/cdn.redhat.com-chain.crt
Repo rhui-REGION-rhel-server-rh-common forced skip_if_unavailable=True due to: /etc/pki/rhui/product/content-rhel6.crt
Repo rhui-REGION-rhel-server-rh-common forced skip_if_unavailable=True due to: /etc/pki/rhui/content-rhel6.key
You need to be root to perform this command.
[ec2-user@ip-172-31-5-78 ~]$ sudo yum install libkadm5
Loaded plugins: amazon-id, rhui-lb, security
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package libkadm5.x86_64 0:1.10.3-65.el6 will be installed
--> Processing Dependency: krb5-libs(x86-64) = 1.10.3-65.el6 for package: libkadm5-1.10.3-65.el6.x86_64
--> Running transaction check
---> Package krb5-libs.x86_64 0:1.10.3-15.el6_5.1 will be updated
--> Processing Dependency: krb5-libs = 1.10.3-15.el6_5.1 for package: krb5-workstation-1.10.3-15.el6_5.1.x86_64
---> Package krb5-libs.x86_64 0:1.10.3-65.el6 will be an update
--> Running transaction check
---> Package krb5-workstation.x86_64 0:1.10.3-15.el6_5.1 will be updated
---> Package krb5-workstation.x86_64 0:1.10.3-65.el6 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
============================================================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================================================
Installing:
libkadm5 x86_64 1.10.3-65.el6 rhui-REGION-rhel-server-releases 143 k
Updating for dependencies:
krb5-libs x86_64 1.10.3-65.el6 rhui-REGION-rhel-server-releases 675 k
krb5-workstation x86_64 1.10.3-65.el6 rhui-REGION-rhel-server-releases 814 k
Transaction Summary
============================================================================================================================================================================
Install 1 Package(s)
Upgrade 2 Package(s)
Total size: 1.6 M
Total download size: 143 k
Is this ok [y/N]: y
Downloading Packages:
libkadm5-1.10.3-65.el6.x86_64.rpm | 143 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Updating : krb5-libs-1.10.3-65.el6.x86_64 1/5
Installing : libkadm5-1.10.3-65.el6.x86_64 2/5
Updating : krb5-workstation-1.10.3-65.el6.x86_64 3/5
Cleanup : krb5-workstation-1.10.3-15.el6_5.1.x86_64 4/5
Cleanup : krb5-libs-1.10.3-15.el6_5.1.x86_64 5/5
Verifying : krb5-libs-1.10.3-65.el6.x86_64 1/5
Verifying : libkadm5-1.10.3-65.el6.x86_64 2/5
Verifying : krb5-workstation-1.10.3-65.el6.x86_64 3/5
Verifying : krb5-libs-1.10.3-15.el6_5.1.x86_64 4/5
Verifying : krb5-workstation-1.10.3-15.el6_5.1.x86_64 5/5
Installed:
libkadm5.x86_64 0:1.10.3-65.el6
Dependency Updated:
krb5-libs.x86_64 0:1.10.3-65.el6 krb5-workstation.x86_64 0:1.10.3-65.el6
Complete!
... View more
03-25-2017
01:25 PM
I have 6 AWS Ec2-Instances (3 Masters / 3 Slaves). 1 has Public IP (My Main Server) and 5 have EIP. (I added EIPs on other servers in order to try and resolve the issue, assuming that some of the libraries might need to be downloaded from net.)
All AWS-EC2 instances are same Instance type-c3.xlarge Availability zone-ap-southeast-2a AMI ID -RHEL-6.5_GA-20140929-x86_64-11-Hourly2-GP2 (ami-eb2a47d1) I downloaded repos onto my Main Sever , changed the BASE URL for ambari.repo and HDP.repo and then pushed these to other servers. I updated the same local URL details in the AMBARI UI -> CLUSTER INSTALL WIZARD.
Issue came up when i tried to install Metrics Collector and Zookeeper on to the EC2 nodes. Repo information: [ec2-user@ip-172-31-7-74 ~]$ sudo yum repolistLoaded plugins: amazon-id, rhui-lb, security
Repository Updates-ambari-2.1.0 is listed more than once in the configuration
Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration
Repository HDP-2.3.0.0 is listed more than once in the configuration
Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration
HDP-UTILS-1.1.0.20 | 2.9 kB 00:00
repo id repo name status
HDP-2.3 HDP-2.3 175
HDP-2.3.0.0 HDP Version - HDP-2.3.0.0 175
HDP-UTILS-1.1.0.20 HDP-UTILS-1.1.0.20 43 Updates-ambari-2.1.0 ambari-2.1.0 - Updates 8
rhui-REGION-client-config-server-6 Red Hat Update Infrastructure 2.0 Client Configuration Server 6 6
rhui-REGION-rhel-server-releases Red Hat Enterprise Linux Server 6 (RPMs) 19,417
rhui-REGION-rhel-server-releases-optional Red Hat Enterprise Linux Server 6 Optional (RPMs) 11,057
rhui-REGION-rhel-server-rh-common Red Hat Enterprise Linux Server 6 RH Common (RPMs) 129
repolist: 31,010I executed the command from the error , on my main EC2 instance , Master01
--I tried to run the command that was erroring , from the log
[ec2-user@ip-172-31-7-74 ~]$ sudo yum -d 0 -e 0 -y install ambari-metrics-monitor
ERROR with rpm_check_debug vs depsolve:libkadm5clnt_mit.so.8()(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64
libkadm5clnt_mit.so.8(kadm5clnt_mit_8_MIT)(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64
libkadm5srv_mit.so.8()(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64
libkadm5srv_mit.so.8(kadm5srv_mit_8_MIT)(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64
You could try running: rpm -Va --nofiles --nodigestYour transaction was saved, rerun it with:
yum load-transaction /tmp/yum_save_tx-2017-03-25-07-01vjZxht.yumtx Content of repo directory on my Main Master [ec2-user@ip-172-31-7-74 ~]$ ls -l /etc/yum.repos.d/
total 44
-rw-r--r--. 1 root root 280 Mar 20 06:35 ambari_bkp.repo
-rw-r--r-- 1 ec2-user ec2-user 317 Mar 25 06:27 ambari.repo
-rw-r--r--. 1 root root 576 Mar 20 06:36 hdp_bkp.repo
-rw-r--r--. 1 root root 660 Mar 20 07:20 hdp.repo
-rw-r--r-- 1 root root 157 Mar 25 06:30 HDP.repo
-rw-r--r-- 1 root root 180 Mar 25 02:11 HDP-UTILS.repo
-rw-r--r-- 1 root root 606 Mar 24 23:27 redhat-rhui-client-config.repo
-rw-r--r--. 1 root root 6300 Mar 24 23:27 redhat-rhui.repo
-rw-r--r--. 1 root root 529 Oct 30 2013 rhel-source.repo
-rw-r--r-- 1 root root 90 Mar 24 23:27 rhui-load-balancers.conf
Content of ambari.repo and HDP.repo on my Main Master [ec2-user@ip-172-31-7-74 ~]$ cat /etc/yum.repos.d/ambari.repo #VERSION_NUMBER=2.1.0-1470
[Updates-ambari-2.1.0]
name=ambari-2.1.0 - Updates
baseurl=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/ambari-2.1.0/centos6/
gpgcheck=1
gpgkey=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/ambari-2.1.0/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[ec2-user@ip-172-31-7-74 ~]$ cat /etc/yum.repos.d/HDP.repo
[HDP-2.3]
name=HDP-2.3
baseurl=http://PublicIP/yum/HDP/centos6/2.x/updates/2.3.0.0
path=/
enabled=1
gpgcheck=0
[ec2-user@ip-172-31-7-74 ~]$ cat /etc/yum.repos.d/hdp.repo
#VERSION_NUMBER=2.3.0.0-2557
[HDP-2.3.0.0]
name=HDP Version - HDP-2.3.0.0
baseurl=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP/centos6/2.x/updates/2.3.0.0
gpgcheck=1
gpgkey=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP/centos6/2.x/updates/2.3.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.20]
name=HDP Utils Version - HDP-UTILS-1.1.0.20
baseurl=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP-UTILS-1.1.0.20/repos/centos6
gpgcheck=1
gpgkey=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP/centos6/2.x/updates/2.3.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
The repos have been copied over to other master and slaves, a view of the contents on those EC2:
[ec2-user@ip-172-31-5-78 ~]$ ls -l /etc/yum.repos.d/
total 36
-rw-r--r-- 1 ec2-user ec2-user 317 Mar 25 06:27 ambari.repo
-rw-r--r--. 1 ec2-user ec2-user 660 Mar 20 07:41 hdp.repo
-rw-r--r-- 1 root root 157 Mar 25 06:30 HDP.repo
-rw-r--r-- 1 root root 180 Mar 25 02:11 HDP-UTILS.repo
-rw-r--r-- 1 root root 606 Mar 20 04:59 redhat-rhui-client-config.repo
-rw-r--r-- 1 root root 6300 Mar 20 04:59 redhat-rhui.repo
-rw-r--r-- 1 root root 529 Oct 30 2013 rhel-source.repo
-rw-r--r--. 1 root root 90 Mar 20 04:59 rhui-load-balancers.conf
[ec2-user@ip-172-31-5-78 ~]$ cat /etc/yum.repos.d/ambari.repo
#VERSION_NUMBER=2.1.0-1470
[Updates-ambari-2.1.0]
name=ambari-2.1.0 - Updates
baseurl=http://ip-172-31-7-74.southeast.compute.internal/yum/ambari-2.1.0/centos6/
gpgcheck=1
gpgkey=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/ambari-2.1.0/centos6/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[ec2-user@ip-172-31-5-78 ~]$ cat /etc/yum.repos.d/hdp.repo
#VERSION_NUMBER=2.3.0.0-2557
[HDP-2.3.0.0]
name=HDP Version - HDP-2.3.0.0
baseurl=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP/centos6/2.x/updates/2.3.0.0
gpgcheck=1
gpgkey=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP/centos6/2.x/updates/2.3.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
[HDP-UTILS-1.1.0.20]
name=HDP Utils Version - HDP-UTILS-1.1.0.20
baseurl=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP-UTILS-1.1.0.20/repos/centos6
gpgcheck=1
gpgkey=http://ip-172-31-7-74.ap-southeast-2.compute.internal/yum/HDP/centos6/2.x/updates/2.3.0.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1
I get the following ambari-metrics-collector information from my main server. [ec2-user@ip-172-31-7-74 ~]$ rpm -qa | grep ambari-metrics
ambari-metrics-collector-2.1.0-1470.x86_64
I could not really find anything specific on net:
https://www.rpmfind.net/linux/rpm2html/search.php?query=libkadm5clnt_mit.so.8()(64bit)
https://rhn.redhat.com/errata/RHBA-2017-0643.html Any information on how to proceed will be helpfull. Errors:
--------------------------------------------------------------------
/var/lib/ambari-agent/data/errors-107.txt
--------------------------------------------------------------------
Repository Updates-ambari-2.1.0 is listed more than once in the configuration
Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration
Repository HDP-2.3.0.0 is listed more than once in the configuration
Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration
Repository Updates-ambari-2.1.0 is listed more than once in the configuration
Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration
Repository HDP-2.3.0.0 is listed more than once in the configuration
Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration
Repository Updates-ambari-2.1.0 is listed more than once in the configuration
Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration
Repository HDP-2.3.0.0 is listed more than once in the configuration
Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration
Repository Updates-ambari-2.1.0 is listed more than once in the configuration
Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration
Repository HDP-2.3.0.0 is listed more than once in the configuration
Repository HDP-UTILS-1.1.0.20 is listed more than once in the configuration
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_monitor.py", line 58, in
AmsMonitor().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 218, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_monitor.py", line 28, in install
self.install_packages(env, exclude_packages = ['ambari-metrics-collector'])
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 376, in install_packages
Package(name)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 157, in init
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 152, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 118, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 45, in action_install
self.install_package(package_name, self.resource.use_repos, self.resource.skip_repos)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 49, in install_package
shell.checked_call(cmd, sudo=True, logoutput=self.get_logoutput())
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 70, in inner
result = function(command, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 92, in checked_call
tries=tries, try_sleep=try_sleep)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 140, in callwrapper
result = call(command, **kwargscopy)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 291, in _call
raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-monitor' returned 1. ERROR with rpm_check_debug vs depsolve:
libkadm5clnt_mit.so.8()(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64
libkadm5clnt_mit.so.8(kadm5clnt_mit_8_MIT)(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64
libkadm5srv_mit.so.8()(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64
libkadm5srv_mit.so.8(kadm5srv_mit_8_MIT)(64bit) is needed by krb5-workstation-1.10.3-65.el6.x86_64
You could try running: rpm -Va --nofiles --nodigest
Your transaction was saved, rerun it with: yum load-transaction /tmp/yum_save_tx-2017-03-25-06-305zHb5j.yumtx --------------------------------------------------------------------
/var/lib/ambari-agent/data/output-107.txt
-------------------------------------------------------------------- 2017-03-25 06:30:20,223 - Directory['/var/lib/ambari-agent/data/tmp/AMBARI-artifacts/'] {'recursive': True}
2017-03-25 06:30:20,225 - File['/var/lib/ambari-agent/data/tmp/AMBARI-artifacts//jce_policy-8.zip'] {'content': DownloadSource('http://ip-172-31-7-74.ap-southeast-2.compute.internal:8080/resources//jce_policy-8.zip')}
2017-03-25 06:30:20,225 - Not downloading the file from http://ip-172-31-7-74.ap-southeast-2.compute.internal:8080/resources//jce_policy-8.zip, because /var/lib/ambari-agent/data/tmp/jce_policy-8.zip already exists
2017-03-25 06:30:20,226 - Group['hadoop'] {'ignore_failures': False}
2017-03-25 06:30:20,226 - User['zookeeper'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2017-03-25 06:30:20,227 - User['ams'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['hadoop']}
2017-03-25 06:30:20,228 - User['ambari-qa'] {'gid': 'hadoop', 'ignore_failures': False, 'groups': ['users']}
2017-03-25 06:30:20,228 - File['/var/lib/ambari-agent/data/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2017-03-25 06:30:20,229 - Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2017-03-25 06:30:20,240 - Skipping Execute['/var/lib/ambari-agent/data/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2017-03-25 06:30:20,256 - Repository['HDP-2.3'] {'base_url': 'http://PublicIP/yum/HDP/centos6/2.x/updates/2.3.0.0', 'action': ['create'], 'components': ['HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP', 'mirror_list': None}
2017-03-25 06:30:20,265 - File['/etc/yum.repos.d/HDP.repo'] {'content': InlineTemplate(...)}
2017-03-25 06:30:20,266 - Writing File['/etc/yum.repos.d/HDP.repo'] because it doesn't exist
2017-03-25 06:30:20,266 - Repository['HDP-UTILS-1.1.0.20'] {'base_url': 'http://PublicIP/yum/HDP-UTILS-1.1.0.20/repos/centos6', 'action': ['create'], 'components': ['HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None} 2017-03-25 06:30:20,269 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': InlineTemplate(...)}
2017-03-25 06:30:20,269 - Package['unzip'] {}
2017-03-25 06:30:20,365 - Skipping installation of existing package unzip
2017-03-25 06:30:20,366 - Package['curl'] {}
2017-03-25 06:30:20,408 - Skipping installation of existing package curl
2017-03-25 06:30:20,408 - Package['hdp-select'] {}
2017-03-25 06:30:20,450 - Installing package hdp-select ('/usr/bin/yum -d 0 -e 0 -y install hdp-select')
2017-03-25 06:30:22,167 - Directory['/var/lib/ambari-agent/data/tmp/AMBARI-artifacts/'] {'recursive': True}
2017-03-25 06:30:22,168 - File['/var/lib/ambari-agent/data/tmp/jdk-8u40-linux-x64.tar.gz'] {'content': DownloadSource('http://ip-172-31-7-74.ap-southeast-2.compute.internal:8080/resources//jdk-8u40-linux-x64.tar.gz'), 'not_if': 'test -f /var/lib/ambari-agent/data/tmp/jdk-8u40-linux-x64.tar.gz'}
2017-03-25 06:30:22,178 - Skipping File['/var/lib/ambari-agent/data/tmp/jdk-8u40-linux-x64.tar.gz'] due to not_if
2017-03-25 06:30:22,179 - Directory['/usr/jdk64'] {}
2017-03-25 06:30:22,180 - Execute['('chmod', 'a+x', '/usr/jdk64')'] {'not_if': 'test -e /usr/jdk64/jdk1.8.0_40/bin/java', 'sudo': True}
2017-03-25 06:30:22,189 - Skipping Execute['('chmod', 'a+x', '/usr/jdk64')'] due to not_if
2017-03-25 06:30:22,190 - Execute['mkdir -p /var/lib/ambari-agent/data/tmp/jdk && cd /var/lib/ambari-agent/data/tmp/jdk && tar -xf /var/lib/ambari-agent/data/tmp/jdk-8u40-linux-x64.tar.gz && ambari-sudo.sh cp -rp /var/lib/ambari-agent/data/tmp/jdk/* /usr/jdk64'] {'not_if': 'test -e /usr/jdk64/jdk1.8.0_40/bin/java'}
2017-03-25 06:30:22,199 - Skipping Execute['mkdir -p /var/lib/ambari-agent/data/tmp/jdk && cd /var/lib/ambari-agent/data/tmp/jdk && tar -xf /var/lib/ambari-agent/data/tmp/jdk-8u40-linux-x64.tar.gz && ambari-sudo.sh cp -rp /var/lib/ambari-agent/data/tmp/jdk/* /usr/jdk64'] due to not_if
2017-03-25 06:30:22,199 - File['/usr/jdk64/jdk1.8.0_40/bin/java'] {'mode': 0755, 'cd_access': 'a'}
2017-03-25 06:30:22,201 - Execute['('chgrp', '-R', 'hadoop', '/usr/jdk64/jdk1.8.0_40')'] {'sudo': True}
2017-03-25 06:30:22,231 - Execute['('chown', '-R', 'root', '/usr/jdk64/jdk1.8.0_40')'] {'sudo': True}
2017-03-25 06:30:22,441 - Package['ambari-metrics-monitor'] {}
2017-03-25 06:30:22,539 - Installing package ambari-metrics-monitor ('/usr/bin/yum -d 0 -e 0 -y install ambari-metrics-monitor')
... View more
Labels:
- Labels:
-
Apache Ambari
12-01-2016
03:44 AM
Can't find the cluster size calculator.. , Same issue.. looks like this thread in not being monitored.
... View more