org.apache.hadoop.hbase.PleaseHoldException: Master is initializing error

[#1]

I am getting following error while listing table from hbase shell. Below is error list-

[root@data1 hbase]# hbase shell SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.4.0.0-169/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] HBase Shell; enter 'help<RETURN>' for list of supported commands. Type "exit<RETURN>" to leave the HBase Shell Version 1.1.2.2.4.0.0-169, r61dfb2b344f424a11f93b3f086eab815c1eb0b6a, Wed Feb 10 07:08:51 UTC 2016 hbase(main):001:0> list TABLE ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing at org.apache.hadoop.hbase.master.HMaster.checkInitialized(HMaster.java:2314) at org.apache.hadoop.hbase.master.MasterRpcServices.getTableDescriptors(MasterRpcServices.java:853) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:53136) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101) at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130) at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107) at java.lang.Thread.run(Thread.java:745)

Hdfs directory structure---

[root@data1 hbase]# hdfs dfs -ls /apps/hbase Found 2 items drwxr-xr-x - hbase hdfs 0 2016-04-04 05:07 /apps/hbase/data drwx--x--x - hbase hdfs 0 2016-03-31 08:12 /apps/hbase/staging

Please let me know how to fix this?

[#2]

Can you paste the master logs?

[#3]

Your hmaster seems to be initialized very slowly. Is there any exceptions or errors in the hmaster's logs?

[#4]

below is hbase master log-

2016-04-04 05:47:05,382 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/usr/hdp/current/hbase-master/conf:/usr/jdk64/jdk1.8.0_60/lib/tools.jar:/usr/hdp/current/hbase-master/bin/..:/usr/hdp/current/hbase-master/bin/../lib/activation-1.1.jar:/usr/hdp/current/hbase-master/bin/../lib/aopalliance-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hbase-master/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hbase-master/bin/../lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hbase-master/bin/../lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hbase-master/bin/../lib/asm-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/avro-1.7.4.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-cli-1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-codec-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-collections-3.2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-compress-1.4.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-configuration-1.6.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-digester-1.8.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-el-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-httpclient-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-io-2.4.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-lang-2.6.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-logging-1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-math-2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-math3-3.1.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-net-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-client-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-framework-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/disruptor-3.3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/hdp/current/hbase-master/bin/../lib/gson-2.2.4.jar:/usr/hdp/current/hbase-master/bin/../lib/guava-12.0.1.jar:/usr/hdp/current/hbase-master/bin/../lib/guice-3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/guice-servlet-3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations-1.1.2.2.4.0.0-169-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-client-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-client.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common-1.1.2.2.4.0.0-169-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-examples-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-examples.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop2-compat-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop2-compat.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop-compat-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop-compat.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it-1.1.2.2.4.0.0-169-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-prefix-tree-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-prefix-tree.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-procedure-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-procedure.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-protocol-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-protocol.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-resource-bundle-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-resource-bundle.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-rest-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-rest.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server-1.1.2.2.4.0.0-169-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-shell-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-shell.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-thrift-1.1.2.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-thrift.jar:/usr/hdp/current/hbase-master/bin/../lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hbase-master/bin/../lib/httpclient-4.2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/httpcore-4.2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-core-2.2.3.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jamon-runtime-2.3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/jasper-compiler-5.5.23.jar:/usr/hdp/current/hbase-master/bin/../lib/jasper-runtime-5.5.23.jar:/usr/hdp/current/hbase-master/bin/../lib/javax.inject-1.jar:/usr/hdp/current/hbase-master/bin/../lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hbase-master/bin/../lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hbase-master/bin/../lib/jcodings-1.0.8.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-client-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-core-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-guice-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-json-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-server-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jets3t-0.9.0.jar:/usr/hdp/current/hbase-master/bin/../lib/jettison-1.3.3.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/joni-2.1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/jruby-complete-1.6.8.jar:/usr/hdp/current/hbase-master/bin/../lib/jsch-0.1.42.jar:/usr/hdp/current/hbase-master/bin/../lib/jsp-2.1-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/jsr305-1.3.9.jar:/usr/hdp/current/hbase-master/bin/../lib/junit-4.11.jar:/usr/hdp/current/hbase-master/bin/../lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hbase-master/bin/../lib/libthrift-0.9.0.jar:/usr/hdp/current/hbase-master/bin/../lib/log4j-1.2.17.jar:/usr/hdp/current/hbase-master/bin/../lib/metrics-core-2.2.0.jar:/usr/hdp/current/hbase-master/bin/../lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hbase-master/bin/../lib/netty-3.2.4.Final.jar:/usr/hdp/current/hbase-master/bin/../lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hbase-master/bin/../lib/ojdbc6.jar:/usr/hdp/current/hbase-master/bin/../lib/okhttp-2.4.0.jar:/usr/hdp/current/hbase-master/bin/../lib/okio-1.4.0.jar:/usr/hdp/current/hbase-master/bin/../lib/paranamer-2.3.jar:/usr/hdp/current/hbase-master/bin/../lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hbase-master/bin/../lib/ranger-hbase-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/hbase-master/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/servlet-api-2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/slf4j-api-1.7.7.jar:/usr/hdp/current/hbase-master/bin/../lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hbase-master/bin/../lib/spymemcached-2.11.6.jar:/usr/hdp/current/hbase-master/bin/../lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hbase-master/bin/../lib/xml-apis-1.3.04.jar:/usr/hdp/current/hbase-master/bin/../lib/xmlenc-0.52.jar:/usr/hdp/current/hbase-master/bin/../lib/xz-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/spark-yarn-shuffle.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ojdbc6.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ranger-yarn-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/activation-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/azure-storage-2.2.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/junit-4.11.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-annotations-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-annotations.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-auth.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-aws-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-aws.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-azure-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-azure.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-common-2.7.1.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-common-tests.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-common.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-nfs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/.//hadoop-nfs.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/./:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-2.7.1.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-nfs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/activation-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jettison-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/objenesis-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-digester-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/commons-net-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/fst-2.24.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/gson-2.2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/guice-3.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/zookeeper-3.4.6.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/javax.inject-1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jsch-0.1.42.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jsp-api-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-api-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-client-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-registry-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-registry.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-tests-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-timeline-plugins-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/guice-3.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/junit-4.11.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/lib/xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jaxb-api-2.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//activation-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-sls-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-sls.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hamcrest-core-1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//api-util-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-beanutils-1.7.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//httpcore-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-configuration-1.6.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//httpclient-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-digester-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-core-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-httpclient-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-lang3-3.3.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//java-xmlbuilder-0.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jets3t-0.9.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//commons-net-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//curator-client-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jackson-xc-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//curator-framework-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//mockito-all-1.8.5.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//curator-recipes-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jettison-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//gson-2.2.4.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-ant-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-ant.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-archives-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//joda-time-2.9.2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-archives.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jsch-0.1.42.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-auth.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-datajoin-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jsp-api-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-datajoin.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-distcp-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-openstack-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-extras-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//junit-4.11.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-extras.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-openstack.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-gridmix-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//metrics-core-3.0.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-gridmix.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-rumen-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-streaming-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.1.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//stax-api-1.0-2.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop-mapreduce/.//zookeeper-3.4.6.2.4.0.0-169.jar::mysql-connector-java.jar:/usr/hdp/2.4.0.0-169/hadoop/conf:/usr/hdp/2.4.0.0-169/hadoop/hadoop-annotations-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-annotations.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-auth.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-aws-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-aws.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-azure-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-azure.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-common-2.7.1.2.4.0.0-169-tests.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-common-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-common-tests.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-common.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-nfs-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/hadoop-nfs.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/spark-yarn-shuffle.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ojdbc6.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/ranger-yarn-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/activation-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/asm-3.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/azure-storage-2.2.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/xz-1.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/junit-4.11.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.4.0.0-169/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper-3.4.6.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/zookeeper/zookeeper.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-http-shared4-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.0.0-169/zookeeper/lib/xercesMinimal-1.9.6.2.jar: 2016-04-04 05:47:05,384 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native 2016-04-04 05:47:05,384 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2016-04-04 05:47:05,384 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2016-04-04 05:47:05,384 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2016-04-04 05:47:05,384 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2016-04-04 05:47:05,384 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-327.3.1.el7.x86_64 2016-04-04 05:47:05,384 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=hbase 2016-04-04 05:47:05,384 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/hbase 2016-04-04 05:47:05,384 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/hbase 2016-04-04 05:47:05,385 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181 sessionTimeout=90000 watcher=master:160000x0, quorum=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181, baseZNode=/hbase-unsecure 2016-04-04 05:47:05,403 INFO [main-SendThread(master2.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server master2.corp.mirrorplus.com/10.1.1.95:2181. Will not attempt to authenticate using SASL (unknown error) 2016-04-04 05:47:05,408 INFO [main-SendThread(master2.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Socket connection established to master2.corp.mirrorplus.com/10.1.1.95:2181, initiating session 2016-04-04 05:47:05,466 INFO [main-SendThread(master2.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server master2.corp.mirrorplus.com/10.1.1.95:2181, sessionid = 0x253e0a3fe3e0007, negotiated timeout = 40000 2016-04-04 05:47:05,551 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: starting 2016-04-04 05:47:05,551 INFO [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: starting 2016-04-04 05:47:05,612 INFO [main] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2016-04-04 05:47:05,617 INFO [main] http.HttpRequestLog: Http request log for http.requests.master is not defined 2016-04-04 05:47:05,631 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2016-04-04 05:47:05,633 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context master 2016-04-04 05:47:05,633 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2016-04-04 05:47:05,633 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2016-04-04 05:47:05,654 INFO [main] http.HttpServer: Jetty bound to port 16010 2016-04-04 05:47:05,654 INFO [main] mortbay.log: jetty-6.1.26.hwx 2016-04-04 05:47:06,186 INFO [main] mortbay.log: Started SelectChannelConnector@0.0.0.0:16010 2016-04-04 05:47:06,191 INFO [main] master.HMaster: hbase.rootdir=hdfs://aimprodcluster/apps/hbase/data1, hbase.cluster.distributed=true 2016-04-04 05:47:06,204 INFO [main] master.HMaster: Adding backup master ZNode /hbase-unsecure/backup-masters/master1.corp.mirrorplus.com,16000,1459763224469 2016-04-04 05:47:06,289 INFO [master1:16000.activeMasterManager] master.ActiveMasterManager: Another master is the active master, master3.corp.mirrorplus.com,16000,1459762844022; waiting to become the next active master 2016-04-04 05:47:06,336 INFO [master/master1.corp.mirrorplus.com/10.1.1.94:16000] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0xd8b2c68 connecting to ZooKeeper ensemble=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181 2016-04-04 05:47:06,336 INFO [master/master1.corp.mirrorplus.com/10.1.1.94:16000] zookeeper.ZooKeeper: Initiating client connection, connectString=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181 sessionTimeout=90000 watcher=hconnection-0xd8b2c680x0, quorum=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181, baseZNode=/hbase-unsecure 2016-04-04 05:47:06,337 INFO [master/master1.corp.mirrorplus.com/10.1.1.94:16000-SendThread(master3.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server master3.corp.mirrorplus.com/10.1.10.20:2181. Will not attempt to authenticate using SASL (unknown error) 2016-04-04 05:47:06,338 INFO [master/master1.corp.mirrorplus.com/10.1.1.94:16000-SendThread(master3.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Socket connection established to master3.corp.mirrorplus.com/10.1.10.20:2181, initiating session 2016-04-04 05:47:06,346 INFO [master/master1.corp.mirrorplus.com/10.1.1.94:16000-SendThread(master3.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server master3.corp.mirrorplus.com/10.1.10.20:2181, sessionid = 0x353e0a406670007, negotiated timeout = 40000 2016-04-04 05:47:06,373 INFO [master/master1.corp.mirrorplus.com/10.1.1.94:16000] regionserver.HRegionServer: ClusterId : 562355a3-3569-4d2e-bf16-efaa82632c96 2016-04-04 05:47:09,438 INFO [master1:16000.activeMasterManager] master.ActiveMasterManager: Another master is the active master, master2.corp.mirrorplus.com,16000,1459763227559; waiting to become the next active master

[#5]

master2.corp.mirrorplus.com,16000 is the active master, not the host whose logs you pasted.

[#6]

Hi Josh,

I changed standby and active masters, below is the log of master2.corp.mirrorplus.com-

2016-04-04 05:47:08,529 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native 2016-04-04 05:47:08,529 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2016-04-04 05:47:08,529 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2016-04-04 05:47:08,529 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2016-04-04 05:47:08,529 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2016-04-04 05:47:08,529 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-327.3.1.el7.x86_64 2016-04-04 05:47:08,529 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=hbase 2016-04-04 05:47:08,529 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/hbase 2016-04-04 05:47:08,529 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/hbase 2016-04-04 05:47:08,530 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181 sessionTimeout=90000 watcher=master:160000x0, quorum=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181, baseZNode=/hbase-unsecure 2016-04-04 05:47:08,575 INFO [main-SendThread(master1.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server master1.corp.mirrorplus.com/10.1.1.94:2181. Will not attempt to authenticate using SASL (unknown error) 2016-04-04 05:47:08,581 INFO [main-SendThread(master1.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Socket connection established to master1.corp.mirrorplus.com/10.1.1.94:2181, initiating session 2016-04-04 05:47:08,589 INFO [main-SendThread(master1.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server master1.corp.mirrorplus.com/10.1.1.94:2181, sessionid = 0x153e0a3fe360005, negotiated timeout = 40000 2016-04-04 05:47:08,648 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: starting 2016-04-04 05:47:08,648 INFO [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: starting 2016-04-04 05:47:08,710 INFO [main] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2016-04-04 05:47:08,716 INFO [main] http.HttpRequestLog: Http request log for http.requests.master is not defined 2016-04-04 05:47:08,730 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2016-04-04 05:47:08,733 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context master 2016-04-04 05:47:08,733 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2016-04-04 05:47:08,733 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2016-04-04 05:47:08,753 INFO [main] http.HttpServer: Jetty bound to port 16010 2016-04-04 05:47:08,753 INFO [main] mortbay.log: jetty-6.1.26.hwx 2016-04-04 05:47:09,146 INFO [main] mortbay.log: Started SelectChannelConnector@0.0.0.0:16010 2016-04-04 05:47:09,151 INFO [main] master.HMaster: hbase.rootdir=hdfs://aimprodcluster/apps/hbase/data1, hbase.cluster.distributed=true 2016-04-04 05:47:09,167 INFO [main] master.HMaster: Adding backup master ZNode /hbase-unsecure/backup-masters/master2.corp.mirrorplus.com,16000,1459763227559 2016-04-04 05:47:09,270 INFO [master2:16000.activeMasterManager] master.ActiveMasterManager: Another master is the active master, master3.corp.mirrorplus.com,16000,1459762844022; waiting to become the next active master 2016-04-04 05:47:09,307 INFO [master/master2.corp.mirrorplus.com/10.1.1.95:16000] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x7063e847 connecting to ZooKeeper ensemble=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181 2016-04-04 05:47:09,307 INFO [master/master2.corp.mirrorplus.com/10.1.1.95:16000] zookeeper.ZooKeeper: Initiating client connection, connectString=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181 sessionTimeout=90000 watcher=hconnection-0x7063e8470x0, quorum=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181, baseZNode=/hbase-unsecure 2016-04-04 05:47:09,308 INFO [master/master2.corp.mirrorplus.com/10.1.1.95:16000-SendThread(master2.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server master2.corp.mirrorplus.com/10.1.1.95:2181. Will not attempt to authenticate using SASL (unknown error) 2016-04-04 05:47:09,308 INFO [master/master2.corp.mirrorplus.com/10.1.1.95:16000-SendThread(master2.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Socket connection established to master2.corp.mirrorplus.com/10.1.1.95:2181, initiating session 2016-04-04 05:47:09,314 INFO [master/master2.corp.mirrorplus.com/10.1.1.95:16000-SendThread(master2.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server master2.corp.mirrorplus.com/10.1.1.95:2181, sessionid = 0x253e0a3fe3e0008, negotiated timeout = 40000 2016-04-04 05:47:09,338 INFO [master/master2.corp.mirrorplus.com/10.1.1.95:16000] regionserver.HRegionServer: ClusterId : 562355a3-3569-4d2e-bf16-efaa82632c96 2016-04-04 05:47:09,437 INFO [master2:16000.activeMasterManager] master.ActiveMasterManager: Deleting ZNode for /hbase-unsecure/backup-masters/master2.corp.mirrorplus.com,16000,1459763227559 from backup master directory 2016-04-04 05:47:09,444 INFO [master2:16000.activeMasterManager] master.ActiveMasterManager: Registered Active Master=master2.corp.mirrorplus.com,16000,1459763227559 2016-04-04 05:47:09,833 INFO [master2:16000.activeMasterManager] util.FSUtils: Created version file at hdfs://aimprodcluster/apps/hbase/data1 with version=8 2016-04-04 05:47:09,934 INFO [master2:16000.activeMasterManager] master.MasterFileSystem: BOOTSTRAP: creating hbase:meta region 2016-04-04 05:47:09,937 INFO [master2:16000.activeMasterManager] regionserver.HRegion: creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', BLOOMFILTER => 'NONE', VERSIONS => '10', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'false', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://aimprodcluster/apps/hbase/data1 Table name == hbase:meta 2016-04-04 05:47:10,010 INFO [master2:16000.activeMasterManager] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2016-04-04 05:47:10,035 INFO [master2:16000.activeMasterManager] wal.FSHLog: WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=hregion-44984721.default, suffix=, logDir=hdfs://aimprodcluster/apps/hbase/data1/WALs/hregion-44984721, archiveDir=hdfs://aimprodcluster/apps/hbase/data1/oldWALs 2016-04-04 05:47:10,085 INFO [master2:16000.activeMasterManager] wal.FSHLog: Slow sync cost: 33 ms, current pipeline: [] 2016-04-04 05:47:10,087 INFO [master2:16000.activeMasterManager] wal.FSHLog: New WAL /apps/hbase/data1/WALs/hregion-44984721/hregion-44984721.default.1459763230035 2016-04-04 05:47:10,182 INFO [StoreOpener-1588230740-1] hfile.CacheConfig: Allocating LruBlockCache size=5.99 GB, blockSize=64 KB

[#7]

Hi Josh,

Below is the log of master3.corp.mirrorplus.com-

2016-04-04 05:47:12,159 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/2.4.0.0-169/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.0.0-169/hadoop/lib/native 2016-04-04 05:47:12,159 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 2016-04-04 05:47:12,159 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 2016-04-04 05:47:12,159 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2016-04-04 05:47:12,159 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2016-04-04 05:47:12,159 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-327.3.1.el7.x86_64 2016-04-04 05:47:12,159 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=hbase 2016-04-04 05:47:12,159 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/hbase 2016-04-04 05:47:12,159 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/hbase 2016-04-04 05:47:12,160 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181 sessionTimeout=90000 watcher=master:160000x0, quorum=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181, baseZNode=/hbase-unsecure 2016-04-04 05:47:12,185 INFO [main-SendThread(master1.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server master1.corp.mirrorplus.com/10.1.1.94:2181. Will not attempt to authenticate using SASL (unknown error) 2016-04-04 05:47:12,192 INFO [main-SendThread(master1.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Socket connection established to master1.corp.mirrorplus.com/10.1.1.94:2181, initiating session 2016-04-04 05:47:12,201 INFO [main-SendThread(master1.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server master1.corp.mirrorplus.com/10.1.1.94:2181, sessionid = 0x153e0a3fe360007, negotiated timeout = 40000 2016-04-04 05:47:12,247 INFO [RpcServer.responder] ipc.RpcServer: RpcServer.responder: starting 2016-04-04 05:47:12,247 INFO [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: starting 2016-04-04 05:47:12,304 INFO [main] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2016-04-04 05:47:12,307 INFO [main] http.HttpRequestLog: Http request log for http.requests.master is not defined 2016-04-04 05:47:12,317 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2016-04-04 05:47:12,320 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context master 2016-04-04 05:47:12,320 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2016-04-04 05:47:12,320 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2016-04-04 05:47:12,335 INFO [main] http.HttpServer: Jetty bound to port 16010 2016-04-04 05:47:12,335 INFO [main] mortbay.log: jetty-6.1.26.hwx 2016-04-04 05:47:12,693 INFO [main] mortbay.log: Started SelectChannelConnector@0.0.0.0:16010 2016-04-04 05:47:12,696 INFO [main] master.HMaster: hbase.rootdir=hdfs://aimprodcluster/apps/hbase/data1, hbase.cluster.distributed=true 2016-04-04 05:47:12,707 INFO [main] master.HMaster: Adding backup master ZNode /hbase-unsecure/backup-masters/master3.corp.mirrorplus.com,16000,1459763231138 2016-04-04 05:47:12,807 INFO [master3:16000.activeMasterManager] master.ActiveMasterManager: Another master is the active master, master2.corp.mirrorplus.com,16000,1459763227559; waiting to become the next active master 2016-04-04 05:47:12,827 INFO [master/master3.corp.mirrorplus.com/10.1.10.20:16000] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x79c1e5c2 connecting to ZooKeeper ensemble=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181 2016-04-04 05:47:12,827 INFO [master/master3.corp.mirrorplus.com/10.1.10.20:16000] zookeeper.ZooKeeper: Initiating client connection, connectString=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181 sessionTimeout=90000 watcher=hconnection-0x79c1e5c20x0, quorum=master3.corp.mirrorplus.com:2181,master2.corp.mirrorplus.com:2181,master1.corp.mirrorplus.com:2181, baseZNode=/hbase-unsecure 2016-04-04 05:47:12,834 INFO [master/master3.corp.mirrorplus.com/10.1.10.20:16000-SendThread(master1.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Opening socket connection to server master1.corp.mirrorplus.com/10.1.1.94:2181. Will not attempt to authenticate using SASL (unknown error) 2016-04-04 05:47:12,835 INFO [master/master3.corp.mirrorplus.com/10.1.10.20:16000-SendThread(master1.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Socket connection established to master1.corp.mirrorplus.com/10.1.1.94:2181, initiating session 2016-04-04 05:47:12,839 INFO [master/master3.corp.mirrorplus.com/10.1.10.20:16000-SendThread(master1.corp.mirrorplus.com:2181)] zookeeper.ClientCnxn: Session establishment complete on server master1.corp.mirrorplus.com/10.1.1.94:2181, sessionid = 0x153e0a3fe360008, negotiated timeout = 40000 2016-04-04 05:47:12,864 INFO [master/master3.corp.mirrorplus.com/10.1.10.20:16000] regionserver.HRegionServer: ClusterId : a330e3a3-4f5c-402c-b934-013315c0b547

[#8 · Accepted solution]

@Raja Ray In the future, please use the attachments feature instead of pasting raw logs. They're not easily readable with the WYSIWYG rendering.

Your master2 node doesn't seem to have any errors, which means it may just be slow to start. You can try taking a thread dump of the master while it's in this initialization state to see what it is stuck doing (jstack, HBase web UI, or `kill -3 <pid>`). There are no obvious errors in the logs (as you can also see) as to why the process has not yet fully started.

[#9]

what is the solution?

[#10]

Josh, thanks a lot. I restarted the cluster. It worked. Probably the issue was happening for slow start.