question Re: HBase server fails to start - File does not exist: MasterProcWALs in Archives of Support Questions (Read Only)

HBase server fails to start - File does not exist: MasterProcWALs

wonhee_choi — Wed, 15 Jun 2016 12:09:12 GMT
Hi All,
HBase server fails to start with the error below.
A resolution suggested in another post of Hbase start issue - Deleting files under WALS path didn't help.
IF you could advise what can be done or investigated that would be a great help.
Regards,
-----------------------------
2016-06-15 15:02:38,636 INFO [ip-172-31-25-3:16000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase-unsecure/flush-table-proc/acquired /hbase-unsecure/flush-table-proc/reached /hbase-unsecure/flush-table-proc/abort 2016-06-15 15:02:38,643 INFO [ip-172-31-25-3:16000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase-unsecure/online-snapshot/acquired /hbase-unsecure/online-snapshot/reached /hbase-unsecure/online-snapshot/abort 2016-06-15 15:02:38,664 INFO [ip-172-31-25-3:16000.activeMasterManager] master.MasterCoprocessorHost: System coprocessor loading is enabled 2016-06-15 15:02:38,676 INFO [ip-172-31-25-3:16000.activeMasterManager] procedure2.ProcedureExecutor: Starting procedure executor threads=5 2016-06-15 15:02:38,677 INFO [ip-172-31-25-3:16000.activeMasterManager] wal.WALProcedureStore: Starting WAL Procedure Store lease recovery 2016-06-15 15:02:38,680 INFO [ip-172-31-25-3:16000.activeMasterManager] util.FSHDFSUtils: Recovering lease on dfs file hdfs://ip-172-31-25-3.ap-southeast-2.compute.internal:8020/apps/hbase/data/MasterProcWALs/state-************00000001.log 2016-06-15 15:02:38,692 FATAL [ip-172-31-25-3:16000.activeMasterManager] master.HMaster: Failed to become active master java.io.FileNotFoundException: File does not exist: /apps/hbase/data/MasterProcWALs/state-************00000001.log at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)
Re: HBase server fails to start - File does not exist: MasterProcWALs

tyu — Wed, 15 Jun 2016 20:16:53 GMT
Which release of hbase are you using ?
Re: HBase server fails to start - File does not exist: MasterProcWALs

tyu — Wed, 15 Jun 2016 20:57:39 GMT
Can you pastebin more of the master log before the error happened ?
Of particular interest is MasterProcWALs
Re: HBase server fails to start - File does not exist: MasterProcWALs

sjiang — Wed, 15 Jun 2016 21:31:02 GMT
A clarification: is this the only master, or is this the backup master trying to become active?
Also, the following trace in the question is incomplete. And I could not know where the hbase master throws the exception:
2016-06-15 15:02:38,692 FATAL [ip-172-31-25-3:16000.activeMasterManager] master.HMaster: Failed to become active master java.io.FileNotFoundException: File does not exist: /apps/hbase/data/MasterProcWALs/state-************00000001.log at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)
@WONHEE CHOI, Please put the full trace in the question (or pastebin master log). It is hard to know where the problem occurred in hbase as the trace only has the last half.
Re: HBase server fails to start - File does not exist: MasterProcWALs

wonhee_choi — Wed, 15 Jun 2016 21:50:28 GMT
Thanks Ted and sjiang.
This issue has gone without any action - not that I know of.
I have stopped the entire cluster since the issue, and when restarted it after a while hbase came up alright. No actions taken.
I will reply to your answers(so you can be notified) the full log if the issue comes back.
Again thanks.
Re: HBase server fails to start - File does not exist: MasterProcWALs

sjiang — Wed, 15 Jun 2016 22:35:57 GMT
Good to know. My guess (with full trace and master log) is that backup master tried to become active, but the active master probably still running and we only allow one master to access WAL file. That is why restarting cluster (or master) makes the problem go away.
Please let us know if the issue comes back again and we will look at the master log and figure out the root cause.
Re: HBase server fails to start - File does not exist: MasterProcWALs

wonhee_choi — Tue, 12 Jul 2016 15:37:26 GMT
Hi All,
This issue came back. We don't explicitly use Hbase in our environment so this intermittent hbase start-up issue is quite puzzling. We don't have any data we explicitly stored in habse.
Below is the full log of /var/log/hbase/hbase-hbase-master-xxx.log at the time of initial start-up and failure.
HDP 2.4
Hbase 1.1.2.2.4
Tue Jul 12 10:20:51 AEST 2016 Starting master on ip-172-31-25-3.ap-southeast-2.compute.internal
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127424
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 32000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 16000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
2016-07-12 10:20:52,975 INFO  [main] util.VersionInfo: HBase 1.1.2.2.4.2.0-258
2016-07-12 10:20:52,976 INFO  [main] util.VersionInfo: Source code repository file:///grid/0/jenkins/workspace/HDP-build-centos6/bigtop/build/hbase/rpm/BUILD/hbase-1.1.2.2.4.2.0 revision=Unknown
2016-07-12 10:20:52,976 INFO  [main] util.VersionInfo: Compiled by jenkins on Mon Apr 25 06:36:21 UTC 2016
2016-07-12 10:20:52,976 INFO  [main] util.VersionInfo: From source with checksum 4f661ee4f9f148ce7bfcad5b0d667c27
2016-07-12 10:20:53,424 INFO  [main] util.ServerCommandLine: env:PATH=/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent
2016-07-12 10:20:53,424 INFO  [main] util.ServerCommandLine: env:HISTCONTROL=ignoredups
2016-07-12 10:20:53,424 INFO  [main] util.ServerCommandLine: env:HBASE_PID_DIR=/var/run/hbase
2016-07-12 10:20:53,424 INFO  [main] util.ServerCommandLine: env:HBASE_REGIONSERVER_OPTS= -Xmn200m -XX:CMSInitiatingOccupancyFraction=70  -Xms1024m -Xmx1024m  
2016-07-12 10:20:53,424 INFO  [main] util.ServerCommandLine: env:HBASE_CONF_DIR=/usr/hdp/current/hbase-master/conf
2016-07-12 10:20:53,424 INFO  [main] util.ServerCommandLine: env:MAIL=/var/spool/mail/hbase
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:LD_LIBRARY_PATH=::/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:LOGNAME=hbase
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:HBASE_REST_OPTS=
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:PWD=/home/hbase
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:HBASE_ROOT_LOGGER=INFO,RFA
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:LESSOPEN=|/usr/bin/lesspipe.sh %s
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:SHELL=/bin/bash
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:HBASE_ENV_INIT=true
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:QTINC=/usr/lib64/qt-3.3/include
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:HBASE_MASTER_OPTS= -Xmx1024m 
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:HBASE_MANAGES_ZK=false
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:HBASE_REGIONSERVERS=/usr/hdp/current/hbase-master/conf/regionservers
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:HADOOP_HOME=/usr/hdp/2.4.2.0-258/hadoop
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:HBASE_NICENESS=0
2016-07-12 10:20:53,425 INFO  [main] util.ServerCommandLine: env:HBASE_OPTS=-Dhdp.version=2.4.2.0-258  -XX:+UseConcMarkSweepGC -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log -Djava.io.tmpdir=/tmp -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/hbase/gc.log-201607121020  -Xmx1024m  -Dhbase.log.dir=/var/log/hbase -Dhbase.log.file=hbase-hbase-master-ip-172-31-25-3.ap-southeast-2.compute.internal.log -Dhbase.home.dir=/usr/hdp/current/hbase-master/bin/.. -Dhbase.id.str=hbase -Dhbase.root.logger=INFO,RFA -Djava.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native -Dhbase.security.logger=INFO,RFAS
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:HBASE_START_FILE=/var/run/hbase/hbase-hbase-master.autorestart
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:HBASE_SECURITY_LOGGER=INFO,RFAS
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:SHLVL=3
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:ZOOKEEPER_HOME=/usr/hdp/2.4.2.0-258/zookeeper
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:HBASE_LOGFILE=hbase-hbase-master-ip-172-31-25-3.ap-southeast-2.compute.internal.log
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:HISTSIZE=1000
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:JAVA_HOME=/usr/jdk64/jdk1.8.0_60
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:HDP_VERSION=2.4.2.0-258
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:LANG=en_US.UTF-8
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:G_BROKEN_FILENAMES=1
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:HBASE_CLASSPATH=/usr/hdp/2.4.2.0-258/hadoop/conf:/usr/hdp/2.4.2.0-258/hadoop/*:/usr/hdp/2.4.2.0-258/hadoop/lib/*:/usr/hdp/2.4.2.0-258/zookeeper/*:/usr/hdp/2.4.2.0-258/zookeeper/lib/*:
2016-07-12 10:20:53,426 INFO  [main] util.ServerCommandLine: env:CVS_RSH=ssh
2016-07-12 10:20:53,427 INFO  [main] util.ServerCommandLine: env:HBASE_IDENT_STRING=hbase
2016-07-12 10:20:53,427 INFO  [main] util.ServerCommandLine: env:HBASE_ZNODE_FILE=/var/run/hbase/hbase-hbase-master.znode
2016-07-12 10:20:53,427 INFO  [main] util.ServerCommandLine: env:HBASE_LOG_PREFIX=hbase-hbase-master-ip-172-31-25-3.ap-southeast-2.compute.internal
2016-07-12 10:20:53,427 INFO  [main] util.ServerCommandLine: env:HBASE_LOG_DIR=/var/log/hbase
2016-07-12 10:20:53,427 INFO  [main] util.ServerCommandLine: env:USER=hbase
2016-07-12 10:20:53,427 INFO  [main] util.ServerCommandLine: env:CLASSPATH=/usr/hdp/current/hbase-master/conf:/usr/jdk64/jdk1.8.0_60/lib/tools.jar:/usr/hdp/current/hbase-master/bin/..:/usr/hdp/current/hbase-master/bin/../lib/activation-1.1.jar:/usr/hdp/current/hbase-master/bin/../lib/aopalliance-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hbase-master/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hbase-master/bin/../lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hbase-master/bin/../lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hbase-master/bin/../lib/asm-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/avro-1.7.4.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-cli-1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-codec-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-collections-3.2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-compress-1.4.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-configuration-1.6.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-digester-1.8.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-el-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-httpclient-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-io-2.4.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-lang-2.6.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-logging-1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-math-2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-math3-3.1.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-net-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-client-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-framework-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/disruptor-3.3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/hdp/current/hbase-master/bin/../lib/gson-2.2.4.jar:/usr/hdp/current/hbase-master/bin/../lib/guava-12.0.1.jar:/usr/hdp/current/hbase-master/bin/../lib/guice-3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/guice-servlet-3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-client-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-client.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-examples-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-examples.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop2-compat-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop2-compat.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop-compat-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop-compat.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-prefix-tree-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-prefix-tree.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-procedure-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-procedure.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-protocol-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-protocol.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-resource-bundle-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-resource-bundle.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-rest-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-rest.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-shell-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-shell.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-thrift-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-thrift.jar:/usr/hdp/current/hbase-master/bin/../lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hbase-master/bin/../lib/httpclient-4.2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/httpcore-4.2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-core-2.2.3.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jamon-runtime-2.3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/jasper-compiler-5.5.23.jar:/usr/hdp/current/hbase-master/bin/../lib/jasper-runtime-5.5.23.jar:/usr/hdp/current/hbase-master/bin/../lib/javax.inject-1.jar:/usr/hdp/current/hbase-master/bin/../lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hbase-master/bin/../lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hbase-master/bin/../lib/jcodings-1.0.8.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-client-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-core-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-guice-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-json-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-server-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jets3t-0.9.0.jar:/usr/hdp/current/hbase-master/bin/../lib/jettison-1.3.3.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/joni-2.1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/jruby-complete-1.6.8.jar:/usr/hdp/current/hbase-master/bin/../lib/jsch-0.1.42.jar:/usr/hdp/current/hbase-master/bin/../lib/jsp-2.1-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/jsr305-1.3.9.jar:/usr/hdp/current/hbase-master/bin/../lib/junit-4.11.jar:/usr/hdp/current/hbase-master/bin/../lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hbase-master/bin/../lib/libthrift-0.9.0.jar:/usr/hdp/current/hbase-master/bin/../lib/log4j-1.2.17.jar:/usr/hdp/current/hbase-master/bin/../lib/metrics-core-2.2.0.jar:/usr/hdp/current/hbase-master/bin/../lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hbase-master/bin/../lib/netty-3.2.4.Final.jar:/usr/hdp/current/hbase-master/bin/../lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hbase-master/bin/../lib/ojdbc6.jar:/usr/hdp/current/hbase-master/bin/../lib/okhttp-2.4.0.jar:/usr/hdp/current/hbase-master/bin/../lib/okio-1.4.0.jar:/usr/hdp/current/hbase-master/bin/../lib/paranamer-2.3.jar:/usr/hdp/current/hbase-master/bin/../lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hbase-master/bin/../lib/ranger-hbase-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/ranger-plugin-classloader-0.5.0.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/servlet-api-2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/slf4j-api-1.7.7.jar:/usr/hdp/current/hbase-master/bin/../lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hbase-master/bin/../lib/spymemcached-2.11.6.jar:/usr/hdp/current/hbase-master/bin/../lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hbase-master/bin/../lib/xml-apis-1.3.04.jar:/usr/hdp/current/hbase-master/bin/../lib/xmlenc-0.52.jar:/usr/hdp/current/hbase-master/bin/../lib/xz-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/zookeeper.jar:/usr/hdp/2.4.2.0-258/hadoop/conf:/usr/hdp/2.4.2.0-258/hadoop/lib/*:/usr/hdp/2.4.2.0-258/hadoop/.//*:/usr/hdp/2.4.2.0-258/hadoop-hdfs/./:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/*:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//*:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/*:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//*:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/*:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//*::mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.4.2.0-258/tez/*:/usr/hdp/2.4.2.0-258/tez/lib/*:/usr/hdp/2.4.2.0-258/tez/conf:/usr/hdp/2.4.2.0-258/hadoop/conf:/usr/hdp/2.4.2.0-258/hadoop/*:/usr/hdp/2.4.2.0-258/hadoop/lib/*:/usr/hdp/2.4.2.0-258/zookeeper/*:/usr/hdp/2.4.2.0-258/zookeeper/lib/*:
2016-07-12 10:20:53,429 INFO  [main] util.ServerCommandLine: env:SERVER_GC_OPTS=-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/var/log/hbase/gc.log-201607121020
2016-07-12 10:20:53,429 INFO  [main] util.ServerCommandLine: env:HADOOP_CONF=/usr/hdp/2.4.2.0-258/hadoop/conf
2016-07-12 10:20:53,429 INFO  [main] util.ServerCommandLine: env:HOSTNAME=ip-172-31-25-3.ap-southeast-2.compute.internal
2016-07-12 10:20:53,429 INFO  [main] util.ServerCommandLine: env:QTDIR=/usr/lib64/qt-3.3
2016-07-12 10:20:53,429 INFO  [main] util.ServerCommandLine: env:NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat
2016-07-12 10:20:53,429 INFO  [main] util.ServerCommandLine: env:HBASE_THRIFT_OPTS=
2016-07-12 10:20:53,429 INFO  [main] util.ServerCommandLine: env:HBASE_HOME=/usr/hdp/current/hbase-master/bin/..
2016-07-12 10:20:53,429 INFO  [main] util.ServerCommandLine: env:QTLIB=/usr/lib64/qt-3.3/lib
2016-07-12 10:20:53,429 INFO  [main] util.ServerCommandLine: env:HOME=/home/hbase
2016-07-12 10:20:53,430 INFO  [main] util.ServerCommandLine: env:MALLOC_ARENA_MAX=4
2016-07-12 10:20:53,432 INFO  [main] util.ServerCommandLine: vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Oracle Corporation, vmVersion=25.60-b23
2016-07-12 10:20:53,433 INFO  [main] util.ServerCommandLine: vmInputArguments=[-Dproc_master, -XX:OnOutOfMemoryError=kill -9 %p, -Dhdp.version=2.4.2.0-258, -XX:+UseConcMarkSweepGC, -XX:ErrorFile=/var/log/hbase/hs_err_pid%p.log, -Djava.io.tmpdir=/tmp, -verbose:gc, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -Xloggc:/var/log/hbase/gc.log-201607121020, -Xmx1024m, -Dhbase.log.dir=/var/log/hbase, -Dhbase.log.file=hbase-hbase-master-ip-172-31-25-3.ap-southeast-2.compute.internal.log, -Dhbase.home.dir=/usr/hdp/current/hbase-master/bin/.., -Dhbase.id.str=hbase, -Dhbase.root.logger=INFO,RFA, -Djava.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native, -Dhbase.security.logger=INFO,RFAS]
2016-07-12 10:20:54,405 INFO  [main] regionserver.RSRpcServices: master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000 server-side HConnection retries=350
2016-07-12 10:20:54,559 INFO  [main] ipc.SimpleRpcScheduler: Using deadline as user call queue, count=3
2016-07-12 10:20:54,586 INFO  [main] ipc.RpcServer: master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000: started 10 reader(s).
2016-07-12 10:20:54,667 INFO  [main] impl.MetricsConfig: loaded properties from hadoop-metrics2-hbase.properties
2016-07-12 10:20:54,700 INFO  [main] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink.
2016-07-12 10:20:54,701 INFO  [main] timeline.HadoopTimelineMetricsSink: Identified hostname = ip-172-31-25-3.ap-southeast-2.compute.internal, serviceName = hbase
2016-07-12 10:20:54,706 INFO  [main] timeline.HadoopTimelineMetricsSink: Collector Uri: http://ip-172-31-25-3.ap-southeast-2.compute.internal:6188/ws/v1/timeline/metrics
2016-07-12 10:20:54,719 INFO  [main] impl.MetricsSinkAdapter: Sink timeline started
2016-07-12 10:20:54,795 INFO  [main] impl.MetricsSystemImpl: Scheduled snapshot period at 60 second(s).
2016-07-12 10:20:54,795 INFO  [main] impl.MetricsSystemImpl: HBase metrics system started
2016-07-12 10:20:55,890 INFO  [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2016-07-12 10:20:56,245 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=master:16000 connecting to ZooKeeper ensemble=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181
2016-07-12 10:20:56,260 INFO  [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-258--1, built on 04/25/2016 05:22 GMT
2016-07-12 10:20:56,260 INFO  [main] zookeeper.ZooKeeper: Client environment:host.name=ip-172-31-25-3.ap-southeast-2.compute.internal
2016-07-12 10:20:56,260 INFO  [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_60
2016-07-12 10:20:56,260 INFO  [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2016-07-12 10:20:56,260 INFO  [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/jdk64/jdk1.8.0_60/jre
2016-07-12 10:20:56,260 INFO  [main] zookeeper.ZooKeeper: Client environment:java.class.path=/usr/hdp/current/hbase-master/conf:/usr/jdk64/jdk1.8.0_60/lib/tools.jar:/usr/hdp/current/hbase-master/bin/..:/usr/hdp/current/hbase-master/bin/../lib/activation-1.1.jar:/usr/hdp/current/hbase-master/bin/../lib/aopalliance-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hbase-master/bin/../lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hbase-master/bin/../lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hbase-master/bin/../lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hbase-master/bin/../lib/asm-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/avro-1.7.4.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-cli-1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-codec-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-collections-3.2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-compress-1.4.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-configuration-1.6.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-daemon-1.0.13.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-digester-1.8.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-el-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-httpclient-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-io-2.4.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-lang-2.6.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-logging-1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-math-2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-math3-3.1.1.jar:/usr/hdp/current/hbase-master/bin/../lib/commons-net-3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-client-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-framework-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/curator-recipes-2.7.1.jar:/usr/hdp/current/hbase-master/bin/../lib/disruptor-3.3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/findbugs-annotations-1.3.9-1.jar:/usr/hdp/current/hbase-master/bin/../lib/gson-2.2.4.jar:/usr/hdp/current/hbase-master/bin/../lib/guava-12.0.1.jar:/usr/hdp/current/hbase-master/bin/../lib/guice-3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/guice-servlet-3.0.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-annotations.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-client-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-client.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-common.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-examples-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-examples.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop2-compat-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop2-compat.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop-compat-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-hadoop-compat.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-it.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-prefix-tree-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-prefix-tree.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-procedure-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-procedure.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-protocol-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-protocol.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-resource-bundle-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-resource-bundle.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-rest-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-rest.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server-1.1.2.2.4.2.0-258-tests.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-server.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-shell-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-shell.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-thrift-1.1.2.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/hbase-thrift.jar:/usr/hdp/current/hbase-master/bin/../lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/current/hbase-master/bin/../lib/httpclient-4.2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/httpcore-4.2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-core-2.2.3.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-core-asl-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jackson-xc-1.9.13.jar:/usr/hdp/current/hbase-master/bin/../lib/jamon-runtime-2.3.1.jar:/usr/hdp/current/hbase-master/bin/../lib/jasper-compiler-5.5.23.jar:/usr/hdp/current/hbase-master/bin/../lib/jasper-runtime-5.5.23.jar:/usr/hdp/current/hbase-master/bin/../lib/javax.inject-1.jar:/usr/hdp/current/hbase-master/bin/../lib/java-xmlbuilder-0.4.jar:/usr/hdp/current/hbase-master/bin/../lib/jaxb-api-2.2.2.jar:/usr/hdp/current/hbase-master/bin/../lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/current/hbase-master/bin/../lib/jcodings-1.0.8.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-client-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-core-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-guice-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-json-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jersey-server-1.9.jar:/usr/hdp/current/hbase-master/bin/../lib/jets3t-0.9.0.jar:/usr/hdp/current/hbase-master/bin/../lib/jettison-1.3.3.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-sslengine-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/current/hbase-master/bin/../lib/joni-2.1.2.jar:/usr/hdp/current/hbase-master/bin/../lib/jruby-complete-1.6.8.jar:/usr/hdp/current/hbase-master/bin/../lib/jsch-0.1.42.jar:/usr/hdp/current/hbase-master/bin/../lib/jsp-2.1-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/jsp-api-2.1-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/jsr305-1.3.9.jar:/usr/hdp/current/hbase-master/bin/../lib/junit-4.11.jar:/usr/hdp/current/hbase-master/bin/../lib/leveldbjni-all-1.8.jar:/usr/hdp/current/hbase-master/bin/../lib/libthrift-0.9.0.jar:/usr/hdp/current/hbase-master/bin/../lib/log4j-1.2.17.jar:/usr/hdp/current/hbase-master/bin/../lib/metrics-core-2.2.0.jar:/usr/hdp/current/hbase-master/bin/../lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/current/hbase-master/bin/../lib/netty-3.2.4.Final.jar:/usr/hdp/current/hbase-master/bin/../lib/netty-all-4.0.23.Final.jar:/usr/hdp/current/hbase-master/bin/../lib/ojdbc6.jar:/usr/hdp/current/hbase-master/bin/../lib/okhttp-2.4.0.jar:/usr/hdp/current/hbase-master/bin/../lib/okio-1.4.0.jar:/usr/hdp/current/hbase-master/bin/../lib/paranamer-2.3.jar:/usr/hdp/current/hbase-master/bin/../lib/protobuf-java-2.5.0.jar:/usr/hdp/current/hbase-master/bin/../lib/ranger-hbase-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/ranger-plugin-classloader-0.5.0.2.4.2.0-258.jar:/usr/hdp/current/hbase-master/bin/../lib/servlet-api-2.5-6.1.14.jar:/usr/hdp/current/hbase-master/bin/../lib/servlet-api-2.5.jar:/usr/hdp/current/hbase-master/bin/../lib/slf4j-api-1.7.7.jar:/usr/hdp/current/hbase-master/bin/../lib/snappy-java-1.0.4.1.jar:/usr/hdp/current/hbase-master/bin/../lib/spymemcached-2.11.6.jar:/usr/hdp/current/hbase-master/bin/../lib/xercesImpl-2.9.1.jar:/usr/hdp/current/hbase-master/bin/../lib/xml-apis-1.3.04.jar:/usr/hdp/current/hbase-master/bin/../lib/xmlenc-0.52.jar:/usr/hdp/current/hbase-master/bin/../lib/xz-1.0.jar:/usr/hdp/current/hbase-master/bin/../lib/zookeeper.jar:/usr/hdp/2.4.2.0-258/hadoop/conf:/usr/hdp/2.4.2.0-258/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/azure-storage-2.2.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ojdbc6.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/asm-3.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/activation-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/junit-4.11.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/zookeeper-3.4.6.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-yarn-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/xz-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/spark-yarn-shuffle.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-aws.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-azure-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-common-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-annotations.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-azure.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-common.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-nfs-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-common-tests.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-aws-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-auth-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-common-2.7.1.2.4.2.0-258-tests.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-annotations-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-nfs.jar:/usr/hdp/2.4.2.0-258/hadoop/.//hadoop-auth.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/./:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/okhttp-2.4.0.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/asm-3.2.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/xercesImpl-2.9.1.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/okio-1.4.0.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/commons-lang-2.6.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/xml-apis-1.3.04.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/netty-all-4.0.23.Final.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//hadoop-hdfs-nfs-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//hadoop-hdfs-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//hadoop-hdfs-2.7.1.2.4.2.0-258-tests.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//hadoop-hdfs-tests.jar:/usr/hdp/2.4.2.0-258/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jsp-api-2.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/paranamer-2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/gson-2.2.4.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/fst-2.24.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/asm-3.2.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/javax.inject-1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jettison-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-digester-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/objenesis-2.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/guice-3.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jsch-0.1.42.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/activation-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-net-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-codec-1.4.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/javassist-3.18.1-GA.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-cli-1.2.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jersey-json-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/zookeeper-3.4.6.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/xmlenc-0.52.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/avro-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-lang-2.6.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jersey-client-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/zookeeper-3.4.6.2.4.2.0-258-tests.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/xz-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/guava-11.0.2.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-timeline-plugins-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-tests-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-timeline-plugins.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-common-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-api-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-registry.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-common-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-client-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/hdp/2.4.2.0-258/hadoop-yarn/.//hadoop-yarn-registry-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/jersey-guice-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/paranamer-2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/asm-3.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/javax.inject-1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/aopalliance-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/guice-3.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/jersey-server-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/junit-4.11.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/avro-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/guice-servlet-3.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/jersey-core-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/xz-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/log4j-1.2.17.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/lib/leveldbjni-all-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jsp-api-2.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-archives-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-common-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//paranamer-2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//gson-2.2.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-datajoin.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-app-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-distcp-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-httpclient-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//asm-3.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-sls.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//api-util-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-extras-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jettison-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-digester-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-extras.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//snappy-java-1.0.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//curator-recipes-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-sls-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jsch-0.1.42.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//httpclient-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//curator-client-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jersey-server-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//activation-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-net-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-beanutils-1.7.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-codec-1.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//joda-time-2.9.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-cli-1.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jersey-json-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-tests.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//junit-4.11.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jaxb-api-2.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-examples-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//zookeeper-3.4.6.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-math3-3.1.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-streaming-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-archives.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-rumen-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-ant-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jackson-xc-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-datajoin-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-openstack.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-logging-1.1.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jackson-core-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-gridmix-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//xmlenc-0.52.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//avro-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-gridmix.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//metrics-core-3.0.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-compress-1.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-lang-2.6.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-core-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-configuration-1.6.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//mockito-all-1.8.5.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//stax-api-1.0-2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-auth-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hamcrest-core-1.3.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jsr305-3.0.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-ant.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jersey-core-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//xz-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//log4j-1.2.17.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jetty-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-collections-3.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-openstack-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.1.2.4.2.0-258-tests.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jets3t-0.9.0.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//guava-11.0.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//java-xmlbuilder-0.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//netty-3.6.2.Final.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-auth.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//httpcore-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//commons-lang3-3.3.2.jar:/usr/hdp/2.4.2.0-258/hadoop-mapreduce/.//curator-framework-2.7.1.jar::mysql-connector-java-5.1.17.jar:mysql-connector-java.jar:/usr/hdp/2.4.2.0-258/tez/tez-yarn-timeline-history-with-fs-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-examples-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-history-parser-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-yarn-timeline-cache-plugin-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-runtime-internals-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-mapreduce-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-yarn-timeline-history-with-acls-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-api-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-runtime-library-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-tests-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-yarn-timeline-history-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-dag-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/tez-common-0.7.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-yarn-server-timeline-plugins-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-azure-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-mapreduce-client-common-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-yarn-server-web-proxy-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-aws-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-collections4-4.1.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-codec-1.4.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-cli-1.2.jar:/usr/hdp/2.4.2.0-258/tez/lib/jersey-json-1.9.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-lang-2.6.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-mapreduce-client-core-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/jersey-client-1.9.jar:/usr/hdp/2.4.2.0-258/tez/lib/jettison-1.3.4.jar:/usr/hdp/2.4.2.0-258/tez/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/tez/lib/jsr305-2.0.3.jar:/usr/hdp/2.4.2.0-258/tez/lib/hadoop-annotations-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/tez/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.2.0-258/tez/lib/guava-11.0.2.jar:/usr/hdp/2.4.2.0-258/tez/lib/commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/tez/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/tez/lib/slf4j-api-1.7.5.jar:/usr/hdp/2.4.2.0-258/tez/conf:/usr/hdp/2.4.2.0-258/hadoop/conf:/usr/hdp/2.4.2.0-258/hadoop/hadoop-aws.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-azure-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-common-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-annotations.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-azure.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-common.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-nfs-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-common-tests.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-aws-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-auth-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-common-2.7.1.2.4.2.0-258-tests.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-annotations-2.7.1.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-nfs.jar:/usr/hdp/2.4.2.0-258/hadoop/hadoop-auth.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/azure-storage-2.2.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ojdbc6.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/gson-2.2.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/htrace-core-3.1.0-incubating.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/asm-3.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/api-util-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-databind-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/protobuf-java-2.5.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-digester-1.8.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-plugin-classloader-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/curator-recipes-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-annotations-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jsch-0.1.42.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/curator-client-2.7.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/activation-1.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-codec-1.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-cli-1.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jersey-json-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/aws-java-sdk-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/junit-4.11.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/zookeeper-3.4.6.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-math3-3.1.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-hdfs-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/ranger-yarn-plugin-shim-0.5.0.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-logging-1.1.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/avro-1.7.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-compress-1.4.1.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-lang-2.6.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-configuration-1.6.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/mockito-all-1.8.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/stax-api-1.0-2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/hamcrest-core-1.3.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jsr305-3.0.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/xz-1.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jetty-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-collections-3.2.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/spark-yarn-shuffle.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/guava-11.0.2.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/netty-3.6.2.Final.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.4.2.0-258/hadoop/lib/curator-framework-2.7.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/zookeeper.jar:/usr/hdp/2.4.2.0-258/zookeeper/zookeeper-3.4.6.2.4.2.0-258.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/httpcore-4.2.3.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/slf4j-api-1.6.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/commons-codec-1.6.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-error-diagnostics-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/netty-3.7.0.Final.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-model-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/wagon-http-2.4.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/httpclient-4.2.3.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-profile-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/jsoup-1.7.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/plexus-utils-3.0.8.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/plexus-interpolation-1.11.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-project-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/nekohtml-1.9.6.2.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/ant-1.8.0.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/wagon-file-1.0-beta-6.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-repository-metadata-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/backport-util-concurrent-3.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/commons-io-2.2.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/classworlds-1.1-alpha-2.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/xercesMinimal-1.9.6.2.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/log4j-1.2.16.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/wagon-http-shared-1.0-beta-6.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-ant-tasks-2.1.3.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/plexus-container-default-1.0-alpha-9-stable-1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/ant-launcher-1.8.0.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/wagon-provider-api-2.4.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-settings-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/wagon-http-lightweight-1.0-beta-6.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-plugin-registry-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/jline-0.9.94.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/commons-logging-1.1.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/slf4j-log4j12-1.6.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-artifact-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/maven-artifact-manager-2.2.1.jar:/usr/hdp/2.4.2.0-258/zookeeper/lib/wagon-http-shared4-2.4.jar:
2016-07-12 10:20:56,280 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=:/usr/hdp/2.4.2.0-258/hadoop/lib/native/Linux-amd64-64:/usr/hdp/2.4.2.0-258/hadoop/lib/native
2016-07-12 10:20:56,280 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2016-07-12 10:20:56,280 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2016-07-12 10:20:56,280 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2016-07-12 10:20:56,280 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2016-07-12 10:20:56,280 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=2.6.32-431.29.2.el6.x86_64
2016-07-12 10:20:56,280 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=hbase
2016-07-12 10:20:56,280 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/home/hbase
2016-07-12 10:20:56,280 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/hbase
2016-07-12 10:20:56,282 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181 sessionTimeout=90000 watcher=master:160000x0, quorum=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181, baseZNode=/hbase-unsecure
2016-07-12 10:20:56,302 INFO  [main-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-12 10:20:56,317 INFO  [main-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181, initiating session
2016-07-12 10:20:56,332 INFO  [main-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181, sessionid = 0x155dc78eeb40000, negotiated timeout = 40000
2016-07-12 10:20:56,422 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: starting
2016-07-12 10:20:56,431 INFO  [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: starting
2016-07-12 10:20:57,088 INFO  [main] mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2016-07-12 10:20:57,095 INFO  [main] http.HttpRequestLog: Http request log for http.requests.master is not defined
2016-07-12 10:20:57,171 INFO  [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter)
2016-07-12 10:20:57,174 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context master
2016-07-12 10:20:57,174 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2016-07-12 10:20:57,174 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2016-07-12 10:20:57,235 INFO  [main] http.HttpServer: Jetty bound to port 16010
2016-07-12 10:20:57,235 INFO  [main] mortbay.log: jetty-6.1.26.hwx
2016-07-12 10:20:58,271 INFO  [main] mortbay.log: Started SelectChannelConnector@0.0.0.0:16010
2016-07-12 10:20:58,281 INFO  [main] master.HMaster: hbase.rootdir=hdfs://ip-172-31-25-3.ap-southeast-2.compute.internal:8020/apps/hbase/data, hbase.cluster.distributed=true
2016-07-12 10:20:58,324 INFO  [main] master.HMaster: Adding backup master ZNode /hbase-unsecure/backup-masters/ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815
2016-07-12 10:20:58,540 INFO  [ip-172-31-25-3:16000.activeMasterManager] master.ActiveMasterManager: Deleting ZNode for /hbase-unsecure/backup-masters/ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815 from backup master directory
2016-07-12 10:20:58,555 INFO  [ip-172-31-25-3:16000.activeMasterManager] master.ActiveMasterManager: Registered Active Master=ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815
2016-07-12 10:20:58,711 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x570673c0 connecting to ZooKeeper ensemble=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181
2016-07-12 10:20:58,723 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181 sessionTimeout=90000 watcher=hconnection-0x570673c00x0, quorum=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181, baseZNode=/hbase-unsecure
2016-07-12 10:20:58,744 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-12 10:20:58,745 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181, initiating session
2016-07-12 10:20:58,752 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181, sessionid = 0x155dc78eeb40001, negotiated timeout = 40000
2016-07-12 10:20:58,822 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] regionserver.HRegionServer: ClusterId : 1e143dd7-84d3-4e4c-ab04-e9bc79728b2f
2016-07-12 10:20:59,725 INFO  [ip-172-31-25-3:16000.activeMasterManager] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2016-07-12 10:20:59,753 INFO  [ip-172-31-25-3:16000.activeMasterManager] coordination.SplitLogManagerCoordination: found orphan task WALs%2Fip-172-31-29-145.ap-southeast-2.compute.internal%2C16020%2C1467338848574-splitting%2Fip-172-31-29-145.ap-southeast-2.compute.internal%252C16020%252C1467338848574..meta.1467339405366.meta
2016-07-12 10:20:59,755 INFO  [ip-172-31-25-3:16000.activeMasterManager] coordination.SplitLogManagerCoordination: Found 1 orphan tasks and 0 rescan nodes
2016-07-12 10:20:59,759 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: creating orphan task /hbase-unsecure/splitWAL/WALs%2Fip-172-31-29-145.ap-southeast-2.compute.internal%2C16020%2C1467338848574-splitting%2Fip-172-31-29-145.ap-southeast-2.compute.internal%252C16020%252C1467338848574..meta.1467339405366.meta
2016-07-12 10:20:59,759 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: resubmitting unassigned orphan task /hbase-unsecure/splitWAL/WALs%2Fip-172-31-29-145.ap-southeast-2.compute.internal%2C16020%2C1467338848574-splitting%2Fip-172-31-29-145.ap-southeast-2.compute.internal%252C16020%252C1467338848574..meta.1467339405366.meta
2016-07-12 10:20:59,760 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: resubmitting task /hbase-unsecure/splitWAL/WALs%2Fip-172-31-29-145.ap-southeast-2.compute.internal%2C16020%2C1467338848574-splitting%2Fip-172-31-29-145.ap-southeast-2.compute.internal%252C16020%252C1467338848574..meta.1467339405366.meta
2016-07-12 10:20:59,784 INFO  [main-EventThread] coordination.SplitLogManagerCoordination: task /hbase-unsecure/splitWAL/RESCAN0000061370 entered state: DONE ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815
2016-07-12 10:20:59,999 INFO  [ip-172-31-25-3:16000.activeMasterManager] zookeeper.RecoverableZooKeeper: Process identifier=hconnection-0x45c82203 connecting to ZooKeeper ensemble=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181
2016-07-12 10:20:59,999 INFO  [ip-172-31-25-3:16000.activeMasterManager] zookeeper.ZooKeeper: Initiating client connection, connectString=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181 sessionTimeout=90000 watcher=hconnection-0x45c822030x0, quorum=ip-172-31-21-31.ap-southeast-2.compute.internal:2181,ip-172-31-25-3.ap-southeast-2.compute.internal:2181, baseZNode=/hbase-unsecure
2016-07-12 10:21:00,022 INFO  [ip-172-31-25-3:16000.activeMasterManager-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Opening socket connection to server ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181. Will not attempt to authenticate using SASL (unknown error)
2016-07-12 10:21:00,022 INFO  [ip-172-31-25-3:16000.activeMasterManager-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Socket connection established to ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181, initiating session
2016-07-12 10:21:00,028 INFO  [ip-172-31-25-3:16000.activeMasterManager-SendThread(ip-172-31-21-31.ap-southeast-2.compute.internal:2181)] zookeeper.ClientCnxn: Session establishment complete on server ip-172-31-21-31.ap-southeast-2.compute.internal/172.31.21.31:2181, sessionid = 0x155dc78eeb40002, negotiated timeout = 40000
2016-07-12 10:21:00,057 INFO  [ip-172-31-25-3:16000.activeMasterManager] balancer.StochasticLoadBalancer: loading config
2016-07-12 10:21:00,201 INFO  [ip-172-31-25-3:16000.activeMasterManager] master.HMaster: Server active/primary master=ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815, sessionid=0x155dc78eeb40000, setting cluster-up flag (Was=true)
2016-07-12 10:21:00,237 INFO  [ip-172-31-25-3:16000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase-unsecure/flush-table-proc/acquired /hbase-unsecure/flush-table-proc/reached /hbase-unsecure/flush-table-proc/abort
2016-07-12 10:21:00,250 INFO  [ip-172-31-25-3:16000.activeMasterManager] procedure.ZKProcedureUtil: Clearing all procedure znodes: /hbase-unsecure/online-snapshot/acquired /hbase-unsecure/online-snapshot/reached /hbase-unsecure/online-snapshot/abort
2016-07-12 10:21:00,305 INFO  [ip-172-31-25-3:16000.activeMasterManager] master.MasterCoprocessorHost: System coprocessor loading is enabled
2016-07-12 10:21:00,369 INFO  [ip-172-31-25-3:16000.activeMasterManager] procedure2.ProcedureExecutor: Starting procedure executor threads=5
2016-07-12 10:21:00,370 INFO  [ip-172-31-25-3:16000.activeMasterManager] wal.WALProcedureStore: Starting WAL Procedure Store lease recovery
2016-07-12 10:21:00,378 INFO  [ip-172-31-25-3:16000.activeMasterManager] util.FSHDFSUtils: Recovering lease on dfs file hdfs://ip-172-31-25-3.ap-southeast-2.compute.internal:8020/apps/hbase/data/MasterProcWALs/state-############00000379.log
2016-07-12 10:21:00,402 FATAL [ip-172-31-25-3:16000.activeMasterManager] master.HMaster: Failed to become active master
java.io.FileNotFoundException: File does not exist: /apps/hbase/data/MasterProcWALs/state-############00000379.log
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
    at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:1261)
    at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:280)
    at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:276)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:276)
    at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverLease(FSHDFSUtils.java:282)
    at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:215)
    at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:162)
    at org.apache.hadoop.hbase.master.procedure.MasterProcedureEnv$WALStoreLeaseRecovery.recoverFileLease(MasterProcedureEnv.java:58)
    at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLogs(WALProcedureStore.java:925)
    at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:276)
    at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:430)
    at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1223)
    at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1122)
    at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:709)
    at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:193)
    at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1793)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /apps/hbase/data/MasterProcWALs/state-############00000379.log
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)

    at org.apache.hadoop.ipc.Client.call(Client.java:1426)
    at org.apache.hadoop.ipc.Client.call(Client.java:1363)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
    at com.sun.proxy.$Proxy16.recoverLease(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:605)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
    at com.sun.proxy.$Proxy17.recoverLease(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
    at com.sun.proxy.$Proxy18.recoverLease(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
    at com.sun.proxy.$Proxy18.recoverLease(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:1259)
    ... 17 more
2016-07-12 10:21:00,409 FATAL [ip-172-31-25-3:16000.activeMasterManager] master.HMaster: Master server abort: loaded coprocessors are: []
2016-07-12 10:21:00,409 FATAL [ip-172-31-25-3:16000.activeMasterManager] master.HMaster: Unhandled exception. Starting shutdown.
java.io.FileNotFoundException: File does not exist: /apps/hbase/data/MasterProcWALs/state-############00000379.log
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
    at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
    at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:1261)
    at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:280)
    at org.apache.hadoop.hdfs.DistributedFileSystem$2.doCall(DistributedFileSystem.java:276)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.recoverLease(DistributedFileSystem.java:276)
    at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverLease(FSHDFSUtils.java:282)
    at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:215)
    at org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:162)
    at org.apache.hadoop.hbase.master.procedure.MasterProcedureEnv$WALStoreLeaseRecovery.recoverFileLease(MasterProcedureEnv.java:58)
    at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLogs(WALProcedureStore.java:925)
    at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:276)
    at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.start(ProcedureExecutor.java:430)
    at org.apache.hadoop.hbase.master.HMaster.startProcedureExecutor(HMaster.java:1223)
    at org.apache.hadoop.hbase.master.HMaster.startServiceThreads(HMaster.java:1122)
    at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:709)
    at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:193)
    at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1793)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /apps/hbase/data/MasterProcWALs/state-############00000379.log
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLease(FSNamesystem.java:2877)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.recoverLease(NameNodeRpcServer.java:753)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.recoverLease(ClientNamenodeProtocolServerSideTranslatorPB.java:671)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2206)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2202)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2200)

    at org.apache.hadoop.ipc.Client.call(Client.java:1426)
    at org.apache.hadoop.ipc.Client.call(Client.java:1363)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
    at com.sun.proxy.$Proxy16.recoverLease(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.recoverLease(ClientNamenodeProtocolTranslatorPB.java:605)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
    at com.sun.proxy.$Proxy17.recoverLease(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
    at com.sun.proxy.$Proxy18.recoverLease(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
    at com.sun.proxy.$Proxy18.recoverLease(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.recoverLease(DFSClient.java:1259)
    ... 17 more
2016-07-12 10:21:00,410 INFO  [ip-172-31-25-3:16000.activeMasterManager] regionserver.HRegionServer: STOPPED: Unhandled exception. Starting shutdown.
2016-07-12 10:21:00,410 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] regionserver.HRegionServer: Stopping infoServer
2016-07-12 10:21:00,421 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] mortbay.log: Stopped SelectChannelConnector@0.0.0.0:16010
2016-07-12 10:21:00,535 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] procedure2.ProcedureExecutor: Stopping the procedure executor
2016-07-12 10:21:00,535 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] wal.WALProcedureStore: Stopping the WAL Procedure Store
2016-07-12 10:21:00,536 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] regionserver.HRegionServer: stopping server ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815
2016-07-12 10:21:00,536 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x155dc78eeb40001
2016-07-12 10:21:00,540 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] zookeeper.ZooKeeper: Session: 0x155dc78eeb40001 closed
2016-07-12 10:21:00,540 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-07-12 10:21:00,541 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] regionserver.HRegionServer: stopping server ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815; all regions closed.
2016-07-12 10:21:00,541 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] hbase.ChoreService: Chore service for: ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815 had [] on shutdown
2016-07-12 10:21:00,552 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x155dc78eeb40002
2016-07-12 10:21:00,556 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] zookeeper.ZooKeeper: Session: 0x155dc78eeb40002 closed
2016-07-12 10:21:00,556 INFO  [ip-172-31-25-3:16000.activeMasterManager-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-07-12 10:21:00,557 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] hbase.ChoreService: Chore service for: ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815_splitLogManager_ had [[ScheduledChore: Name: SplitLogManager Timeout Monitor Period: 1000 Unit: MILLISECONDS]] on shutdown
2016-07-12 10:21:00,557 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] flush.MasterFlushTableProcedureManager: stop: server shutting down.
2016-07-12 10:21:00,557 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] ipc.RpcServer: Stopping server on 16000
2016-07-12 10:21:00,557 INFO  [RpcServer.listener,port=16000] ipc.RpcServer: RpcServer.listener,port=16000: stopping
2016-07-12 10:21:00,558 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopped
2016-07-12 10:21:00,560 INFO  [RpcServer.responder] ipc.RpcServer: RpcServer.responder: stopping
2016-07-12 10:21:00,569 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] zookeeper.RecoverableZooKeeper: Node /hbase-unsecure/rs/ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815 already deleted, retry=false
2016-07-12 10:21:00,573 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] zookeeper.ZooKeeper: Session: 0x155dc78eeb40000 closed
2016-07-12 10:21:00,573 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down
2016-07-12 10:21:00,573 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] regionserver.HRegionServer: stopping server ip-172-31-25-3.ap-southeast-2.compute.internal,16000,1468282854815; zookeeper connection closed.
2016-07-12 10:21:00,573 INFO  [master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000] regionserver.HRegionServer: master/ip-172-31-25-3.ap-southeast-2.compute.internal/172.31.25.3:16000 exiting
Please advise.
Many Thanks,
@sjiang
@Ted Yu
Re: HBase server fails to start - File does not exist: MasterProcWALs

sjiang — Tue, 12 Jul 2016 20:46:34 GMT
Could you check whether the log file exist in /apps/hbase/data/MasterProcWALs/state-############00000379.log?
If file exists, then it is likely the same issue as https://issues.apache.org/jira/browse/HBASE-16056 (which will be fixed in the next HDP release).
Re: HBase server fails to start - File does not exist: MasterProcWALs

wonhee_choi — Wed, 13 Jul 2016 07:17:46 GMT
Hi sjiang,
The error message in the jira post seems to be similar to the above.
Yes, the file exists in the path.
Two points from me:
1) We don't have any secondary/backup hbase master in place. Only 1 master and 2 region servers. Does the above jira case still apply to mine?
2) If yes, is there any workaround just to get it started? We don't really care about the data integrity here because whatever data in hbase now, is not what we have created explicitly. Cluster restarting doesn't resolve this. It always fails to start up at the moment.
Thanks!
Re: HBase server fails to start - File does not exist: MasterProcWALs

wonhee_choi — Thu, 14 Jul 2016 10:45:08 GMT
Sorry. I hate to repeat this, but this issue is gone.
Only change I made as per HBase is the below ambari metrics parameter values following Ambari's recommendation.
hbase_master_xmn_size -> 384 hbase_master_heapsize -> 1115
Upon cluster restart, Hbase came up alright.
Re: HBase server fails to start - File does not exist: MasterProcWALs

wonhee_choi — Fri, 15 Jul 2016 10:02:38 GMT
Sorry, I hate to repeat this, but this issue came back again. I am not trying to fool anyone here. Sorry for the misleading clue above.
This time no change has been made but the cluster restart. Same error. Restarting doesn't solve the problem.
This issue is so intermittent so very hard to troubleshoot.