Thu Jul 11 19:11:51 JST 2019 Starting master on kvm07 core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 124478 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 32768 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 65536 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited 2019-07-11 19:11:52,138 INFO [main] master.HMaster: STARTING service HMaster 2019-07-11 19:11:52,140 INFO [main] util.VersionInfo: HBase 2.0.2.3.1.0.0-78 2019-07-11 19:11:52,140 INFO [main] util.VersionInfo: Source code repository git://ctr-e138-1518143905142-586755-01-000023.hwx.site/grid/0/jenkins/workspace/HDP-parallel-centos7/SOURCES/hbase revision= 2019-07-11 19:11:52,140 INFO [main] util.VersionInfo: Compiled by jenkins on Thu Dec 6 12:27:45 UTC 2018 2019-07-11 19:11:52,140 INFO [main] util.VersionInfo: From source with checksum 015c34650c163b249d16fc7e496a030e 2019-07-11 19:11:52,682 INFO [main] master.HMasterCommandLine: Starting a zookeeper cluster 2019-07-11 19:11:52,724 INFO [main] server.ZooKeeperServer: Server environment:zookeeper.version=3.4.6-78--1, built on 12/06/2018 11:34 GMT 2019-07-11 19:11:52,724 INFO [main] server.ZooKeeperServer: Server environment:host.name=kvm07 2019-07-11 19:11:52,724 INFO [main] server.ZooKeeperServer: Server environment:java.version=1.8.0_212 2019-07-11 19:11:52,724 INFO [main] server.ZooKeeperServer: Server environment:java.vendor=Oracle Corporation 2019-07-11 19:11:52,724 INFO [main] server.ZooKeeperServer: Server environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre 2019-07-11 19:11:52,725 INFO [main] server.ZooKeeperServer: Server environment:java.class.path=/etc/ams-hbase/conf:/usr/lib/jvm/java-1.8.0/lib/tools.jar:/usr/lib/ams-hbase/:/usr/lib/ams-hbase//lib/antlr-runtime-3.5.2.jar:/usr/lib/ams-hbase//lib/aopalliance-1.0.jar:/usr/lib/ams-hbase//lib/aopalliance-repackaged-2.5.0-b32.jar:/usr/lib/ams-hbase//lib/audience-annotations-0.5.0.jar:/usr/lib/ams-hbase//lib/avro-1.7.7.jar:/usr/lib/ams-hbase//lib/aws-java-sdk-bundle-1.11.271.jar:/usr/lib/ams-hbase//lib/commons-beanutils-1.9.3.jar:/usr/lib/ams-hbase//lib/commons-cli-1.2.jar:/usr/lib/ams-hbase//lib/commons-codec-1.10.jar:/usr/lib/ams-hbase//lib/commons-collections-3.2.2.jar:/usr/lib/ams-hbase//lib/commons-compress-1.4.1.jar:/usr/lib/ams-hbase//lib/commons-configuration2-2.1.1.jar:/usr/lib/ams-hbase//lib/commons-crypto-1.0.0.jar:/usr/lib/ams-hbase//lib/commons-csv-1.0.jar:/usr/lib/ams-hbase//lib/commons-daemon-1.0.13.jar:/usr/lib/ams-hbase//lib/commons-io-2.5.jar:/usr/lib/ams-hbase//lib/commons-lang-2.6.jar:/usr/lib/ams-hbase//lib/commons-lang3-3.6.jar:/usr/lib/ams-hbase//lib/commons-logging-1.2.jar:/usr/lib/ams-hbase//lib/commons-math3-3.6.1.jar:/usr/lib/ams-hbase//lib/commons-net-3.6.jar:/usr/lib/ams-hbase//lib/curator-client-4.0.0.jar:/usr/lib/ams-hbase//lib/curator-framework-4.0.0.jar:/usr/lib/ams-hbase//lib/curator-recipes-4.0.0.jar:/usr/lib/ams-hbase//lib/disruptor-3.3.6.jar:/usr/lib/ams-hbase//lib/dnsjava-2.1.7.jar:/usr/lib/ams-hbase//lib/ehcache-3.3.1.jar:/usr/lib/ams-hbase//lib/fst-2.50.jar:/usr/lib/ams-hbase//lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/lib/ams-hbase//lib/gson-2.2.4.jar:/usr/lib/ams-hbase//lib/guava-11.0.2.jar:/usr/lib/ams-hbase//lib/guice-4.0.jar:/usr/lib/ams-hbase//lib/guice-servlet-4.0.jar:/usr/lib/ams-hbase//lib/hadoop-annotations-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-auth-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-aws-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-client-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-common-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-distcp-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-hdfs-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-mapreduce-client-app-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-mapreduce-client-common-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-mapreduce-client-core-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-mapreduce-client-hs-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-mapreduce-client-jobclient-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-mapreduce-client-shuffle-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-minicluster-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-api-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-client-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-common-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-registry-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-server-applicationhistoryservice-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-server-common-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-server-nodemanager-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-server-resourcemanager-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-server-timelineservice-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-server-web-proxy-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hamcrest-core-1.3.jar:/usr/lib/ams-hbase//lib/hbase-annotations-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-backup-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-client-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-common-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-endpoint-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-examples-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-external-blockcache-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-hadoop2-compat-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-hadoop-compat-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-http-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-it-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-mapreduce-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-metrics-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-metrics-api-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-procedure-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-protocol-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-protocol-shaded-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-replication-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-resource-bundle-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-rest-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-rsgroup-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-server-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-shaded-client-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-shaded-mapreduce-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-shaded-miscellaneous-2.1.0.jar:/usr/lib/ams-hbase//lib/hbase-shaded-netty-2.1.0.jar:/usr/lib/ams-hbase//lib/hbase-shaded-protobuf-2.1.0.jar:/usr/lib/ams-hbase//lib/hbase-shell-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-spark-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-testing-util-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-thrift-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-zookeeper-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/HikariCP-java7-2.4.12.jar:/usr/lib/ams-hbase//lib/hk2-api-2.5.0-b32.jar:/usr/lib/ams-hbase//lib/hk2-locator-2.5.0-b32.jar:/usr/lib/ams-hbase//lib/hk2-utils-2.5.0-b32.jar:/usr/lib/ams-hbase//lib/htrace-core-3.2.0-incubating.jar:/usr/lib/ams-hbase//lib/htrace-core4-4.2.0-incubating.jar:/usr/lib/ams-hbase//lib/httpclient-4.5.3.jar:/usr/lib/ams-hbase//lib/httpcore-4.4.6.jar:/usr/lib/ams-hbase//lib/jackson-annotations-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-core-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-core-asl-1.9.13.jar:/usr/lib/ams-hbase//lib/jackson-databind-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-jaxrs-1.9.2.jar:/usr/lib/ams-hbase//lib/jackson-jaxrs-base-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-jaxrs-json-provider-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/ams-hbase//lib/jackson-module-jaxb-annotations-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-module-paranamer-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-module-scala_2.11-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-xc-1.9.2.jar:/usr/lib/ams-hbase//lib/jamon-runtime-2.4.1.jar:/usr/lib/ams-hbase//lib/javassist-3.20.0-GA.jar:/usr/lib/ams-hbase//lib/java-util-1.9.0.jar:/usr/lib/ams-hbase//lib/javax.annotation-api-1.2.jar:/usr/lib/ams-hbase//lib/javax.el-3.0.1-b08.jar:/usr/lib/ams-hbase//lib/javax.inject-2.5.0-b32.jar:/usr/lib/ams-hbase//lib/javax.servlet-api-3.1.0.jar:/usr/lib/ams-hbase//lib/javax.servlet.jsp-2.3.2.jar:/usr/lib/ams-hbase//lib/javax.servlet.jsp-api-2.3.1.jar:/usr/lib/ams-hbase//lib/javax.servlet.jsp.jstl-1.2.0.v201105211821.jar:/usr/lib/ams-hbase//lib/javax.servlet.jsp.jstl-1.2.2.jar:/usr/lib/ams-hbase//lib/javax.ws.rs-api-2.0.1.jar:/usr/lib/ams-hbase//lib/jaxb-api-2.2.12.jar:/usr/lib/ams-hbase//lib/jaxb-impl-2.2.3-1.jar:/usr/lib/ams-hbase//lib/jcip-annotations-1.0-1.jar:/usr/lib/ams-hbase//lib/jcodings-1.0.18.jar:/usr/lib/ams-hbase//lib/jersey-client-2.25.1.jar:/usr/lib/ams-hbase//lib/jersey-common-2.25.1.jar:/usr/lib/ams-hbase//lib/jersey-container-servlet-core-2.25.1.jar:/usr/lib/ams-hbase//lib/jersey-guava-2.25.1.jar:/usr/lib/ams-hbase//lib/jersey-media-jaxb-2.25.1.jar:/usr/lib/ams-hbase//lib/jersey-server-2.25.1.jar:/usr/lib/ams-hbase//lib/jettison-1.3.8.jar:/usr/lib/ams-hbase//lib/jetty-http-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-io-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-jmx-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-jsp-9.2.26.v20180806.jar:/usr/lib/ams-hbase//lib/jetty-schemas-3.1.M0.jar:/usr/lib/ams-hbase//lib/jetty-security-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-server-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-servlet-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-util-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-util-ajax-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-webapp-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-xml-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/joni-2.1.11.jar:/usr/lib/ams-hbase//lib/jsch-0.1.54.jar:/usr/lib/ams-hbase//lib/json-io-2.5.1.jar:/usr/lib/ams-hbase//lib/jsr311-api-1.1.1.jar:/usr/lib/ams-hbase//lib/junit-4.12.jar:/usr/lib/ams-hbase//lib/leveldbjni-all-1.8.jar:/usr/lib/ams-hbase//lib/libthrift-0.9.3.jar:/usr/lib/ams-hbase//lib/log4j-1.2.17.jar:/usr/lib/ams-hbase//lib/metrics-core-3.2.1.jar:/usr/lib/ams-hbase//lib/mssql-jdbc-6.2.1.jre7.jar:/usr/lib/ams-hbase//lib/netty-all-4.0.52.Final.jar:/usr/lib/ams-hbase//lib/netty-buffer-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/netty-codec-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/netty-codec-http-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/netty-common-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/netty-handler-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/netty-resolver-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/netty-transport-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/nimbus-jose-jwt-4.41.1.jar:/usr/lib/ams-hbase//lib/objenesis-2.1.jar:/usr/lib/ams-hbase//lib/okhttp-2.7.5.jar:/usr/lib/ams-hbase//lib/okio-1.6.0.jar:/usr/lib/ams-hbase//lib/org.eclipse.jdt.core-3.8.2.v20130121.jar:/usr/lib/ams-hbase//lib/osgi-resource-locator-1.0.1.jar:/usr/lib/ams-hbase//lib/paranamer-2.3.jar:/usr/lib/ams-hbase//lib/phoenix-5.0.0.3.1.0.0-78-server.jar:/usr/lib/ams-hbase//lib/phoenix-core-5.0.0.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/protobuf-java-2.5.0.jar:/usr/lib/ams-hbase//lib/re2j-1.1.jar:/usr/lib/ams-hbase//lib/slf4j-api-1.7.25.jar:/usr/lib/ams-hbase//lib/slf4j-log4j12-1.7.25.jar:/usr/lib/ams-hbase//lib/snappy-java-1.0.5.jar:/usr/lib/ams-hbase//lib/spymemcached-2.12.2.jar:/usr/lib/ams-hbase//lib/stax2-api-3.1.4.jar:/usr/lib/ams-hbase//lib/validation-api-1.1.0.Final.jar:/usr/lib/ams-hbase//lib/woodstox-core-5.0.3.jar:/usr/lib/ams-hbase//lib/xz-1.0.jar:/usr/lib/ams-hbase//lib/zookeeper-3.4.6.3.1.0.0-78.jar: 2019-07-11 19:11:52,728 INFO [main] server.ZooKeeperServer: Server environment:java.library.path=/usr/lib/ams-hbase/lib/hadoop-native/ 2019-07-11 19:11:52,728 INFO [main] server.ZooKeeperServer: Server environment:java.io.tmpdir=/var/lib/ambari-metrics-collector/hbase-tmp 2019-07-11 19:11:52,728 INFO [main] server.ZooKeeperServer: Server environment:java.compiler= 2019-07-11 19:11:52,729 INFO [main] server.ZooKeeperServer: Server environment:os.name=Linux 2019-07-11 19:11:52,729 INFO [main] server.ZooKeeperServer: Server environment:os.arch=amd64 2019-07-11 19:11:52,729 INFO [main] server.ZooKeeperServer: Server environment:os.version=3.10.0-957.12.1.el7.x86_64 2019-07-11 19:11:52,729 INFO [main] server.ZooKeeperServer: Server environment:user.name=ams 2019-07-11 19:11:52,729 INFO [main] server.ZooKeeperServer: Server environment:user.home=/home/ams 2019-07-11 19:11:52,729 INFO [main] server.ZooKeeperServer: Server environment:user.dir=/home/ams 2019-07-11 19:11:52,774 INFO [main] server.ZooKeeperServer: Created server with tickTime 6000 minSessionTimeout 12000 maxSessionTimeout 120000 datadir /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper/zookeeper_0/version-2 snapdir /var/lib/ambari-metrics-collector/hbase-tmp/zookeeper/zookeeper_0/version-2 2019-07-11 19:11:52,774 INFO [main] server.ZooKeeperServer: minSessionTimeout set to -1 2019-07-11 19:11:52,774 INFO [main] server.ZooKeeperServer: maxSessionTimeout set to -1 2019-07-11 19:11:52,819 INFO [main] server.NIOServerCnxnFactory: binding to port 0.0.0.0/0.0.0.0:61181 2019-07-11 19:11:53,376 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /127.0.0.1:60328 2019-07-11 19:11:53,391 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ServerCnxn: The list of known four letter word commands is : [{1936881266=srvr, 1937006964=stat, 2003003491=wchc, 1685417328=dump, 1668445044=crst, 1936880500=srst, 1701738089=envi, 1668247142=conf, 2003003507=wchs, 2003003504=wchp, 1668247155=cons, 1835955314=mntr, 1769173615=isro, 1920298859=ruok, 1735683435=gtmk, 1937010027=stmk}] 2019-07-11 19:11:53,391 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ServerCnxn: The list of enabled four letter word commands is : [[wchs, stat, wchp, stmk, conf, ruok, mntr, srvr, wchc, envi, srst, isro, dump, gtmk, crst, cons]] 2019-07-11 19:11:53,392 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Processing stat command from /127.0.0.1:60328 2019-07-11 19:11:53,395 INFO [Thread-2] server.NIOServerCnxn: Stat command output 2019-07-11 19:11:53,396 INFO [Thread-2] server.NIOServerCnxn: Closed socket connection for client /127.0.0.1:60328 (no session established for client) 2019-07-11 19:11:53,397 INFO [main] zookeeper.MiniZooKeeperCluster: Started MiniZooKeeperCluster and ran successful 'stat' on client port=61181 2019-07-11 19:11:53,397 INFO [main] master.HMasterCommandLine: Starting up instance of localHBaseCluster; master=1, regionserversCount=1 2019-07-11 19:11:53,605 WARN [main] util.MemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2019-07-11 19:11:53,887 INFO [main] impl.MetricsConfig: Loaded properties from hadoop-metrics2-hbase.properties 2019-07-11 19:11:54,364 INFO [main] timeline.HadoopTimelineMetricsSink: Initializing Timeline metrics sink. 2019-07-11 19:11:54,371 INFO [main] timeline.HadoopTimelineMetricsSink: Identified hostname = kvm07.hadoop.local, serviceName = ams-hbase 2019-07-11 19:11:54,404 INFO [main] timeline.HadoopTimelineMetricsSink: No suitable collector found. 2019-07-11 19:11:54,427 INFO [main] impl.MetricsSinkAdapter: Sink timeline started 2019-07-11 19:11:54,442 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37768 2019-07-11 19:11:54,448 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37768 2019-07-11 19:11:54,452 INFO [SyncThread:0] persistence.FileTxnLog: Creating new log file: log.ea3 2019-07-11 19:11:54,463 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a30000 with negotiated timeout 120000 for client /10.3.0.25:37768 2019-07-11 19:11:54,533 INFO [main] impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 2019-07-11 19:11:54,533 INFO [main] impl.MetricsSystemImpl: HBase metrics system started 2019-07-11 19:11:54,575 INFO [main] metrics.MetricRegistries: Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2019-07-11 19:11:55,242 INFO [main] regionserver.RSRpcServices: master/kvm07:61300 server-side Connection retries=45 2019-07-11 19:11:55,320 INFO [main] ipc.RpcExecutor: Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=3, maxQueueLength=300, handlerCount=30 2019-07-11 19:11:55,325 INFO [main] ipc.RpcExecutor: Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=300, handlerCount=20 2019-07-11 19:11:55,325 INFO [main] ipc.RpcExecutor: Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=300, handlerCount=3 2019-07-11 19:11:55,607 INFO [main] ipc.RpcServerFactory: Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientService, hbase.pb.AdminService 2019-07-11 19:11:56,212 INFO [main] ipc.NettyRpcServer: Bind to /10.3.0.25:61300 2019-07-11 19:11:56,357 WARN [main] util.MemorySizeUtil: hbase.regionserver.global.memstore.lowerLimit is deprecated. Instead use hbase.regionserver.global.memstore.size.lower.limit 2019-07-11 19:11:56,365 INFO [main] hfile.CacheConfig: Created cacheConfig: CacheConfig:disabled 2019-07-11 19:11:56,366 INFO [main] hfile.CacheConfig: Created cacheConfig: CacheConfig:disabled 2019-07-11 19:11:56,680 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=master:61300 connecting to ZooKeeper ensemble=kvm07.hadoop.local:61181 2019-07-11 19:11:56,688 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-78--1, built on 12/06/2018 11:34 GMT 2019-07-11 19:11:56,688 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=kvm07 2019-07-11 19:11:56,688 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_212 2019-07-11 19:11:56,688 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2019-07-11 19:11:56,688 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre 2019-07-11 19:11:56,688 INFO [main] zookeeper.ZooKeeper: Client environment:java.class.path=/etc/ams-hbase/conf:/usr/lib/jvm/java-1.8.0/lib/tools.jar:/usr/lib/ams-hbase/:/usr/lib/ams-hbase//lib/antlr-runtime-3.5.2.jar:/usr/lib/ams-hbase//lib/aopalliance-1.0.jar:/usr/lib/ams-hbase//lib/aopalliance-repackaged-2.5.0-b32.jar:/usr/lib/ams-hbase//lib/audience-annotations-0.5.0.jar:/usr/lib/ams-hbase//lib/avro-1.7.7.jar:/usr/lib/ams-hbase//lib/aws-java-sdk-bundle-1.11.271.jar:/usr/lib/ams-hbase//lib/commons-beanutils-1.9.3.jar:/usr/lib/ams-hbase//lib/commons-cli-1.2.jar:/usr/lib/ams-hbase//lib/commons-codec-1.10.jar:/usr/lib/ams-hbase//lib/commons-collections-3.2.2.jar:/usr/lib/ams-hbase//lib/commons-compress-1.4.1.jar:/usr/lib/ams-hbase//lib/commons-configuration2-2.1.1.jar:/usr/lib/ams-hbase//lib/commons-crypto-1.0.0.jar:/usr/lib/ams-hbase//lib/commons-csv-1.0.jar:/usr/lib/ams-hbase//lib/commons-daemon-1.0.13.jar:/usr/lib/ams-hbase//lib/commons-io-2.5.jar:/usr/lib/ams-hbase//lib/commons-lang-2.6.jar:/usr/lib/ams-hbase//lib/commons-lang3-3.6.jar:/usr/lib/ams-hbase//lib/commons-logging-1.2.jar:/usr/lib/ams-hbase//lib/commons-math3-3.6.1.jar:/usr/lib/ams-hbase//lib/commons-net-3.6.jar:/usr/lib/ams-hbase//lib/curator-client-4.0.0.jar:/usr/lib/ams-hbase//lib/curator-framework-4.0.0.jar:/usr/lib/ams-hbase//lib/curator-recipes-4.0.0.jar:/usr/lib/ams-hbase//lib/disruptor-3.3.6.jar:/usr/lib/ams-hbase//lib/dnsjava-2.1.7.jar:/usr/lib/ams-hbase//lib/ehcache-3.3.1.jar:/usr/lib/ams-hbase//lib/fst-2.50.jar:/usr/lib/ams-hbase//lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/lib/ams-hbase//lib/gson-2.2.4.jar:/usr/lib/ams-hbase//lib/guava-11.0.2.jar:/usr/lib/ams-hbase//lib/guice-4.0.jar:/usr/lib/ams-hbase//lib/guice-servlet-4.0.jar:/usr/lib/ams-hbase//lib/hadoop-annotations-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-auth-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-aws-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-client-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-common-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-distcp-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-hdfs-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-hdfs-client-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-mapreduce-client-app-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-mapreduce-client-common-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-mapreduce-client-core-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-mapreduce-client-hs-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-mapreduce-client-jobclient-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-mapreduce-client-shuffle-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-minicluster-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-api-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-client-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-common-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-registry-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-server-applicationhistoryservice-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-server-common-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-server-nodemanager-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-server-resourcemanager-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-server-timelineservice-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hadoop-yarn-server-web-proxy-3.1.1.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hamcrest-core-1.3.jar:/usr/lib/ams-hbase//lib/hbase-annotations-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-backup-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-client-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-common-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-endpoint-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-examples-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-external-blockcache-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-hadoop2-compat-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-hadoop-compat-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-http-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-it-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-mapreduce-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-metrics-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-metrics-api-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-procedure-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-protocol-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-protocol-shaded-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-replication-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-resource-bundle-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-rest-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-rsgroup-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-server-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-shaded-client-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-shaded-mapreduce-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-shaded-miscellaneous-2.1.0.jar:/usr/lib/ams-hbase//lib/hbase-shaded-netty-2.1.0.jar:/usr/lib/ams-hbase//lib/hbase-shaded-protobuf-2.1.0.jar:/usr/lib/ams-hbase//lib/hbase-shell-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-spark-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-testing-util-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-thrift-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/hbase-zookeeper-2.0.2.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/HikariCP-java7-2.4.12.jar:/usr/lib/ams-hbase//lib/hk2-api-2.5.0-b32.jar:/usr/lib/ams-hbase//lib/hk2-locator-2.5.0-b32.jar:/usr/lib/ams-hbase//lib/hk2-utils-2.5.0-b32.jar:/usr/lib/ams-hbase//lib/htrace-core-3.2.0-incubating.jar:/usr/lib/ams-hbase//lib/htrace-core4-4.2.0-incubating.jar:/usr/lib/ams-hbase//lib/httpclient-4.5.3.jar:/usr/lib/ams-hbase//lib/httpcore-4.4.6.jar:/usr/lib/ams-hbase//lib/jackson-annotations-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-core-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-core-asl-1.9.13.jar:/usr/lib/ams-hbase//lib/jackson-databind-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-jaxrs-1.9.2.jar:/usr/lib/ams-hbase//lib/jackson-jaxrs-base-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-jaxrs-json-provider-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/ams-hbase//lib/jackson-module-jaxb-annotations-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-module-paranamer-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-module-scala_2.11-2.9.5.jar:/usr/lib/ams-hbase//lib/jackson-xc-1.9.2.jar:/usr/lib/ams-hbase//lib/jamon-runtime-2.4.1.jar:/usr/lib/ams-hbase//lib/javassist-3.20.0-GA.jar:/usr/lib/ams-hbase//lib/java-util-1.9.0.jar:/usr/lib/ams-hbase//lib/javax.annotation-api-1.2.jar:/usr/lib/ams-hbase//lib/javax.el-3.0.1-b08.jar:/usr/lib/ams-hbase//lib/javax.inject-2.5.0-b32.jar:/usr/lib/ams-hbase//lib/javax.servlet-api-3.1.0.jar:/usr/lib/ams-hbase//lib/javax.servlet.jsp-2.3.2.jar:/usr/lib/ams-hbase//lib/javax.servlet.jsp-api-2.3.1.jar:/usr/lib/ams-hbase//lib/javax.servlet.jsp.jstl-1.2.0.v201105211821.jar:/usr/lib/ams-hbase//lib/javax.servlet.jsp.jstl-1.2.2.jar:/usr/lib/ams-hbase//lib/javax.ws.rs-api-2.0.1.jar:/usr/lib/ams-hbase//lib/jaxb-api-2.2.12.jar:/usr/lib/ams-hbase//lib/jaxb-impl-2.2.3-1.jar:/usr/lib/ams-hbase//lib/jcip-annotations-1.0-1.jar:/usr/lib/ams-hbase//lib/jcodings-1.0.18.jar:/usr/lib/ams-hbase//lib/jersey-client-2.25.1.jar:/usr/lib/ams-hbase//lib/jersey-common-2.25.1.jar:/usr/lib/ams-hbase//lib/jersey-container-servlet-core-2.25.1.jar:/usr/lib/ams-hbase//lib/jersey-guava-2.25.1.jar:/usr/lib/ams-hbase//lib/jersey-media-jaxb-2.25.1.jar:/usr/lib/ams-hbase//lib/jersey-server-2.25.1.jar:/usr/lib/ams-hbase//lib/jettison-1.3.8.jar:/usr/lib/ams-hbase//lib/jetty-http-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-io-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-jmx-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-jsp-9.2.26.v20180806.jar:/usr/lib/ams-hbase//lib/jetty-schemas-3.1.M0.jar:/usr/lib/ams-hbase//lib/jetty-security-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-server-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-servlet-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-util-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-util-ajax-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-webapp-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/jetty-xml-9.3.25.v20180904.jar:/usr/lib/ams-hbase//lib/joni-2.1.11.jar:/usr/lib/ams-hbase//lib/jsch-0.1.54.jar:/usr/lib/ams-hbase//lib/json-io-2.5.1.jar:/usr/lib/ams-hbase//lib/jsr311-api-1.1.1.jar:/usr/lib/ams-hbase//lib/junit-4.12.jar:/usr/lib/ams-hbase//lib/leveldbjni-all-1.8.jar:/usr/lib/ams-hbase//lib/libthrift-0.9.3.jar:/usr/lib/ams-hbase//lib/log4j-1.2.17.jar:/usr/lib/ams-hbase//lib/metrics-core-3.2.1.jar:/usr/lib/ams-hbase//lib/mssql-jdbc-6.2.1.jre7.jar:/usr/lib/ams-hbase//lib/netty-all-4.0.52.Final.jar:/usr/lib/ams-hbase//lib/netty-buffer-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/netty-codec-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/netty-codec-http-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/netty-common-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/netty-handler-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/netty-resolver-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/netty-transport-4.1.17.Final.jar:/usr/lib/ams-hbase//lib/nimbus-jose-jwt-4.41.1.jar:/usr/lib/ams-hbase//lib/objenesis-2.1.jar:/usr/lib/ams-hbase//lib/okhttp-2.7.5.jar:/usr/lib/ams-hbase//lib/okio-1.6.0.jar:/usr/lib/ams-hbase//lib/org.eclipse.jdt.core-3.8.2.v20130121.jar:/usr/lib/ams-hbase//lib/osgi-resource-locator-1.0.1.jar:/usr/lib/ams-hbase//lib/paranamer-2.3.jar:/usr/lib/ams-hbase//lib/phoenix-5.0.0.3.1.0.0-78-server.jar:/usr/lib/ams-hbase//lib/phoenix-core-5.0.0.3.1.0.0-78.jar:/usr/lib/ams-hbase//lib/protobuf-java-2.5.0.jar:/usr/lib/ams-hbase//lib/re2j-1.1.jar:/usr/lib/ams-hbase//lib/slf4j-api-1.7.25.jar:/usr/lib/ams-hbase//lib/slf4j-log4j12-1.7.25.jar:/usr/lib/ams-hbase//lib/snappy-java-1.0.5.jar:/usr/lib/ams-hbase//lib/spymemcached-2.12.2.jar:/usr/lib/ams-hbase//lib/stax2-api-3.1.4.jar:/usr/lib/ams-hbase//lib/validation-api-1.1.0.Final.jar:/usr/lib/ams-hbase//lib/woodstox-core-5.0.3.jar:/usr/lib/ams-hbase//lib/xz-1.0.jar:/usr/lib/ams-hbase//lib/zookeeper-3.4.6.3.1.0.0-78.jar: 2019-07-11 19:11:56,689 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/lib/ams-hbase/lib/hadoop-native/ 2019-07-11 19:11:56,689 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/var/lib/ambari-metrics-collector/hbase-tmp 2019-07-11 19:11:56,689 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler= 2019-07-11 19:11:56,689 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux 2019-07-11 19:11:56,689 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64 2019-07-11 19:11:56,689 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-957.12.1.el7.x86_64 2019-07-11 19:11:56,689 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=ams 2019-07-11 19:11:56,689 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/ams 2019-07-11 19:11:56,689 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/ams 2019-07-11 19:11:56,691 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=kvm07.hadoop.local:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@7fab4be7 2019-07-11 19:11:56,729 INFO [main-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Opening socket connection to server kvm07/10.3.0.25:61181. Will not attempt to authenticate using SASL (unknown error) 2019-07-11 19:11:56,735 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37770 2019-07-11 19:11:56,738 INFO [main-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.3.0.25:37770, server: kvm07/10.3.0.25:61181 2019-07-11 19:11:56,740 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37770 2019-07-11 19:11:56,749 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a30001 with negotiated timeout 120000 for client /10.3.0.25:37770 2019-07-11 19:11:56,750 INFO [main-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Session establishment complete on server kvm07/10.3.0.25:61181, sessionid = 0x16be08492a30001, negotiated timeout = 120000 2019-07-11 19:11:56,766 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x16be08492a30001 type:create cxid:0x1 zxid:0xea5 txntype:-1 reqpath:n/a Error Path:/ams-hbase-unsecure Error:KeeperErrorCode = NodeExists for /ams-hbase-unsecure 2019-07-11 19:11:56,993 INFO [main] util.log: Logging initialized @5587ms 2019-07-11 19:11:57,157 INFO [main] http.HttpRequestLog: Http request log for http.requests.master is not defined 2019-07-11 19:11:57,209 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2019-07-11 19:11:57,211 INFO [main] http.HttpServer: Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2019-07-11 19:11:57,218 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context master 2019-07-11 19:11:57,218 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2019-07-11 19:11:57,219 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2019-07-11 19:11:57,290 INFO [main] http.HttpServer: Jetty bound to port 61310 2019-07-11 19:11:57,294 INFO [main] server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-05T06:11:46+09:00, git hash: 3ce520221d0240229c862b122d2b06c12a625732 2019-07-11 19:11:57,400 INFO [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@b8e246c{/logs,file:///var/log/ambari-metrics-collector/,AVAILABLE} 2019-07-11 19:11:57,401 INFO [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@7cb2651f{/static,file:///usr/lib/ams-hbase/hbase-webapps/static/,AVAILABLE} 2019-07-11 19:11:57,628 INFO [main] handler.ContextHandler: Started o.e.j.w.WebAppContext@288cdaab{/,file:///usr/lib/ams-hbase/hbase-webapps/master/,AVAILABLE}{file:/usr/lib/ams-hbase/hbase-webapps/master} 2019-07-11 19:11:57,639 INFO [main] server.AbstractConnector: Started ServerConnector@5e5073ab{HTTP/1.1,[http/1.1]}{0.0.0.0:61310} 2019-07-11 19:11:57,639 INFO [main] server.Server: Started @6233ms 2019-07-11 19:11:57,643 INFO [main] master.HMaster: hbase.rootdir=file:/var/lib/ambari-metrics-collector/hbase, hbase.cluster.distributed=false 2019-07-11 19:11:57,727 WARN [main] util.MemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2019-07-11 19:11:57,734 INFO [main] regionserver.RSRpcServices: regionserver/kvm07:61320 server-side Connection retries=45 2019-07-11 19:11:57,735 INFO [main] ipc.RpcExecutor: Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=3, maxQueueLength=300, handlerCount=30 2019-07-11 19:11:57,735 INFO [main] ipc.RpcExecutor: Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=300, handlerCount=20 2019-07-11 19:11:57,735 INFO [main] ipc.RpcExecutor: Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=300, handlerCount=3 2019-07-11 19:11:57,742 INFO [main] ipc.RpcServerFactory: Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2019-07-11 19:11:57,747 INFO [main] io.ByteBufferPool: Created with bufferSize=64 KB and maxPoolSize=1.88 KB 2019-07-11 19:11:57,782 INFO [main] ipc.NettyRpcServer: Bind to /10.3.0.25:61320 2019-07-11 19:11:57,783 WARN [main] util.MemorySizeUtil: hbase.regionserver.global.memstore.lowerLimit is deprecated. Instead use hbase.regionserver.global.memstore.size.lower.limit 2019-07-11 19:11:57,783 INFO [main] hfile.CacheConfig: Allocating onheap LruBlockCache size=622.09 MB, blockSize=64 KB 2019-07-11 19:11:57,809 INFO [main] hfile.CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=467.58 KB, freeSize=621.63 MB, maxSize=622.09 MB, heapSize=467.58 KB, minSize=590.98 MB, minFactor=0.95, multiSize=295.49 MB, multiFactor=0.5, singleSize=147.75 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2019-07-11 19:11:57,811 INFO [main] hfile.CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=467.58 KB, freeSize=621.63 MB, maxSize=622.09 MB, heapSize=467.58 KB, minSize=590.98 MB, minFactor=0.95, multiSize=295.49 MB, multiFactor=0.5, singleSize=147.75 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2019-07-11 19:11:57,816 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=regionserver:61320 connecting to ZooKeeper ensemble=kvm07.hadoop.local:61181 2019-07-11 19:11:57,816 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=kvm07.hadoop.local:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@2324bfe7 2019-07-11 19:11:57,824 INFO [main-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Opening socket connection to server kvm07/10.3.0.25:61181. Will not attempt to authenticate using SASL (unknown error) 2019-07-11 19:11:57,826 INFO [main-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.3.0.25:37774, server: kvm07/10.3.0.25:61181 2019-07-11 19:11:57,829 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37774 2019-07-11 19:11:57,829 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37774 2019-07-11 19:11:57,849 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a30002 with negotiated timeout 120000 for client /10.3.0.25:37774 2019-07-11 19:11:57,856 INFO [main-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Session establishment complete on server kvm07/10.3.0.25:61181, sessionid = 0x16be08492a30002, negotiated timeout = 120000 2019-07-11 19:11:57,929 INFO [main] http.HttpRequestLog: Http request log for http.requests.regionserver is not defined 2019-07-11 19:11:57,932 INFO [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter) 2019-07-11 19:11:57,932 INFO [main] http.HttpServer: Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter) 2019-07-11 19:11:57,936 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver 2019-07-11 19:11:57,937 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2019-07-11 19:11:57,937 INFO [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2019-07-11 19:11:57,961 INFO [main] http.HttpServer: Jetty bound to port 61330 2019-07-11 19:11:57,962 INFO [main] server.Server: jetty-9.3.25.v20180904, build timestamp: 2018-09-05T06:11:46+09:00, git hash: 3ce520221d0240229c862b122d2b06c12a625732 2019-07-11 19:11:57,977 INFO [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@46cb98a3{/logs,file:///var/log/ambari-metrics-collector/,AVAILABLE} 2019-07-11 19:11:57,978 INFO [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@4da9f723{/static,file:///usr/lib/ams-hbase/hbase-webapps/static/,AVAILABLE} 2019-07-11 19:11:58,073 INFO [main] handler.ContextHandler: Started o.e.j.w.WebAppContext@96a75da{/,file:///usr/lib/ams-hbase/hbase-webapps/regionserver/,AVAILABLE}{file:/usr/lib/ams-hbase/hbase-webapps/regionserver} 2019-07-11 19:11:58,074 INFO [main] server.AbstractConnector: Started ServerConnector@61e7bf2f{HTTP/1.1,[http/1.1]}{0.0.0.0:61330} 2019-07-11 19:11:58,075 INFO [main] server.Server: Started @6668ms 2019-07-11 19:11:58,094 INFO [Thread-23] master.HMaster: Adding backup master ZNode /ams-hbase-unsecure/backup-masters/kvm07,61300,1562839913433 2019-07-11 19:11:58,193 INFO [M:0;kvm07:61300] regionserver.HRegionServer: ClusterId : 02d1c8e7-2295-4458-924e-84016b4d551a 2019-07-11 19:11:58,246 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x16be08492a30001 type:create cxid:0x11 zxid:0xea8 txntype:-1 reqpath:n/a Error Path:/ams-hbase-unsecure/master Error:KeeperErrorCode = NodeExists for /ams-hbase-unsecure/master 2019-07-11 19:11:58,264 INFO [Thread-23] master.ActiveMasterManager: Current master has this master's address, kvm07,61300,1562826670695; master was restarted? Deleting node. 2019-07-11 19:11:58,275 INFO [Thread-23] master.ActiveMasterManager: Deleting ZNode for /ams-hbase-unsecure/backup-masters/kvm07,61300,1562839913433 from backup master directory 2019-07-11 19:11:58,282 INFO [Thread-23] master.ActiveMasterManager: Registered as active master=kvm07,61300,1562839913433 2019-07-11 19:11:58,290 INFO [Thread-23] regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk size 2 MB, max count 326, initial count 0 2019-07-11 19:11:58,293 INFO [Thread-23] regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 362, initial count 0 2019-07-11 19:11:58,501 INFO [Thread-23] coordination.SplitLogManagerCoordination: Found 0 orphan tasks and 0 rescan nodes 2019-07-11 19:11:58,598 INFO [Thread-23] zookeeper.ReadOnlyZKClient: Connect 0x70918201 to kvm07.hadoop.local:61181 with session timeout=120000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2019-07-11 19:11:58,613 INFO [ReadOnlyZKClient-kvm07.hadoop.local:61181@0x70918201] zookeeper.ZooKeeper: Initiating client connection, connectString=kvm07.hadoop.local:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$55/461882570@5e8549de 2019-07-11 19:11:58,624 INFO [ReadOnlyZKClient-kvm07.hadoop.local:61181@0x70918201-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Opening socket connection to server kvm07/10.3.0.25:61181. Will not attempt to authenticate using SASL (unknown error) 2019-07-11 19:11:58,625 INFO [ReadOnlyZKClient-kvm07.hadoop.local:61181@0x70918201-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.3.0.25:37778, server: kvm07/10.3.0.25:61181 2019-07-11 19:11:58,626 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37778 2019-07-11 19:11:58,626 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37778 2019-07-11 19:11:58,632 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a30003 with negotiated timeout 120000 for client /10.3.0.25:37778 2019-07-11 19:11:58,634 INFO [ReadOnlyZKClient-kvm07.hadoop.local:61181@0x70918201-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Session establishment complete on server kvm07/10.3.0.25:61181, sessionid = 0x16be08492a30003, negotiated timeout = 120000 2019-07-11 19:11:58,723 INFO [Thread-23] procedure2.ProcedureExecutor: Starting 16 core workers (bigger of cpus/4 or 16) with max (burst) worker count=160 2019-07-11 19:11:58,731 INFO [Thread-23] wal.ProcedureWALFile: Opening file:/var/lib/ambari-metrics-collector/hbase/MasterProcWALs/pv2-00000000000000000001.log length=45336 2019-07-11 19:11:58,743 WARN [Thread-23] wal.WALProcedureStore: Unable to read tracker for file:/var/lib/ambari-metrics-collector/hbase/MasterProcWALs/pv2-00000000000000000001.log org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat$InvalidWALDataException: Invalid Trailer version. got 48 expected 1 at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.readTrailer(ProcedureWALFormat.java:189) at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFile.readTrailer(ProcedureWALFile.java:93) at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFile.readTracker(ProcedureWALFile.java:100) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLog(WALProcedureStore.java:1386) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLogs(WALProcedureStore.java:1335) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:416) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:714) at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1398) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:857) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2225) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:568) at java.lang.Thread.run(Thread.java:748) 2019-07-11 19:11:58,746 INFO [Thread-23] wal.ProcedureWALFile: Closed file:/var/lib/ambari-metrics-collector/hbase/MasterProcWALs/pv2-00000000000000000001.log length=45336 2019-07-11 19:11:58,790 INFO [Thread-23] wal.WALProcedureStore: Rolled new Procedure Store WAL, id=2 2019-07-11 19:11:58,792 INFO [Thread-23] procedure2.ProcedureExecutor: Recovered WALProcedureStore lease in 64msec 2019-07-11 19:11:58,812 INFO [Thread-23] wal.ProcedureWALFile: Opening file:/var/lib/ambari-metrics-collector/hbase/MasterProcWALs/pv2-00000000000000000001.log length=45336 2019-07-11 19:11:58,812 INFO [Thread-23] wal.ProcedureWALFormatReader: Rebuilding tracker for file:/var/lib/ambari-metrics-collector/hbase/MasterProcWALs/pv2-00000000000000000001.log 2019-07-11 19:11:58,869 WARN [Thread-23] wal.ProcedureWALFormatReader: Nothing left to decode. Exiting with missing EOF, log=file:/var/lib/ambari-metrics-collector/hbase/MasterProcWALs/pv2-00000000000000000001.log 2019-07-11 19:11:58,869 INFO [Thread-23] wal.ProcedureWALFormatReader: Read 182 entries in file:/var/lib/ambari-metrics-collector/hbase/MasterProcWALs/pv2-00000000000000000001.log 2019-07-11 19:11:58,872 INFO [Thread-23] wal.ProcedureWALFile: Closed file:/var/lib/ambari-metrics-collector/hbase/MasterProcWALs/pv2-00000000000000000001.log length=45336 2019-07-11 19:11:59,021 INFO [Thread-23] procedure2.ProcedureExecutor: Loaded WALProcedureStore in 228msec 2019-07-11 19:11:59,021 INFO [Thread-23] procedure2.RemoteProcedureDispatcher: Instantiated, coreThreads=128 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2019-07-11 19:11:59,427 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562760622135 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,427 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562752572671 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,427 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562749382700 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,427 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562746533577 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,427 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562748839530 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,428 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562753114618 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,428 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562816660840 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,428 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562748298796 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,428 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562747076054 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,428 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562750005281 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,429 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562751906066 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,429 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562757470863 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,429 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562811505670 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,429 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562761983211 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,429 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562747752975 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,430 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562756459304 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,430 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562755917696 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,430 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562753745187 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,430 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562755368961 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,430 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562751193583 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,431 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562750551634 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,431 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562751758883 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,431 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562825936305 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,431 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562745981179 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,431 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562754831633 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,469 INFO [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562826676313 but NOT online 2019-07-11 19:11:59,470 INFO [Thread-23] master.ServerManager: Processing expiration of kvm07,61320,1562826676313 on kvm07,61300,1562839913433 2019-07-11 19:11:59,636 WARN [Thread-23] master.ServerManager: Expiration called on kvm07,61320,1562754291388 but crash processing in progress, serverStateNode=null 2019-07-11 19:11:59,692 INFO [Thread-23] balancer.BaseLoadBalancer: slop=0.001, systemTablesOnMaster=false 2019-07-11 19:11:59,715 INFO [Thread-23] balancer.StochasticLoadBalancer: Loaded config; maxSteps=1000000, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, etc. 2019-07-11 19:11:59,767 INFO [Thread-23] master.HMaster: Active/primary master=kvm07,61300,1562839913433, sessionid=0x16be08492a30001, setting cluster-up flag (Was=true) 2019-07-11 19:11:59,787 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x16be08492a30001 type:create cxid:0x2b zxid:0xeae txntype:-1 reqpath:n/a Error Path:/ams-hbase-unsecure/flush-table-proc/acquired Error:KeeperErrorCode = NodeExists for /ams-hbase-unsecure/flush-table-proc/acquired 2019-07-11 19:11:59,805 INFO [Thread-23] snapshot.SnapshotManager: Snapshot feature is not enabled, missing log and hfile cleaners. 2019-07-11 19:11:59,808 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x16be08492a30001 type:create cxid:0x31 zxid:0xeaf txntype:-1 reqpath:n/a Error Path:/ams-hbase-unsecure/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for /ams-hbase-unsecure/online-snapshot/acquired 2019-07-11 19:11:59,914 INFO [RS:0;kvm07:61320] regionserver.HRegionServer: ClusterId : 02d1c8e7-2295-4458-924e-84016b4d551a 2019-07-11 19:11:59,958 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x16be08492a30002 type:create cxid:0x9 zxid:0xeb0 txntype:-1 reqpath:n/a Error Path:/ams-hbase-unsecure/flush-table-proc/acquired Error:KeeperErrorCode = NodeExists for /ams-hbase-unsecure/flush-table-proc/acquired 2019-07-11 19:11:59,966 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Got user-level KeeperException when processing sessionid:0x16be08492a30002 type:create cxid:0xc zxid:0xeb1 txntype:-1 reqpath:n/a Error Path:/ams-hbase-unsecure/online-snapshot/acquired Error:KeeperErrorCode = NodeExists for /ams-hbase-unsecure/online-snapshot/acquired 2019-07-11 19:11:59,987 INFO [Thread-23] master.HMaster: hbase:meta {1588230740 state=OPEN, ts=1562839919335, server=kvm07,61320,1562251031381} 2019-07-11 19:11:59,977 INFO [RS:0;kvm07:61320] zookeeper.ReadOnlyZKClient: Connect 0x0f554089 to kvm07.hadoop.local:61181 with session timeout=120000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2019-07-11 19:11:59,995 INFO [ReadOnlyZKClient-kvm07.hadoop.local:61181@0x0f554089] zookeeper.ZooKeeper: Initiating client connection, connectString=kvm07.hadoop.local:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$55/461882570@5e8549de 2019-07-11 19:12:00,003 INFO [ReadOnlyZKClient-kvm07.hadoop.local:61181@0x0f554089-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Opening socket connection to server kvm07/10.3.0.25:61181. Will not attempt to authenticate using SASL (unknown error) 2019-07-11 19:12:00,008 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37782 2019-07-11 19:12:00,008 INFO [ReadOnlyZKClient-kvm07.hadoop.local:61181@0x0f554089-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.3.0.25:37782, server: kvm07/10.3.0.25:61181 2019-07-11 19:12:00,013 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37782 2019-07-11 19:12:00,023 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a30004 with negotiated timeout 120000 for client /10.3.0.25:37782 2019-07-11 19:12:00,025 INFO [ReadOnlyZKClient-kvm07.hadoop.local:61181@0x0f554089-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Session establishment complete on server kvm07/10.3.0.25:61181, sessionid = 0x16be08492a30004, negotiated timeout = 120000 2019-07-11 19:12:00,045 INFO [Thread-23] procedure2.TimeoutExecutorThread: ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.ProcedureExecutor$CompletedProcedureCleaner; timeout=30000, timestamp=1562839950044 2019-07-11 19:12:00,067 INFO [Thread-23] zookeeper.RecoverableZooKeeper: Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=kvm07.hadoop.local:61181 2019-07-11 19:12:00,076 INFO [Thread-23] zookeeper.ZooKeeper: Initiating client connection, connectString=kvm07.hadoop.local:61181 sessionTimeout=120000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@1b9a7485 2019-07-11 19:12:00,082 INFO [Thread-23-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Opening socket connection to server kvm07/10.3.0.25:61181. Will not attempt to authenticate using SASL (unknown error) 2019-07-11 19:12:00,084 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37784 2019-07-11 19:12:00,092 INFO [Thread-23-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Socket connection established, initiating session, client: /10.3.0.25:37784, server: kvm07/10.3.0.25:61181 2019-07-11 19:12:00,094 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37784 2019-07-11 19:12:00,098 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a30005 with negotiated timeout 120000 for client /10.3.0.25:37784 2019-07-11 19:12:00,098 INFO [Thread-23-SendThread(kvm07:61181)] zookeeper.ClientCnxn: Session establishment complete on server kvm07/10.3.0.25:61181, sessionid = 0x16be08492a30005, negotiated timeout = 120000 2019-07-11 19:12:00,106 INFO [Thread-23] cleaner.CleanerChore: Cleaner pool size is 2 2019-07-11 19:12:00,106 INFO [Thread-23] cleaner.LogCleaner: Creating OldWALs cleaners with size=2 2019-07-11 19:12:00,193 INFO [RS:0;kvm07:61320] regionserver.RegionServerCoprocessorHost: System coprocessor loading is enabled 2019-07-11 19:12:00,193 INFO [RS:0;kvm07:61320] regionserver.RegionServerCoprocessorHost: Table coprocessor loading is enabled 2019-07-11 19:12:00,197 INFO [Thread-23] master.HMaster: Master startup: status=Wait for region servers to report in, state=RUNNING, startTime=1562839918240, completionTime=-1 2019-07-11 19:12:00,197 INFO [Thread-23] master.ServerManager: Waiting on regionserver count=0; waited=0ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=0ms 2019-07-11 19:12:00,203 INFO [RS:0;kvm07:61320] regionserver.HRegionServer: reportForDuty to master=kvm07,61300,1562839913433 with port=61320, startcode=1562839917726 2019-07-11 19:12:00,264 INFO [WALProcedureStoreSyncThread] wal.WALProcedureStore: Remove all state logs with ID less than 1, since all the active procedures are in the latest log 2019-07-11 19:12:00,266 INFO [WALProcedureStoreSyncThread] wal.ProcedureWALFile: Archiving file:/var/lib/ambari-metrics-collector/hbase/MasterProcWALs/pv2-00000000000000000001.log to file:/var/lib/ambari-metrics-collector/hbase/oldWALs/pv2-00000000000000000001.log 2019-07-11 19:12:00,271 INFO [PEWorker-3] procedure.ServerCrashProcedure: Start pid=27, state=RUNNABLE:SERVER_CRASH_START, locked=true; ServerCrashProcedure server=kvm07,61320,1562826676313, splitWal=true, meta=false 2019-07-11 19:12:00,289 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=61300] master.ServerManager: Registering regionserver=kvm07,61320,1562839917726 2019-07-11 19:12:00,298 INFO [Thread-23] master.ServerManager: Waiting on regionserver count=1; waited=101ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=0ms 2019-07-11 19:12:00,325 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node created, adding [kvm07,61320,1562839917726] 2019-07-11 19:12:00,604 INFO [RS:0;kvm07:61320] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2019-07-11 19:12:00,663 INFO [RS:0;kvm07:61320] regionserver.MetricsRegionServerWrapperImpl: Computing regionserver metrics every 5000 milliseconds 2019-07-11 19:12:00,711 INFO [RS:0;kvm07:61320] regionserver.MemStoreFlusher: globalMemStoreLimit=725.8 M, globalMemStoreLimitLowMark=622.1 M, Offheap=false 2019-07-11 19:12:00,723 INFO [RS:0;kvm07:61320] throttle.PressureAwareCompactionThroughputController: Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2019-07-11 19:12:00,727 INFO [RS:0;kvm07:61320] regionserver.HRegionServer: CompactionChecker runs every PT10S 2019-07-11 19:12:00,791 INFO [SplitLogWorker-kvm07:61320] regionserver.SplitLogWorker: SplitLogWorker kvm07,61320,1562839917726 starting 2019-07-11 19:12:00,795 WARN [RS:0;kvm07:61320] util.MemorySizeUtil: hbase.regionserver.global.memstore.upperLimit is deprecated by hbase.regionserver.global.memstore.size 2019-07-11 19:12:00,799 INFO [RS:0;kvm07:61320] regionserver.HeapMemoryManager: Starting, tuneOn=false 2019-07-11 19:12:00,803 WARN [RS:0;kvm07:61320] util.MemorySizeUtil: hbase.regionserver.global.memstore.lowerLimit is deprecated. Instead use hbase.regionserver.global.memstore.size.lower.limit 2019-07-11 19:12:00,821 INFO [ReplicationExecutor-0] regionserver.ReplicationSourceManager: Current list of replicators: [kvm07,61320,1562826676313, kvm07,61320,1562839917726] other RSs: [kvm07,61320,1562839917726] 2019-07-11 19:12:00,846 INFO [RS:0;kvm07:61320] regionserver.HRegionServer: Serving as kvm07,61320,1562839917726, RpcServer on kvm07/10.3.0.25:61320, sessionid=0x16be08492a30002 2019-07-11 19:12:00,852 INFO [RS:0;kvm07:61320] quotas.RegionServerRpcQuotaManager: Quota support disabled 2019-07-11 19:12:00,853 INFO [RS:0;kvm07:61320] quotas.RegionServerSpaceQuotaManager: Quota support disabled, not starting space quota manager. 2019-07-11 19:12:01,821 INFO [Thread-23] master.ServerManager: Waiting on regionserver count=1; waited=1623ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=-1522ms 2019-07-11 19:12:03,326 INFO [Thread-23] master.ServerManager: Waiting on regionserver count=1; waited=3129ms, expecting min=1 server(s), max=NO_LIMIT server(s), timeout=4500ms, lastChange=-3028ms 2019-07-11 19:12:03,924 INFO [RS:0;kvm07:61320] wal.AbstractFSWAL: WAL configuration: blocksize=64 MB, rollsize=32 MB, prefix=kvm07%2C61320%2C1562839917726, suffix=, logDir=file:/var/lib/ambari-metrics-collector/hbase/WALs/kvm07,61320,1562839917726, archiveDir=file:/var/lib/ambari-metrics-collector/hbase/oldWALs 2019-07-11 19:12:03,975 INFO [RS:0;kvm07:61320] wal.AbstractFSWAL: New WAL /var/lib/ambari-metrics-collector/hbase/WALs/kvm07,61320,1562839917726/kvm07%2C61320%2C1562839917726.1562839923950 2019-07-11 19:12:04,494 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37788 2019-07-11 19:12:04,497 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37788 2019-07-11 19:12:04,501 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a30006 with negotiated timeout 120000 for client /10.3.0.25:37788 2019-07-11 19:12:04,741 INFO [Thread-23] master.ServerManager: Finished waiting on RegionServer count=1; waited=4544ms, expected min=1 server(s), max=NO_LIMIT server(s), master is running 2019-07-11 19:12:04,746 WARN [Thread-23] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1562839919335, server=kvm07,61320,1562251031381}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2019-07-11 19:12:05,746 WARN [Thread-23] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1562839919335, server=kvm07,61320,1562251031381}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2019-07-11 19:12:07,747 WARN [Thread-23] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1562839919335, server=kvm07,61320,1562251031381}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2019-07-11 19:12:10,139 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x16be08492a30006, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:748) 2019-07-11 19:12:10,142 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.3.0.25:37788 which had sessionid 0x16be08492a30006 2019-07-11 19:12:11,748 WARN [Thread-23] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1562839919335, server=kvm07,61320,1562251031381}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2019-07-11 19:12:19,749 WARN [Thread-23] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1562839919335, server=kvm07,61320,1562251031381}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2019-07-11 19:12:28,586 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37808 2019-07-11 19:12:28,590 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37808 2019-07-11 19:12:28,593 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a30007 with negotiated timeout 120000 for client /10.3.0.25:37808 2019-07-11 19:12:34,566 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x16be08492a30007, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:748) 2019-07-11 19:12:34,568 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.3.0.25:37808 which had sessionid 0x16be08492a30007 2019-07-11 19:12:35,749 WARN [Thread-23] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1562839919335, server=kvm07,61320,1562251031381}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2019-07-11 19:12:50,366 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37822 2019-07-11 19:12:50,373 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37822 2019-07-11 19:12:50,378 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a30008 with negotiated timeout 120000 for client /10.3.0.25:37822 2019-07-11 19:12:55,683 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x16be08492a30008, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:748) 2019-07-11 19:12:55,684 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.3.0.25:37822 which had sessionid 0x16be08492a30008 2019-07-11 19:12:58,639 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16be08492a30003 2019-07-11 19:12:58,642 INFO [ReadOnlyZKClient-kvm07.hadoop.local:61181@0x70918201] zookeeper.ZooKeeper: Session: 0x16be08492a30003 closed 2019-07-11 19:12:58,644 INFO [ReadOnlyZKClient-kvm07.hadoop.local:61181@0x70918201-EventThread] zookeeper.ClientCnxn: EventThread shut down 2019-07-11 19:12:58,647 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.3.0.25:37778 which had sessionid 0x16be08492a30003 2019-07-11 19:13:00,031 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16be08492a30004 2019-07-11 19:13:00,036 INFO [ReadOnlyZKClient-kvm07.hadoop.local:61181@0x0f554089-EventThread] zookeeper.ClientCnxn: EventThread shut down 2019-07-11 19:13:00,034 INFO [ReadOnlyZKClient-kvm07.hadoop.local:61181@0x0f554089] zookeeper.ZooKeeper: Session: 0x16be08492a30004 closed 2019-07-11 19:13:00,041 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.3.0.25:37782 which had sessionid 0x16be08492a30004 2019-07-11 19:13:07,750 WARN [Thread-23] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1562839919335, server=kvm07,61320,1562251031381}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2019-07-11 19:13:10,739 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37834 2019-07-11 19:13:10,747 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37834 2019-07-11 19:13:10,750 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a30009 with negotiated timeout 120000 for client /10.3.0.25:37834 2019-07-11 19:13:14,551 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 2019-07-11 19:13:17,127 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x16be08492a30009, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:748) 2019-07-11 19:13:17,129 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.3.0.25:37834 which had sessionid 0x16be08492a30009 2019-07-11 19:13:31,077 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37888 2019-07-11 19:13:31,082 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37888 2019-07-11 19:13:31,086 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a3000a with negotiated timeout 120000 for client /10.3.0.25:37888 2019-07-11 19:13:36,608 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x16be08492a3000a, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:748) 2019-07-11 19:13:36,610 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.3.0.25:37888 which had sessionid 0x16be08492a3000a 2019-07-11 19:13:54,001 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16bdfba811a0005, timeout of 120000ms exceeded 2019-07-11 19:13:54,001 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16bdfba811a0001, timeout of 120000ms exceeded 2019-07-11 19:13:54,001 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16bdfba811a000c, timeout of 120000ms exceeded 2019-07-11 19:13:54,001 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16bdfba811a000d, timeout of 120000ms exceeded 2019-07-11 19:13:54,001 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16bdfba811a0000, timeout of 120000ms exceeded 2019-07-11 19:13:54,001 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16bdfba811a000e, timeout of 120000ms exceeded 2019-07-11 19:13:54,002 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16bdfba811a000a, timeout of 120000ms exceeded 2019-07-11 19:13:54,002 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16bdfba811a000b, timeout of 120000ms exceeded 2019-07-11 19:13:54,002 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16bdfba811a0009, timeout of 120000ms exceeded 2019-07-11 19:13:54,002 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16bdfba811a0005 2019-07-11 19:13:54,003 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16bdfba811a0001 2019-07-11 19:13:54,003 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16bdfba811a000c 2019-07-11 19:13:54,003 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16bdfba811a000d 2019-07-11 19:13:54,003 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16bdfba811a0000 2019-07-11 19:13:54,003 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16bdfba811a000e 2019-07-11 19:13:54,003 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16bdfba811a000a 2019-07-11 19:13:54,003 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16bdfba811a000b 2019-07-11 19:13:54,003 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16bdfba811a0009 2019-07-11 19:13:54,438 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37902 2019-07-11 19:13:54,443 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37902 2019-07-11 19:13:54,446 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a3000b with negotiated timeout 120000 for client /10.3.0.25:37902 2019-07-11 19:14:01,466 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x16be08492a3000b, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:748) 2019-07-11 19:14:01,468 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.3.0.25:37902 which had sessionid 0x16be08492a3000b 2019-07-11 19:14:11,752 WARN [Thread-23] master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=OPEN, ts=1562839919335, server=kvm07,61320,1562251031381}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined. 2019-07-11 19:14:12,000 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16be08492a30006, timeout of 120000ms exceeded 2019-07-11 19:14:12,000 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16be08492a30006 2019-07-11 19:14:17,667 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37918 2019-07-11 19:14:17,676 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37918 2019-07-11 19:14:17,678 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a3000c with negotiated timeout 120000 for client /10.3.0.25:37918 2019-07-11 19:14:23,456 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x16be08492a3000c, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:748) 2019-07-11 19:14:23,456 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.3.0.25:37918 which had sessionid 0x16be08492a3000c 2019-07-11 19:14:34,554 INFO [timeline] timeline.HadoopTimelineMetricsSink: No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 2019-07-11 19:14:36,000 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16be08492a30007, timeout of 120000ms exceeded 2019-07-11 19:14:36,001 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16be08492a30007 2019-07-11 19:14:40,843 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37974 2019-07-11 19:14:40,847 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37974 2019-07-11 19:14:40,849 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a3000d with negotiated timeout 120000 for client /10.3.0.25:37974 2019-07-11 19:14:45,846 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x16be08492a3000d, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:748) 2019-07-11 19:14:45,848 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.3.0.25:37974 which had sessionid 0x16be08492a3000d 2019-07-11 19:14:59,473 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:37990 2019-07-11 19:14:59,478 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:37990 2019-07-11 19:14:59,481 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a3000e with negotiated timeout 120000 for client /10.3.0.25:37990 2019-07-11 19:15:00,000 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16be08492a30008, timeout of 120000ms exceeded 2019-07-11 19:15:00,001 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16be08492a30008 2019-07-11 19:15:05,225 WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x16be08492a3000e, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:748) 2019-07-11 19:15:05,227 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.3.0.25:37990 which had sessionid 0x16be08492a3000e 2019-07-11 19:15:18,000 INFO [SessionTracker] server.ZooKeeperServer: Expiring session 0x16be08492a30009, timeout of 120000ms exceeded 2019-07-11 19:15:18,000 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16be08492a30009 2019-07-11 19:15:20,219 ERROR [main] master.HMasterCommandLine: Master exiting java.lang.RuntimeException: Master not initialized after 200000ms at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:229) at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:197) at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:413) at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:232) at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3100) 2019-07-11 19:15:20,236 INFO [shutdown-hook-0] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@4bee18dc 2019-07-11 19:15:20,236 INFO [shutdown-hook-0] regionserver.HRegionServer: ***** STOPPING region server 'kvm07,61320,1562839917726' ***** 2019-07-11 19:15:20,237 INFO [shutdown-hook-0] regionserver.HRegionServer: STOPPED: Shutdown hook 2019-07-11 19:15:20,237 INFO [RS:0;kvm07:61320] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread 2019-07-11 19:15:20,238 INFO [RS:0;kvm07:61320] regionserver.HRegionServer: Stopping infoServer 2019-07-11 19:15:20,238 INFO [SplitLogWorker-kvm07:61320] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting. 2019-07-11 19:15:20,239 INFO [SplitLogWorker-kvm07:61320] regionserver.SplitLogWorker: SplitLogWorker kvm07,61320,1562839917726 exiting 2019-07-11 19:15:20,266 INFO [RS:0;kvm07:61320] handler.ContextHandler: Stopped o.e.j.w.WebAppContext@96a75da{/,null,UNAVAILABLE}{file:/usr/lib/ams-hbase/hbase-webapps/regionserver} 2019-07-11 19:15:20,288 INFO [RS:0;kvm07:61320] server.AbstractConnector: Stopped ServerConnector@61e7bf2f{HTTP/1.1,[http/1.1]}{0.0.0.0:61330} 2019-07-11 19:15:20,289 INFO [RS:0;kvm07:61320] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@4da9f723{/static,file:///usr/lib/ams-hbase/hbase-webapps/static/,UNAVAILABLE} 2019-07-11 19:15:20,290 INFO [RS:0;kvm07:61320] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@46cb98a3{/logs,file:///var/log/ambari-metrics-collector/,UNAVAILABLE} 2019-07-11 19:15:20,292 INFO [RS:0;kvm07:61320] regionserver.HeapMemoryManager: Stopping 2019-07-11 19:15:20,293 INFO [RS:0;kvm07:61320] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager gracefully. 2019-07-11 19:15:20,293 INFO [RS:0;kvm07:61320] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager gracefully. 2019-07-11 19:15:20,293 INFO [RS:0;kvm07:61320] regionserver.HRegionServer: stopping server kvm07,61320,1562839917726 2019-07-11 19:15:20,294 INFO [RS:0;kvm07:61320] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x0f554089 to kvm07.hadoop.local:61181 2019-07-11 19:15:20,293 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting 2019-07-11 19:15:20,301 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting 2019-07-11 19:15:20,301 INFO [RS:0;kvm07:61320] regionserver.HRegionServer: stopping server kvm07,61320,1562839917726; all regions closed. 2019-07-11 19:15:20,311 INFO [RS:0;kvm07:61320] wal.AbstractFSWAL: Closed WAL: AsyncFSWAL kvm07%2C61320%2C1562839917726:(num 1562839923950) 2019-07-11 19:15:20,312 INFO [RS:0;kvm07:61320] regionserver.Leases: Closed leases 2019-07-11 19:15:20,315 INFO [RS:0;kvm07:61320] hbase.ChoreService: Chore service for: regionserver/kvm07:61320 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 10000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region kvm07,61320,1562839917726 Period: 120000 Unit: MILLISECONDS]] on shutdown 2019-07-11 19:15:20,315 INFO [RS:0;kvm07:61320] regionserver.CompactSplit: Waiting for Split Thread to finish... 2019-07-11 19:15:20,315 INFO [regionserver/kvm07:61320.logRoller] regionserver.LogRoller: LogRoller exiting. 2019-07-11 19:15:20,315 INFO [RS:0;kvm07:61320] regionserver.CompactSplit: Waiting for Large Compaction Thread to finish... 2019-07-11 19:15:20,316 INFO [RS:0;kvm07:61320] regionserver.CompactSplit: Waiting for Small Compaction Thread to finish... 2019-07-11 19:15:20,318 INFO [RS:0;kvm07:61320] ipc.NettyRpcServer: Stopping server on /10.3.0.25:61320 2019-07-11 19:15:20,418 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxnFactory: Accepted socket connection from /10.3.0.25:38012 2019-07-11 19:15:20,418 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.ZooKeeperServer: Client attempting to establish new session at /10.3.0.25:38012 2019-07-11 19:15:20,422 INFO [ProcessThread(sid:0 cport:-1):] server.PrepRequestProcessor: Processed session termination for sessionid: 0x16be08492a30002 2019-07-11 19:15:20,497 INFO [SyncThread:0] server.ZooKeeperServer: Established session 0x16be08492a3000f with negotiated timeout 120000 for client /10.3.0.25:38012 2019-07-11 19:15:20,512 INFO [RS:0;kvm07:61320] zookeeper.ZooKeeper: Session: 0x16be08492a30002 closed 2019-07-11 19:15:20,512 INFO [RS:0;kvm07:61320] regionserver.HRegionServer: Exiting; stopping=kvm07,61320,1562839917726; zookeeper connection closed. 2019-07-11 19:15:20,518 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down 2019-07-11 19:15:20,520 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:61181] server.NIOServerCnxn: Closed socket connection for client /10.3.0.25:37774 which had sessionid 0x16be08492a30002 2019-07-11 19:15:20,526 INFO [shutdown-hook-0] regionserver.ShutdownHook: Starting fs shutdown hook thread. 2019-07-11 19:15:20,526 INFO [RegionServerTracker-0] master.RegionServerTracker: RegionServer ephemeral node deleted, processing expiration [kvm07,61320,1562839917726] 2019-07-11 19:15:20,527 INFO [RegionServerTracker-0] master.ServerManager: Processing expiration of kvm07,61320,1562839917726 on kvm07,61300,1562839913433 2019-07-11 19:15:20,528 INFO [shutdown-hook-0] regionserver.ShutdownHook: Shutdown hook finished.