Member since
03-30-2017
7
Posts
0
Kudos Received
0
Solutions
04-06-2017
02:13 PM
In additional I should correct previous information: ZooKeeper at NameNode doesn't accept unbroken node, it says that node needs another server. Yes! Another server - it's a hbase-master. According to the ZooKeeper log at hbase-master, it seems that ZooKeeper doesn't refuse unbroken node...
... View more
04-06-2017
02:09 PM
Ok, what I found (I think, that I'm on a right way): according to the table at this page: HDP we have ZooKeeper ver. 3.4.6. Actually the current version of the ZooKeeper is 3.5.2 (almost 3.5.3). I've looked through ZooKeeper ver. 3.4.6 code briefly and found that there is no check if new node belongs to the cluster quorum or not! It seems that ZooKeeper ver. 3.4.6 don't control this. Then I looked through ZooKeeper ver. 3.5.3_rc0 code and found, that it checks if new node belongs to the cluster quorum or not. I still have to understand if it rejects node, which is not from the cluster. And then I should perform the same code check for HBase sources.
... View more
04-05-2017
10:31 PM
Surprise! ZooKeeper doesn't accept unbroken node: 2017-03-30 10:51:24,552 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.44.3.244:40718
2017-03-30 10:51:24,552 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@841] - Refusing session request for client /10.44.3.244:40718 as it has seen zxid 0x1000000fc our last
zxid is 0x1000000d9 client must try another server
2017-03-30 10:51:24,553 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.44.3.244:40718 (no session established for client)
2017-03-30 10:51:26,984 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.44.3.244:40729
2017-03-30 10:51:26,985 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@841] - Refusing session request for client /10.44.3.244:40729 as it has seen zxid 0x100000117 our last
zxid is 0x1000000e1 client must try another server
2017-03-30 10:51:26,985 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.44.3.244:40729 (no session established for client)
2017-03-30 10:51:27,719 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.44.3.244:40733
2017-03-30 10:51:27,720 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@841] - Refusing session request for client /10.44.3.244:40733 as it has seen zxid 0x1000000fc our last
zxid is 0x1000000e2 client must try another server
2017-03-30 10:51:27,720 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.44.3.244:40733 (no session established for client)
2017-03-30 10:51:30,276 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /0:0:0:0:0:0:0:1:56983 which had sessionid 0x15b1fadae6e0010
2017-03-30 10:51:30,412 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:60303
2017-03-30 10:51:30,413 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /127.0.0.1:60303
2017-03-30 10:51:30,416 [myid:1] - INFO [CommitProcessor:1:ZooKeeperServer@617] - Established session 0x15b1fadae6e0011 with negotiated timeout 90000 for client /127.0.0.1:60303
2017-03-30 10:51:31,892 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.44.3.244:40749
2017-03-30 10:51:31,893 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x25b1f8e14be0006 at /10.44.3.244:40749
2017-03-30 10:51:31,893 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x25b1f8e14be0006
2017-03-30 10:51:31,894 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@610] - Invalid session 0x25b1f8e14be0006 for client /10.44.3.244:40749, probably expired
2017-03-30 10:51:31,894 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.44.3.244:40749 which had sessionid 0x25b1f8e14be0006
2017-03-30 10:51:32,033 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.44.3.244:40751
2017-03-30 10:51:32,033 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@841] - Refusing session request for client /10.44.3.244:40751 as it has seen zxid 0x100000117 our last
zxid is 0x1000000fd client must try another server
2017-03-30 10:51:32,033 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.44.3.244:40751 (no session established for client)
2017-03-30 10:51:35,825 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.44.3.244:40763
2017-03-30 10:51:35,826 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@841] - Refusing session request for client /10.44.3.244:40763 as it has seen zxid 0x100000117 our last
zxid is 0x100000107 client must try another server
2017-03-30 10:51:35,826 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.44.3.244:40763 (no session established for client)
2017-03-30 10:51:39,468 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /127.0.0.1:60303 which had sessionid 0x15b1fadae6e0011
2017-03-30 10:51:39,600 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /127.0.0.1:60311
2017-03-30 10:51:39,601 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@868] - Client attempting to establish new session at /127.0.0.1:60311
2017-03-30 10:51:39,604 [myid:1] - INFO [CommitProcessor:1:ZooKeeperServer@617] - Established session 0x15b1fadae6e0012 with negotiated timeout 90000 for client /127.0.0.1:603112017-03-30 10:51:40,224 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] - Accepted socket connection from /10.44.3.244:40779
2017-03-30 10:51:40,225 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@861] - Client attempting to renew session 0x35b1f8e14bf0003 at /10.44.3.244:40779
2017-03-30 10:51:40,226 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:Learner@108] - Revalidating client: 0x35b1f8e14bf0003
2017-03-30 10:51:40,227 [myid:1] - INFO [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@610] - Invalid session 0x35b1f8e14bf0003 for client /10.44.3.244:40779, probably expired
2017-03-30 10:51:40,228 [myid:1] - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.44.3.244:40779 which had sessionid 0x35b1f8e14bf0003
??? How does HBase accept it?
... View more
04-03-2017
01:09 PM
I just can't understand: is it a feature or a bug that ZooKeeper accepts node with name and IP it hasn't in config files? May be it should be fixed in source code of ZooKeeper?
... View more
04-03-2017
11:54 AM
Thanks for your answers. Actually I have very stupid and uncomfortable situation. It's not me, who partially destroys cluster. I can't impact on those persons. I can't explain them how to do this in a proper way. I only have to solve the situation with such unbroken nodes. Moreover, I can't change nodes' names and IPs. So, in perfect case it should be some option for ZooKeeper or HBase, which does not allow to accept nodes not from config files. And it should works in automatic mode without the need to monitor it and do some manipulations manually. Is it possible?
... View more
03-31-2017
02:27 PM
It seems that ZooKeeper accepts Heartbeat signal from this unbroken node. But why? ZooKeeper at new cluster doesn't have this node at his zoo.cfg. Is it a normal situation, that ZooKeeper accepts any Heartbeat signal (not only from nodes at zoo.cfg)? And if it's a normal situation, how would I prevent this?
... View more
03-31-2017
02:27 PM
Here is a situation:
We have a cluster with 5 nodes for instance: vm240 (IP 10.44.3.240), vm241 (IP 10.44.3.241), vm242 (IP 10.44.3.242), vm243 (IP 10.44.3.243) and vm244 (IP 10.44.3.244). Then we destroy first 4 nodes (including NameNode vm240) and leave 5th node (vm244) as is without destroying. Then we deploy new cluster on the 4 nodes. Unexpectedly we find in logs such records (vm244 with IP 10.44.3.244 is the 5th unbroken node, vm241 - new NameNode, vm240 - HBase-master): root@vm241 ~ # grep -r "10.44.3.244" /var/log/hadoop/
Binary file /var/log/hadoop/hdfs/.hadoop-hdfs-namenode-vm241.log.swp matches
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 09:33:27,359 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 8020, call org.apache.hadoop.hdfs.server.protocol.DatanodeProtocol.sendHeartbeat from 10.44.3.244:55774 Call#1027 Retry#0
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:org.apache.hadoop.hdfs.protocol.UnregisteredNodeException: Unregistered server: DatanodeRegistration(10.44.3.244:50010, datanodeUuid=f50fada0-4518-4af3-a9f7-8f7d5f1294b2, infoPort=50075, infoSecurePort=50475, ipcPort=50020, storageInfo=lv=-56;cid=CID-f13f3676-f02d-4949-b13b-d524ad1bc4a2;nsid=2057042102;c=0)
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 09:33:39,584 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from 10.44.3.244:55883 Call#122 Retry#15: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 09:33:40,590 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from 10.44.3.244:55883 Call#123 Retry#0: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 09:33:41,909 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from 10.44.3.244:55883 Call#123 Retry#2: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 09:33:48,641 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from 10.44.3.244:55883 Call#123 Retry#4: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 09:34:16,807 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from 10.44.3.244:56217 Call#123 Retry#6: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 09:34:42,941 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.renewLease from 10.44.3.244:56449 Call#123 Retry#8: org.apache.hadoop.ipc.StandbyException: Operation category WRITE is not supported in state standby
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 09:43:08,909 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 10.44.3.244:59554 Call#140 Retry#0: java.io.FileNotFoundException: Parent directory doesn't exist: /apps/hbase/data/WALs/vm244-int,60020,1490704985478
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 09:44:00,788 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741868_1044{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490708640664
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 09:44:00,963 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490708640664 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 10:44:01,098 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741877_1053{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-f6f210ae-3df9-4cbc-8141-ea9ec28e9999:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490712241062
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 10:44:01,120 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490712241062 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 10:44:01,138 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490708640664 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 11:44:01,233 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741883_1059{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490715841201
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 11:44:01,252 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490715841201 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 11:44:01,271 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490712241062 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 12:44:01,356 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741889_1065{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f6f210ae-3df9-4cbc-8141-ea9ec28e9999:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-ad6dda9b-df16-4e10-bb5b-243a8695e19b:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-c41163f0-c146-48d9-b2e4-389b9d794164:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490719441324
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 12:44:01,377 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490719441324 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 12:44:01,395 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490715841201 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 13:44:01,481 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741895_1071{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490723041449
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 13:44:01,503 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490723041449 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 13:44:01,525 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490719441324 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 14:44:01,600 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741901_1077{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f6f210ae-3df9-4cbc-8141-ea9ec28e9999:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-ad6dda9b-df16-4e10-bb5b-243a8695e19b:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490726641580
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 14:44:01,617 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490726641580 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 14:44:01,636 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490723041449 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 15:44:01,727 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741907_1083{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490730241701
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 15:44:01,751 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490730241701 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 15:44:01,768 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490726641580 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 16:44:01,855 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741913_1089{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490733841834
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 16:44:01,873 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490733841834 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 16:44:01,891 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490730241701 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 17:44:01,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741919_1095{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-c41163f0-c146-48d9-b2e4-389b9d794164:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490737441953
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 17:44:01,988 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490737441953 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 17:44:02,005 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490733841834 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 18:44:02,107 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741925_1101{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-f6f210ae-3df9-4cbc-8141-ea9ec28e9999:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490741042087
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 18:44:02,125 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490741042087 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 18:44:02,141 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490737441953 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 19:44:02,231 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741931_1107{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490744642205
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 19:44:02,249 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490744642205 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 19:44:02,270 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490741042087 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 20:44:02,358 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741937_1113{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490748242331
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 20:44:02,374 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490748242331 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 20:44:02,391 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490744642205 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 21:44:02,475 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741943_1119{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490751842451
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 21:44:02,492 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490751842451 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 21:44:02,510 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490748242331 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 22:44:02,596 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741949_1125{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f6f210ae-3df9-4cbc-8141-ea9ec28e9999:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-ad6dda9b-df16-4e10-bb5b-243a8695e19b:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490755442570
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 22:44:02,613 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490755442570 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 22:44:02,634 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490751842451 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 23:44:02,717 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741955_1131{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c41163f0-c146-48d9-b2e4-389b9d794164:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490759042696
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 23:44:02,734 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490759042696 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-28 23:44:02,752 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490755442570 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 00:44:02,904 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741961_1137{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c41163f0-c146-48d9-b2e4-389b9d794164:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-ad6dda9b-df16-4e10-bb5b-243a8695e19b:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490762642879
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 00:44:02,920 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490762642879 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 00:44:02,938 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490759042696 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 01:44:03,025 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741967_1143{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490766242999
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 01:44:03,042 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490766242999 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 01:44:03,059 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490762642879 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 02:44:03,147 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741973_1149{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-c41163f0-c146-48d9-b2e4-389b9d794164:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490769843121
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 02:44:03,165 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490769843121 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 02:44:03,182 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490766242999 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 03:44:03,270 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741979_1155{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-c41163f0-c146-48d9-b2e4-389b9d794164:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-f6f210ae-3df9-4cbc-8141-ea9ec28e9999:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490773443246
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 03:44:03,285 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490773443246 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 03:44:03,302 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490769843121 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 04:44:03,391 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741985_1161{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-ad6dda9b-df16-4e10-bb5b-243a8695e19b:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490777043367
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 04:44:03,407 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490777043367 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 04:44:03,424 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490773443246 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 05:44:03,520 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741991_1167{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490780643488
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 05:44:03,538 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490780643488 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 05:44:03,558 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490777043367 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 06:44:03,639 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073741997_1173{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-c41163f0-c146-48d9-b2e4-389b9d794164:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490784243618
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 06:44:03,660 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490784243618 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 06:44:03,676 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490780643488 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 07:44:03,764 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742003_1179{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490787843735
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 07:44:03,782 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490787843735 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 07:44:03,801 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490784243618 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 08:44:03,883 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742009_1185{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-c41163f0-c146-48d9-b2e4-389b9d794164:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490791443859
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 08:44:03,898 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490791443859 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 08:44:03,917 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490787843735 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 09:44:03,994 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742015_1191{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-f6f210ae-3df9-4cbc-8141-ea9ec28e9999:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-ad6dda9b-df16-4e10-bb5b-243a8695e19b:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490795043974
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 09:44:04,013 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490795043974 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 09:44:04,029 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490791443859 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 10:44:04,104 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742021_1197{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-ad6dda9b-df16-4e10-bb5b-243a8695e19b:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490798644085
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 10:44:04,121 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490798644085 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 10:44:04,140 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490795043974 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 11:44:04,210 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742027_1203{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490802244194
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 11:44:04,225 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490802244194 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 11:44:04,242 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490798644085 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 12:44:04,314 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742033_1209{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-ad6dda9b-df16-4e10-bb5b-243a8695e19b:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490805844300
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 12:44:04,336 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490805844300 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 12:44:04,354 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490802244194 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 13:44:04,422 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742039_1215{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-f6f210ae-3df9-4cbc-8141-ea9ec28e9999:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-ad6dda9b-df16-4e10-bb5b-243a8695e19b:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490809444412
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 13:44:04,436 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490809444412 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 13:44:04,452 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490805844300 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 14:44:04,522 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742045_1221{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-f6f210ae-3df9-4cbc-8141-ea9ec28e9999:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490813044509
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 14:44:04,542 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490813044509 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 14:44:04,560 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490809444412 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 15:44:04,637 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742051_1227{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490816644624
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 15:44:04,655 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490816644624 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 15:44:04,673 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490813044509 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 16:44:04,750 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742057_1233{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-c41163f0-c146-48d9-b2e4-389b9d794164:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490820244737
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 16:44:04,767 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490820244737 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 16:44:04,786 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490816644624 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 17:44:04,865 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742063_1239{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490823844850
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 17:44:04,884 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490823844850 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 17:44:04,901 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490820244737 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 18:44:04,976 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742069_1245{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490827444964
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 18:44:04,992 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490827444964 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 18:44:05,010 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490823844850 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 19:44:05,085 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742075_1251{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-ad6dda9b-df16-4e10-bb5b-243a8695e19b:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490831045072
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 19:44:05,105 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490831045072 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 19:44:05,123 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490827444964 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 20:44:05,202 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742081_1257{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490834645187
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 20:44:05,218 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490834645187 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 20:44:05,242 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490831045072 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 21:44:05,325 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742087_1263{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-f6f210ae-3df9-4cbc-8141-ea9ec28e9999:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490838245302
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 21:44:05,340 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490838245302 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 21:44:05,361 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490834645187 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 22:44:05,442 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742093_1269{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-c41163f0-c146-48d9-b2e4-389b9d794164:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490841845418
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 22:44:05,459 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490841845418 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 22:44:05,476 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490838245302 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 23:44:05,561 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742099_1275{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490845445532
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 23:44:05,580 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490845445532 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-29 23:44:05,596 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490841845418 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 00:44:05,663 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742105_1281{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490849045645
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 00:44:05,679 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490849045645 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 00:44:05,696 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490845445532 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 01:44:05,768 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742111_1287{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-af1b673a-2ecf-40a2-9a99-d797efe3943d:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490852645757
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 01:44:05,789 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490852645757 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 01:44:05,807 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490849045645 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 02:44:05,889 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742117_1293{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-c41163f0-c146-48d9-b2e4-389b9d794164:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490856245873
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 02:44:05,905 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490856245873 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 02:44:05,925 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490852645757 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 03:44:06,004 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742123_1299{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490859845987
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 03:44:06,021 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490859845987 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 03:44:06,038 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490856245873 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 04:44:06,114 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742129_1305{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-f6f210ae-3df9-4cbc-8141-ea9ec28e9999:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490863446096
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 04:44:06,130 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490863446096 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 04:44:06,149 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490859845987 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 05:44:06,226 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742135_1311{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-dfef3360-3c35-4f69-95e5-6f6bdd54ed0a:NORMAL:10.44.3.240:50010|RBW], ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490867046203
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 05:44:06,241 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490867046203 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 05:44:06,258 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490863446096 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 06:44:06,332 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742141_1317{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-e3dcfbf3-f942-4306-a7f2-afc25edb7995:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-c41163f0-c146-48d9-b2e4-389b9d794164:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490870646310
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 06:44:06,348 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490870646310 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 06:44:06,365 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490867046203 is closed by DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 07:44:06,436 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1073742147_1323{UCState=UNDER_CONSTRUCTION, truncateBlock=null, primaryNodeIndex=-1, replicas=[ReplicaUC[[DISK]DS-87f075bc-b11c-4154-aa49-9032cba5b5f0:NORMAL:10.44.3.243:50010|RBW], ReplicaUC[[DISK]DS-5e18542c-82a7-4b98-aa4a-d01edbb6a763:NORMAL:10.44.3.242:50010|RBW], ReplicaUC[[DISK]DS-0d46b140-7644-4ab6-acdf-904f623dcd63:NORMAL:10.44.3.240:50010|RBW]]} for /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490874246419
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 07:44:06,454 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* fsync: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490874246419 for DFSClient_NONMAPREDUCE_-613389291_1
/var/log/hadoop/hdfs/hadoop-hdfs-namenode-vm241.log:2017-03-30 07:44:06,471 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /apps/hbase/data/WALs/10.44.3.244,60020,1490708637722/10.44.3.244%2C60020%2C1490708637722.default.1490870646310 is closed by DFSClient_NONMAPREDUCE_-613389291_1
root@vm240 ~ # grep -r "10.44.3.244" /var/log/hadoop/
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-28 13:44:01,473 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073741895_1071 src: /10.44.3.244:35335 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-28 14:44:01,623 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:35335, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073741895_1071, duration: 3600142664668
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-28 18:44:02,118 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073741925_1101 src: /10.44.3.244:46090 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-28 19:44:02,268 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:46090, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073741925_1101, duration: 3600143804770
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-28 21:44:02,483 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073741943_1119 src: /10.44.3.244:41250 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-28 22:44:02,629 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:41250, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073741943_1119, duration: 3600139823109
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-28 23:44:02,726 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073741955_1131 src: /10.44.3.244:56847 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 00:44:02,913 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073741961_1137 src: /10.44.3.244:36413 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 00:44:02,935 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:56847, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073741955_1131, duration: 3600202583432
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 01:44:03,057 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:36413, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073741961_1137, duration: 3600137961113
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 03:44:03,280 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073741979_1155 src: /10.44.3.244:59801 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 04:44:03,421 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:59801, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073741979_1155, duration: 3600134677474
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 07:44:03,774 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073742003_1179 src: /10.44.3.244:34529 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 08:44:03,915 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:34529, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073742003_1179, duration: 3600135831742
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 09:44:04,004 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073742015_1191 src: /10.44.3.244:50114 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 10:44:04,117 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073742021_1197 src: /10.44.3.244:57907 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 10:44:04,140 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:50114, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073742015_1191, duration: 3600129871895
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 11:44:04,242 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:57907, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073742021_1197, duration: 3600119392486
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 13:44:04,431 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073742039_1215 src: /10.44.3.244:53063 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 14:44:04,533 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073742045_1221 src: /10.44.3.244:60860 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 14:44:04,559 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:53063, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073742039_1215, duration: 3600123050809
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 15:44:04,648 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073742051_1227 src: /10.44.3.244:40423 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 15:44:04,672 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:60860, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073742045_1221, duration: 3600132460337
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 16:44:04,781 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:40423, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073742051_1227, duration: 3600126413043
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-29 23:44:05,571 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073742099_1275 src: /10.44.3.244:46310 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-30 00:44:05,697 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:46310, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073742099_1275, duration: 3600119479703
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-30 05:44:06,236 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-1918911033-10.44.3.241-1490707991735:blk_1073742135_1311 src: /10.44.3.244:36621 dest: /10.44.3.240:50010
/var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm240.log:2017-03-30 06:44:06,366 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /10.44.3.244:36621, dest: /10.44.3.240:50010, bytes: 91, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_-613389291_1, offset: 0, srvID: 2f5feff8-ce95-4aae-9ea5-b79e94b09771, blockid: BP-1918911033-10.44.3.241-1490707991735:blk_1073742135_1311, duration: 3600123615444
root@vm240 ~ # grep -r "10.44.3.244" /var/log/hbase/
/var/log/hbase/hbase-hbase-master-vm240.log:2017-03-28 09:44:00,457 INFO [PriorityRpcServer.handler=6,queue=0,port=60000] master.ServerManager: Registering server=10.44.3.244,60020,1490708637722
/var/log/hbase/hbase-hbase-master-vm240.log:2017-03-28 09:46:54,181 INFO [vm240-int,60000,1490708164034_ChoreService_1] master.HMaster: balance hri=ddnbindex,,1490708260006.199b8b1bae036e47545df3350591eeaa., src=vm240-int,60020,1490708207969, dest=10.44.3.244,60020,1490708637722
/var/log/hbase/hbase-hbase-master-vm240.log:2017-03-28 09:46:54,246 INFO [AM.ZK.Worker-pool3-t36] master.AssignmentManager: Assigning ddnbindex,,1490708260006.199b8b1bae036e47545df3350591eeaa. to 10.44.3.244,60020,1490708637722
/var/log/hbase/hbase-hbase-master-vm240.log:2017-03-28 09:46:54,246 INFO [AM.ZK.Worker-pool3-t36] master.RegionStates: Transition {199b8b1bae036e47545df3350591eeaa state=OFFLINE, ts=1490708814241, server=vm240-int,60020,1490708207969} to {199b8b1bae036e47545df3350591eeaa state=PENDING_OPEN, ts=1490708814246, server=10.44.3.244,60020,1490708637722}
/var/log/hbase/hbase-hbase-master-vm240.log:2017-03-28 09:46:54,396 INFO [AM.ZK.Worker-pool3-t38] master.RegionStates: Transition {199b8b1bae036e47545df3350591eeaa state=PENDING_OPEN, ts=1490708814246, server=10.44.3.244,60020,1490708637722} to {199b8b1bae036e47545df3350591eeaa state=OPENING, ts=1490708814396, server=10.44.3.244,60020,1490708637722}
/var/log/hbase/hbase-hbase-master-vm240.log:2017-03-28 09:46:54,894 INFO [AM.ZK.Worker-pool3-t39] master.RegionStates: Transition {199b8b1bae036e47545df3350591eeaa state=OPENING, ts=1490708814396, server=10.44.3.244,60020,1490708637722} to {199b8b1bae036e47545df3350591eeaa state=OPEN, ts=1490708814894, server=10.44.3.244,60020,1490708637722}
/var/log/hbase/hbase-hbase-regionserver-vm240.log:2017-03-28 09:46:54,189 INFO [PriorityRpcServer.handler=2,queue=0,port=60020] regionserver.RSRpcServices: Close 199b8b1bae036e47545df3350591eeaa, moving to 10.44.3.244,60020,1490708637722
/var/log/hbase/hbase-hbase-regionserver-vm240.log:2017-03-28 09:46:54,234 INFO [RS_CLOSE_REGION-vm240-int:60020-2] regionserver.HRegionServer: Adding moved region record: 199b8b1bae036e47545df3350591eeaa to 10.44.3.244,60020,1490708637722 as of 4
/var/log/hbase/SecurityAuth.audit:2017-03-28 09:44:00,455 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.44.3.244 port: 45674 with version info: version: "1.1.2.2.3.4.0-3485" url: "git://c66-slave-20176e25-2/grid/0/jenkins/workspace/HDP-build-centos6/bigtop/build/hbase/rpm/BUILD/hbase-1.1.2.2.3.4.0" revision: "a2b7fd38f4dcca6eb81f99b5d37b2ea3beeef09e" user: "jenkins" date: "Wed Dec 16 03:49:28 UTC 2015" src_checksum: "2272034a01b6bfb7ab2804c1d8be0c67"
/var/log/hbase/SecurityAuth.audit:2017-03-28 09:46:54,863 INFO SecurityLogger.org.apache.hadoop.hbase.Server: Connection from 10.44.3.244 port: 53963 with version info: version: "1.1.2.2.3.4.0-3485" url: "git://c66-slave-20176e25-2/grid/0/jenkins/workspace/HDP-build-centos6/bigtop/build/hbase/rpm/BUILD/hbase-1.1.2.2.3.4.0" revision: "a2b7fd38f4dcca6eb81f99b5d37b2ea3beeef09e" user: "jenkins" date: "Wed Dec 16 03:49:28 UTC 2015" src_checksum: "2272034a01b6bfb7ab2804c1d8be0c67"
root@vm244 ~ # less /var/log/hadoop/hdfs/hadoop-hdfs-datanode-vm244.log
2017-03-28 09:33:18,191 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: vm241-int/10.44.3.241:8020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep
(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-03-28 09:33:18,191 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
java.net.ConnectException: Call From vm244/10.44.3.244 to vm241-int:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apac
he.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor9.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732)
at org.apache.hadoop.ipc.Client.call(Client.java:1431)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy13.sendHeartbeat(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:155)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:551)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:654)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:843)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:612)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:710)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
at org.apache.hadoop.ipc.Client.call(Client.java:1397)
... 8 more
2017-03-28 09:33:18,654 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: vm240-int/10.44.3.240:8020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep
(maxRetries=10, sleepTime=1000 MILLISECONDS)
.
.
.
2017-03-28 09:33:27,198 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: vm241-int/10.44.3.241:8020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-03-28 09:33:27,348 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-1929398724-10.44.3.240-1490704766649 (Datanode Uuid f50fada0-4518-4af3-a9f7-8f7d5f1294b2) service to vm241-int/10.44.3.241:8020 is shutting down
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.UnregisteredNodeException): Unregistered server: DatanodeRegistration(10.44.3.244:50010, datanodeUuid=f50fada0-4518-4af3-a9f7-8f7d5f1294b2, infoPort=50075, infoSecurePort=50475, ipcPort=50020, storageInfo=lv=-56;cid=CID-f13f3676-f02d-4949-b13b-d524ad1bc4a2;nsid=2057042102;c=0)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.verifyRequest(NameNodeRpcServer.java:1476)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.sendHeartbeat(NameNodeRpcServer.java:1361)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.sendHeartbeat(DatanodeProtocolServerSideTranslatorPB.java:114)
at org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:29064)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:421)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2145)
at org.apache.hadoop.ipc.Client.call(Client.java:1427)
at org.apache.hadoop.ipc.Client.call(Client.java:1358)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy13.sendHeartbeat(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:155)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:551)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:654)
at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:843)
at java.lang.Thread.run(Thread.java:745)
2017-03-28 09:33:27,348 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-1929398724-10.44.3.240-1490704766649 (Datanode Uuid f50fada0-4518-4af3-a9f7-8f7d5f1294b2) service to vm241-int/10.44.3.241:8020
2017-03-28 09:33:27,661 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: vm240-int/10.44.3.240:8020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
So, questions: 1. Why does vm244 DataNode from the destroyed cluster send requests to vm241 NameNode from the new cluster? Why do they start to communicate? New NameNode have no information about vm244 in slaves, old DataNode (vm244) have no information about new NameNode (vm241) in masters. How do they establish communication? 2. Who can help to understand from logs what happen with vm244? Was it accepted by new NameNode and HBase-master or it was refused? 3. How can I prevent the connection of such old unbroken nodes to a new cluster?
... View more
Labels:
