2017-08-25 16:10:15,125 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 17013 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:16,125 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 18014 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:17,127 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 19015 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:18,114 WARN namenode.FSEditLog (JournalSet.java:selectInputStreams(280)) - Unable to determine input streams from QJM to [172.16.0.76:8485, 172.16.0.77:8485, 172.16.0.75:8485]. Skipping. java.io.IOException: Timed out waiting 20000ms for a quorum of nodes to respond. at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:471) at org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:278) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1528) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1552) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:652) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1015) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:690) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:688) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:752) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:992) at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:976) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1686) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1754) 2017-08-25 16:10:18,115 INFO namenode.FSImage (FSImage.java:loadFSImage(669)) - No edit log streams selected. 2017-08-25 16:10:18,116 INFO namenode.FSImage (FSImage.java:loadFSImageFile(736)) - Planning to load image: FSImageFile(file=/data/hadoop/hdfs/namenode/current/fsimage_0000000000010557071, cpktTxId=0000000000010557071) 2017-08-25 16:10:18,158 INFO namenode.FSImageFormatPBINode (FSImageFormatPBINode.java:loadINodeSection(255)) - Loading 19841 INodes. 2017-08-25 16:10:18,429 INFO namenode.FSImageFormatProtobuf (FSImageFormatProtobuf.java:load(184)) - Loaded FSImage in 0 seconds. 2017-08-25 16:10:18,429 INFO namenode.FSImage (FSImage.java:loadFSImage(976)) - Loaded image for txid 10557071 from /data/hadoop/hdfs/namenode/current/fsimage_0000000000010557071 2017-08-25 16:10:18,463 INFO namenode.FSNamesystem (FSNamesystem.java:loadFSImage(1022)) - Need to save fs image? false (staleImage=true, haEnabled=true, isRollingUpgrade=false) 2017-08-25 16:10:18,463 INFO namenode.NameCache (NameCache.java:initialized(143)) - initialized with 282 entries 3975 lookups 2017-08-25 16:10:18,464 INFO namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(697)) - Finished loading FSImage in 20721 msecs 2017-08-25 16:10:18,649 INFO namenode.NameNode (NameNodeRpcServer.java:(422)) - RPC server is binding to chtcuxhd07:8020 2017-08-25 16:10:18,654 INFO ipc.CallQueueManager (CallQueueManager.java:(75)) - Using callQueue: class java.util.concurrent.LinkedBlockingQueue scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler 2017-08-25 16:10:18,666 INFO ipc.Server (Server.java:run(811)) - Starting Socket Reader #1 for port 8020 2017-08-25 16:10:18,709 INFO namenode.FSNamesystem (FSNamesystem.java:registerMBean(6289)) - Registered FSNamesystemState MBean 2017-08-25 16:10:18,710 WARN common.Util (Util.java:stringAsURI(56)) - Path /data/hadoop/hdfs/namenode should be specified as a URI in configuration files. Please update hdfs configuration. 2017-08-25 16:10:18,720 INFO namenode.LeaseManager (LeaseManager.java:getNumUnderConstructionBlocks(138)) - Number of blocks under construction: 3 2017-08-25 16:10:18,720 INFO hdfs.StateChange (FSNamesystem.java:reportStatus(5724)) - STATE* Safe mode ON. The reported blocks 0 needs additional 25942 blocks to reach the threshold 0.9900 of total blocks 26204. The number of live datanodes 0 has reached the minimum number 0. Safe mode will be turned off automatically once the thresholds have been reached. 2017-08-25 16:10:18,727 INFO blockmanagement.DatanodeDescriptor (DatanodeDescriptor.java:updateHeartbeatState(451)) - Number of failed storage changes from 0 to 0 2017-08-25 16:10:18,728 INFO block.BlockTokenSecretManager (BlockTokenSecretManager.java:updateKeys(222)) - Updating block keys 2017-08-25 16:10:18,754 INFO ipc.Server (Server.java:run(1045)) - IPC Server Responder: starting 2017-08-25 16:10:18,755 INFO ipc.Server (Server.java:run(881)) - IPC Server listener on 8020: starting 2017-08-25 16:10:18,790 INFO namenode.NameNode (NameNode.java:startCommonServices(876)) - NameNode RPC up at: chtcuxhd07/172.16.0.76:8020 2017-08-25 16:10:18,791 INFO namenode.FSNamesystem (FSNamesystem.java:startStandbyServices(1286)) - Starting services required for standby state 2017-08-25 16:10:18,867 INFO ha.EditLogTailer (EditLogTailer.java:(117)) - Will roll logs on active node at chtcuxhd08/172.16.0.77:8020 every 120 seconds. 2017-08-25 16:10:18,882 INFO ha.StandbyCheckpointer (StandbyCheckpointer.java:start(129)) - Starting standby checkpoint thread... Checkpointing active NN at http://chtcuxhd08:50070 Serving checkpoints at http://chtcuxhd07:50070 2017-08-25 16:10:24,885 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 6001 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:25,886 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 7002 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:26,887 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 8004 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:27,889 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 9005 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:28,890 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 10006 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:29,891 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 11007 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:30,892 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 12009 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:31,894 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 13010 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:32,895 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 14011 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:33,896 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 15012 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:34,899 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 16016 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:35,901 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 17017 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:36,902 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 18018 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:37,903 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 19019 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2017-08-25 16:10:38,885 WARN namenode.FSEditLog (JournalSet.java:selectInputStreams(280)) - Unable to determine input streams from QJM to [172.16.0.76:8485, 172.16.0.77:8485, 172.16.0.75:8485]. Skipping. java.io.IOException: Timed out waiting 20000ms for a quorum of nodes to respond. at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:471) at org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:278) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1528) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1552) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:214) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:331) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:284) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:301) at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:449) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:297) 2017-08-25 16:10:38,892 INFO namenode.FSNamesystem (FSNamesystem.java:stopStandbyServices(1329)) - Stopping services started for standby state 2017-08-25 16:10:38,896 WARN ha.EditLogTailer (EditLogTailer.java:doWork(349)) - Edit log tailer interrupted java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:347) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$200(EditLogTailer.java:284) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:301) at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:449) at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:297) 2017-08-25 16:10:39,005 INFO namenode.FSNamesystem (FSNamesystem.java:startActiveServices(1130)) - Starting services required for active state 2017-08-25 16:10:39,016 INFO client.QuorumJournalManager (QuorumJournalManager.java:recoverUnfinalizedSegments(435)) - Starting recovery process for unclosed journal segments... 2017-08-25 16:11:15,018 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 36002 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:16,020 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 37003 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:17,021 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 38004 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:18,021 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 39005 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:19,022 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 40006 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:20,024 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 41007 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:21,025 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 42008 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:22,026 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 43009 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:23,027 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 44010 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:24,027 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 45011 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:25,029 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 46012 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:26,030 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 47013 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:27,031 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 48014 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:28,032 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 49015 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:29,033 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 50016 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:30,034 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 51017 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:31,035 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 52019 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:32,037 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 53020 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:33,038 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 54021 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:34,038 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 55022 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:35,040 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 56023 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:36,041 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 57024 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:37,042 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 58026 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:38,044 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 59027 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:39,045 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 60028 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:40,045 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 61029 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:41,047 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 62030 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:42,048 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 63031 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:43,049 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 64033 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:44,051 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 65034 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:45,052 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 66035 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:46,052 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 67036 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:47,054 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 68037 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:48,055 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 69038 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:49,056 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 70040 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:50,058 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 71041 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:51,059 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 72042 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:52,060 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 73044 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:53,062 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 74045 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:54,063 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 75046 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:55,064 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 76048 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:56,066 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 77049 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:57,067 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 78050 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:58,067 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 79051 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:11:59,069 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 80052 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:00,070 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 81053 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:01,071 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 82055 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:02,073 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 83056 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:03,074 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 84057 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:04,075 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 85058 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:05,076 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 86059 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:06,076 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 87060 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:07,078 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 88061 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:08,079 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 89062 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:09,079 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 90063 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:10,081 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 91064 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:11,082 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 92065 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:12,082 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 93066 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:13,084 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 94067 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:14,085 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 95068 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:15,087 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 96070 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:16,088 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 97071 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:17,089 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 98072 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:18,089 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 99073 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:19,091 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 100074 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:20,092 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 101075 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:21,093 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 102076 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:22,094 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 103077 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:23,094 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 104078 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:24,096 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 105079 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:25,097 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 106080 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:26,097 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 107081 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:27,099 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 108082 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:28,100 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 109083 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:29,100 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 110084 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:30,102 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 111085 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:31,103 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 112086 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:32,104 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 113088 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:33,106 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 114089 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:34,107 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 115090 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:35,108 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 116091 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:36,110 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 117093 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:37,110 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 118094 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:38,112 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 119095 ms (timeout=120000 ms) for a response for getJournalState(). No responses yet. 2017-08-25 16:12:39,018 FATAL namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(398)) - Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [172.16.0.76:8485, 172.16.0.77:8485, 172.16.0.75:8485], stream=null)) java.io.IOException: Timed out waiting 120000ms for a quorum of nodes to respond. at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createNewUniqueEpoch(QuorumJournalManager.java:182) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.recoverUnfinalizedSegments(QuorumJournalManager.java:436) at org.apache.hadoop.hdfs.server.namenode.JournalSet$8.apply(JournalSet.java:624) at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393) at org.apache.hadoop.hdfs.server.namenode.JournalSet.recoverUnfinalizedSegments(JournalSet.java:621) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.recoverUnclosedStreams(FSEditLog.java:1459) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:1139) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1915) at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61) at org.apache.hadoop.hdfs.server.namenode.ha.HAState.setStateInternal(HAState.java:64) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.setState(StandbyState.java:49) at org.apache.hadoop.hdfs.server.namenode.NameNode.transitionToActive(NameNode.java:1783) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.transitionToActive(NameNodeRpcServer.java:1631) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.transitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107) at org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:4460) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) 2017-08-25 16:12:39,020 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 2017-08-25 16:12:39,022 INFO namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at chtcuxhd07/172.16.0.76 ************************************************************/