[root@master03 hdfs]# tail -100 hadoop-hdfs-namenode-master03.sys674.com.log 2017-12-05 21:45:05,832 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 51019 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:06,833 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 52020 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:07,834 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 53021 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:08,835 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 54022 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:09,836 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 55023 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:10,837 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 56024 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:11,838 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 57025 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:12,839 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 58026 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:13,840 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 59027 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:14,841 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 60028 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:15,842 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 61029 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:16,843 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 62030 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:17,844 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 63031 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:18,845 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 64032 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:19,846 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 65033 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:20,847 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 66034 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:21,849 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 67036 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:22,850 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 68037 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:23,851 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 69038 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:24,852 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 70039 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:25,853 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 71041 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:26,855 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 72043 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:27,856 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 73044 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:28,858 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 74045 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:29,859 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 75046 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:30,860 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 76047 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:31,861 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 77048 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:32,862 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 78049 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:33,863 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 79050 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:34,864 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 80051 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:35,865 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 81053 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:36,867 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 82055 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:37,868 INFO client.QuorumJournalManager (QuorumCall.java:waitFor(136)) - Waited 83056 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:38,870 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 84058 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:39,872 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 85059 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:40,873 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 86060 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:41,874 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 87061 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:42,875 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 88062 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:43,876 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 89063 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:44,877 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 90064 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:45,878 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 91066 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:46,880 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 92068 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:47,881 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 93069 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:48,882 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 94070 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:49,884 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 95072 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:50,886 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 96073 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:51,887 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 97074 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:52,888 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 98075 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:53,889 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 99077 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:54,891 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 100078 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:55,892 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 101079 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:56,893 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 102080 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:57,894 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 103081 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:58,895 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 104082 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:45:59,896 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 105083 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:00,898 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 106085 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:01,899 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 107086 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:02,900 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 108088 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:03,901 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 109089 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:04,902 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 110090 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:05,903 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 111091 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:06,905 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 112093 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:07,908 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 113095 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:08,909 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 114096 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:09,910 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 115097 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:10,911 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 116098 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:11,911 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 117099 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:12,912 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 118100 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:13,914 WARN client.QuorumJournalManager (QuorumCall.java:waitFor(134)) - Waited 119101 ms (timeout=120000 ms) for a response for getJournalState(). Succeeded so far: [10.164.27.162:8485] 2017-12-05 21:46:14,814 FATAL namenode.FSEditLog (JournalSet.java:mapJournalsAndReportErrors(398)) - Error: recoverUnfinalizedSegments failed for required journal (JournalAndStream(mgr=QJM to [10.164.28.153:8485, 10.164.28.152:8485, 10.164.27.162:8485], stream=null)) java.io.IOException: Timed out waiting 120000ms for a quorum of nodes to respond. at org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.waitForWriteQuorum(AsyncLoggerSet.java:137) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createNewUniqueEpoch(QuorumJournalManager.java:182) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.recoverUnfinalizedSegments(QuorumJournalManager.java:436) at org.apache.hadoop.hdfs.server.namenode.JournalSet$8.apply(JournalSet.java:624) at org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:393) at org.apache.hadoop.hdfs.server.namenode.JournalSet.recoverUnfinalizedSegments(JournalSet.java:621) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.recoverUnclosedStreams(FSEditLog.java:1521) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:1169) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1936) at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61) at org.apache.hadoop.hdfs.server.namenode.ha.HAState.setStateInternal(HAState.java:64) at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.setState(StandbyState.java:49) at org.apache.hadoop.hdfs.server.namenode.NameNode.transitionToActive(NameNode.java:1798) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.transitionToActive(NameNodeRpcServer.java:1654) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.transitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107) at org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:4460) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2345) 2017-12-05 21:46:14,821 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1 2017-12-05 21:46:14,827 INFO namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at master03.sys674.com/10.164.27.162 ************************************************************/