astappiev
New Contributor
My Accepted Solutions
Show All

Command 'DecommissionWithWait' failed for service ...

by New Contributor in Support Questions
‎02-25-2020 03:00 AM
‎02-25-2020 03:00 AM
Hi, I'm trying to start maintenance mode on few nodes using "Take DataNode offline" option. But getting this error: Command 'DecommissionWithWait' failed for service 'hbase1'. Can't find anything usefull in logs, only this piece, but I do not understand how to fix it.   stderr ++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' hbase.keytab '*jceks' supervisor_status + find /var/run/cloudera-scm-agent/process/46575-hbase-MASTER-togglebalancer -type f '!' -path '/var/run/cloudera-scm-agent/process/46575-hbase-MASTER-togglebalancer/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name hbase.keytab '!' -name '*jceks' '!' -name supervisor_status -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/46575-hbase-MASTER-togglebalancer#g' '{}' ';' + make_scripts_executable + find /var/run/cloudera-scm-agent/process/46575-hbase-MASTER-togglebalancer -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';' + acquire_kerberos_tgt hbase.keytab + '[' -z hbase.keytab ']' + KERBEROS_PRINCIPAL= + '[' '!' -z '' ']' + '[' -n '' ']' + export 'HBASE_OPTS=-Djava.net.preferIPv4Stack=true ' + HBASE_OPTS='-Djava.net.preferIPv4Stack=true ' + locate_hbase_script + '[' 6 -ge 5 ']' + export BIGTOP_DEFAULTS_DIR= + BIGTOP_DEFAULTS_DIR= + HBASE_BIN=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hbase/../../bin/hbase + '[' upgrade = toggle_balancer ']' + '[' region_mover = toggle_balancer ']' + '[' toggle_balancer = toggle_balancer ']' + SHELL_CMD='balance_switch false' ++ mktemp + tmpfile=/tmp/tmp.sW8v9Fngcq + echo balance_switch false + echo exit + /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hbase/../../bin/hbase --config /var/run/cloudera-scm-agent/process/46575-hbase-MASTER-togglebalancer shell -n /tmp/tmp.sW8v9Fngcq LoadError: no such file to load -- /tmp/tmp.sW8v9Fngcq load at org/jruby/RubyKernel.java:974 <main> at /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hbase/bin/hirb.rb:186 + RET=1 + rm /tmp/tmp.sW8v9Fngcq + exit 1 Best regards, Oleh   ... View more
Labels:

Re: Failed to start namenode. java.io.IOException:...

by New Contributor in Support Questions
‎02-13-2020 12:19 PM
‎02-13-2020 12:19 PM
Solved by copying /dfs/jn folder from master01.ib (one nodes in sync) to master03.ib. ... View more

Failed to start namenode. java.io.IOException: Tim...

by New Contributor in Support Questions
‎02-13-2020 09:54 AM
‎02-13-2020 09:54 AM
Hi,   I'm upgrading CDH from 5.13.0 to 6.3.1 and can not proceed after " Upgrade HDFS Metadata ".   On a 3rd step, after "Starting the JournalNodes." and " Starting metadata upgrade on Active NameNode of nameservice nameservice1. ". I cannot proceed with " Waiting for NameNode (master02) to start responding to RPCs.".   It just freezes, but according to log records, it will not continue because the process failed. And it is failed because of a quorum of JournalNodes is not succeeded. master03.ib (10.12.0.3) is not responding.   What can I do? What can cause the issue? Can I run following steps manually?   The logs says the following 6:43:03.610 PM WARN QuorumJournalManager Waited 55044 ms (timeout=60000 ms) for a response for getJournalCTime. Succeeded so far: [10.12.0.2:8485,10.12.0.1:8485] 6:43:04.611 PM WARN QuorumJournalManager Waited 56045 ms (timeout=60000 ms) for a response for getJournalCTime. Succeeded so far: [10.12.0.2:8485,10.12.0.1:8485] 6:43:05.611 PM WARN QuorumJournalManager Waited 57046 ms (timeout=60000 ms) for a response for getJournalCTime. Succeeded so far: [10.12.0.2:8485,10.12.0.1:8485] 6:43:06.613 PM WARN QuorumJournalManager Waited 58047 ms (timeout=60000 ms) for a response for getJournalCTime. Succeeded so far: [10.12.0.2:8485,10.12.0.1:8485] 6:43:07.614 PM WARN QuorumJournalManager Waited 59048 ms (timeout=60000 ms) for a response for getJournalCTime. Succeeded so far: [10.12.0.2:8485,10.12.0.1:8485] 6:43:08.673 PM INFO FSNamesystem FSNamesystem write lock held for 60244 ms via java.lang.Thread.getStackTrace(Thread.java:1559) org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032) org.apache.hadoop.hdfs.server.namenode.FSNamesystemLock.writeUnlock(FSNamesystemLock.java:263) org.apache.hadoop.hdfs.server.namenode.FSNamesystem.writeUnlock(FSNamesystem.java:1604) org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1111) org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:709) org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:665) org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:727) org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:950) org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:929) org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653) org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720) Number of suppressed write-lock reports: 0 Longest write-lock held interval: 60244 6:43:08.675 PM WARN FSNamesystem Encountered exception loading fsimage java.io.IOException: Timed out waiting for getJournalCTime() response at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.getJournalCTime(QuorumJournalManager.java:678) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.getSharedLogCTime(FSEditLog.java:1613) at org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:829) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:683) at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:443) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:310) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1084) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:709) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:665) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:727) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:950) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:929) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720) 6:43:08.698 PM INFO ContextHandler Stopped o.e.j.w.WebAppContext@52045dbe{/,null,UNAVAILABLE}{/hdfs} 6:43:08.704 PM INFO AbstractConnector Stopped ServerConnector@34997338{HTTP/1.1,[http/1.1]}{master02.ib:9870} 6:43:08.705 PM INFO ContextHandler Stopped o.e.j.s.ServletContextHandler@4d722ac9{/static,file:///opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-hdfs/webapps/static/,UNAVAILABLE} 6:43:08.705 PM INFO ContextHandler Stopped o.e.j.s.ServletContextHandler@2320fa6f{/logs,file:///var/log/hadoop-hdfs/,UNAVAILABLE} 6:43:08.709 PM INFO MetricsSystemImpl Stopping NameNode metrics system... 6:43:08.710 PM INFO MetricsSystemImpl NameNode metrics system stopped. 6:43:08.710 PM INFO MetricsSystemImpl NameNode metrics system shutdown complete. 6:43:08.711 PM ERROR NameNode Failed to start namenode. java.io.IOException: Timed out waiting for getJournalCTime() response at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.getJournalCTime(QuorumJournalManager.java:678) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.getSharedLogCTime(FSEditLog.java:1613) at org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:829) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:683) at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:443) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:310) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1084) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:709) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:665) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:727) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:950) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:929) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1653) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720) 6:43:08.714 PM INFO ExitUtil Exiting with status 1: java.io.IOException: Timed out waiting for getJournalCTime() response 6:43:08.717 PM INFO NameNode SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at master02.ib/10.12.0.2 ************************************************************/ Thanks and best regards, Oleh ... View more
Contact Me
Online
Offline
Last Visited
‎10-22-2020 04:02 PM
Public Statistics
Date Registered ‎02-13-2020 09:45 AM
Last Visited ‎10-22-2020 04:02 PM
Total Messages Posted 3