second is it safe to delete the folder - hdfs dfs -rm -r /tmp/hive/hive ... View more

@dear jay - what is the meaning of hive.scratchdir.lock when is set to false? ... View more

from - https://stackoverflow.com/questions/44235019/delete-files-older-than-10days-on-hdfs ... View more

we try the following to remove files that older then 10 days , but because there are so many files then files not deleted at all hdfs dfs -ls /tmp/hive/hive   |   tr -s " "    |    cut -d' ' -f6-8    |     grep "^[0-9]"    |    awk 'BEGIN{ MIN=14400; LAST=60*MIN; "date +%s" | getline NOW } { cmd="date -d'\''"$1" "$2"'\'' +%s"; cmd | getline WHEN; DIFF=NOW-WHEN; if(DIFF > LAST){ print "Deleting: "$3; system("hdfs dfs -rm -r "$3) }}' ... View more

also I cant start the journal node ( on the bade namenode ) 2019-06-16 05:29:39,734 WARN  namenode.FSImage (EditLogFileInputStream.java:scanEditLog(359)) - Caught exception after scanning through 0 ops from /hadoop/hdfs/journal/hdfsha/current/edits_inprogress_0000000000018783114 while determining its valid length. Position was 1032192 java.io.IOException: Can't scan a pre-transactional edit log.         at org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$LegacyReader.scanOp(FSEditLogOp.java:4974)         at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.scanNextOp(EditLogFileInputStream.java:245)         at org.apache.hadoop.hdfs.server.namenode.EditLogFileInputStream.scanEditLog(EditLogFileInputStream.java:355)         at org.apache.hadoop.hdfs.server.namenode.FileJournalManager$EditLogFile.scanLog(FileJournalManager.java:551)         at org.apache.hadoop.hdfs.qjournal.server.Journal.scanStorageForLatestEdits(Journal.java:192)         at org.apache.hadoop.hdfs.qjournal.server.Journal.<init>(Journal.java:152)         at org.apache.hadoop.hdfs.qjournal.server.JournalNode.getOrCreateJournal(JournalNode.java:90)         at org.apache.hadoop.hdfs.qjournal.server.JournalNode.getOrCreateJournal(JournalNode.java:99)         at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.getJournalState(JournalNodeRpcServer.java:127)         at org.apache.hadoop.hdfs.qjournal.protocolPB.QJournalProtocolServerSideTranslatorPB.getJournalState(QJournalProtocolServerSideTranslatorPB.java:118)         at org.apache.hadoop.hdfs.qjournal.protocol.QJournalProtocolProtos$QJournalProtocolService$2.callBlockingMethod(QJournalProtocolProtos.java:25415)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)         at java.security.AccessController.doPrivileged(Native Method)         at javax.security.auth.Subject.doAs(Subject.java:422)         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347) 2019-06-16 05:29:39,734 WARN  namenode.FSImage (EditLogFileInputStream.java:scanEditLog(364)) - After resync, position is 1032192 ... View more

@Geoffrey Shelton Okot no both namenode started as standby and then the namenode stooped ... View more

so according to this info do you recommended to set the value as is ( 31 ) or other ? ... View more

on the good name node we have the number 31 on the bad name node we have also the number 31 on the other journal node we have the number 28 ... View more