Hi @George-Megre , Looks like your SBNN is not performing checkpoint. Can you perform one manual checkpoint now and see what's happening on SBNN side?  Verify if Solr server and port are reachable ip-10-2-0-224.ec2.internal:8993. ... View more

Then you need to run bootstrap standby on non running Namenode or sync nn datadir folder from running nn to down nn. ... View more

Then some other Namenode is talking to this journal node process ? Any way you can find other Namenode which is active and also configured to talk to this. ? ... View more

Hi @George-Megre , I dont think HMS can write to edit log. Only Namenodes are allowed to write unless u r using some different plugin to do that. I am not too sure of that use case . Can you please run commands like below to find out who is writing to edit file ? [root@c4265-node3 current]# lsof /data/dfs/nn/current/edits_inprogress_0000000000003796588 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 1133 hdfs 324u REG 0,87 1048576 21715133 /data/dfs/nn/current/edits_inprogress_0000000000003796588 [root@c4265-node3 current]# ps -h 1133 1133 ? Sl 5:18 /usr/java/jdk1.8.0_232-cloudera/bin/java -Dproc_namenode -Dhdfs.audit.logger=INFO,RFAAUDIT -Dsecurity.audit.logger=INFO,RFAS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.conf=/etc/krb5.conf -Xms1073741824 -Xm [root@c4265-node3 current]#   ... View more

Hi @George-Megre , When namenode itself is down , who is writing into edits_inprogress_0000000000011353525 as of Dec 20th ? ... View more

Hi @michalLi ,  I have been trying this in CDP PvC now but does not seem to work . Here is the behavior i see for spark history server web ui (7.1.7.2000) TLS enabled and kerberos enabled : without keytab https://172.25.42.2:18088 works fine TLS disabled and kerberos enabled : with/without keytab http://172.25.42.2:18088 is failing for 401 Auth in Mac OS/Chrome   ... View more

Hi @George-Megre , Looks like there is lot of mismatch . I could see below edit file being updated by other Namenode or some other Namenode.  -rw-r--r-- 1 hdfs hdfs 17825792 Dec 20 12:21 edits_inprogress_0000000000011353525 Could you please verify and make sure no other NN is using these JNs . I would request you to attach both Namenodes' data directories along with all 3 JN's data directories. ... View more

Sorry @George-Megre , I was on vacation. I am back now. If you can provide me the listing of both NN data dir and JN directories, I can review edit logs and suggest you quickly. ... View more

Hi @George-Megre ,  Looks like the ownership of the /data/disk1/dfs/nn/ is showing correct as hdfs:hadoop but /data/disk1/dfs/nn/current onwards shows as root:root. That might be cause as well. Please run the command "chown -R hdfs:hdfs /data/disk1/dfs/nn/current" and try to start the Namenode process. ... View more

Hi @George-Megre ,  Running a CLI command on Namenode managed by Cloudera manager will be little tricky as you need to set right environment and hadoop config directory path etc. . Let's make it work through CM itself. Please make sure below things are verified. "/data/disk1/dfs/nn/current/VERSION" file is present with valid content.  Verify if another old instance of Namenode is already running or not by executing "ps -ef | grep -I namenode" and kill it. Attach recursive directory listing of /data/disk1/dfs/nn/ along with core-site.xml/hdfs-site.xml if above steps don't help. ... View more