Skip to main content
George-Megre
George-Megre
Silver ⭐️⭐️⭐️
November 30, 2023
Question

Encountered exception loading fsimage, NameNode is not formatted

NameNode is crashed when trying to restart it via CM due to: 

INFO Lock on /data/disk1/dfs/nn/in_use.lock acquired by nodename 26396@ip-10-2-0-224.ec2.internal

Warning Encountered exception loading fsimage
java.io.IOException: NameNode is not formatted.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:251)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1166)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:757)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:642)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:713)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:956)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:931)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1666)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1733)

Failed to start namenode.
java.io.IOException: NameNode is not formatted.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:251)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1166)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:757)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:642)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:713)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:956)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:931)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1666)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1733)

 And when i try to sudo -u hdfs hdfs namenode -format on the nn i get this error:

23/11/30 12:21:35 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Running in secure mode, but config doesn't have a keytab
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:306)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1136)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1623)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1733)
23/11/30 12:21:35 INFO util.ExitUtil: Exiting with status 1: java.io.IOException: Running in secure mode, but config doesn't have a keytab
23/11/30 12:21:35 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ip-10-2-0-224.ec2.internal/10.2.0.224

 Please help me get the HDFS NN running

20 replies

Majeti
Gold ⭐️
Majeti
Gold ⭐️
December 8, 2023

Hi @George-Megre ,

 Running a CLI command on Namenode managed by Cloudera manager will be little tricky as you need to set right environment and hadoop config directory path etc. . Let's make it work through CM itself. Please make sure below things are verified.

  1. "/data/disk1/dfs/nn/current/VERSION" file is present with valid content. 
  2. Verify if another old instance of Namenode is already running or not by executing "ps -ef | grep -I namenode" and kill it.

Attach recursive directory listing of /data/disk1/dfs/nn/ along with core-site.xml/hdfs-site.xml if above steps don't help.

Translate
George-Megre
George-MegreAuthor
Silver ⭐️⭐️⭐️
December 11, 2023

Hey @Majeti,

Unfortunately the offered solution didn't help,  VERSION file is present with valid content and there is no Namenode process is running (at all 🤔) in the machine after executing ps command. adding recursive directory listing of /data/disk1/dfs/nn/: 

/data/disk1/dfs$ sudo ls -la nn/current
total 1588
drwx------ 2 root root 4096 Nov 22 10:36 .
drwx------ 3 hdfs hadoop 4096 Dec 11 09:19 ..
-rw-r--r-- 1 root root 191 Nov 22 10:36 VERSION
-rw-r--r-- 1 root root 2894 Nov 22 10:36 edits_0000000000002786579-0000000000002786605
-rw-r--r-- 1 root root 8466 Nov 22 10:36 edits_0000000000002786606-0000000000002786671
-rw-r--r-- 1 root root 8216 Nov 22 10:36 edits_0000000000002786672-0000000000002786743
-rw-r--r-- 1 root root 4767 Nov 22 10:36 edits_0000000000002786744-0000000000002786772
-rw-r--r-- 1 root root 8207 Nov 22 10:36 edits_0000000000002786773-0000000000002786818
-rw-r--r-- 1 root root 1965 Nov 22 10:36 edits_0000000000002786819-0000000000002786837
-rw-r--r-- 1 root root 15365 Nov 22 10:36 edits_0000000000002786838-0000000000002786940
-rw-r--r-- 1 root root 15786 Nov 22 10:36 edits_0000000000002786941-0000000000002787070
-rw-r--r-- 1 root root 2162 Nov 22 10:36 edits_0000000000002787071-0000000000002787088
-rw-r--r-- 1 root root 4940 Nov 22 10:36 edits_0000000000002787089-0000000000002787116
-rw-r--r-- 1 root root 3316 Nov 22 10:36 edits_0000000000002787117-0000000000002787144
-rw-r--r-- 1 root root 7246 Nov 22 10:36 edits_0000000000002787145-0000000000002787187
-rw-r--r-- 1 root root 1646 Nov 22 10:36 edits_0000000000002787188-0000000000002787202
-rw-r--r-- 1 root root 1589 Nov 22 10:36 edits_0000000000002787203-0000000000002787216
-rw-r--r-- 1 root root 4332 Nov 22 10:36 edits_0000000000002787217-0000000000002787239
-rw-r--r-- 1 root root 1587 Nov 22 10:36 edits_0000000000002787240-0000000000002787253
-rw-r--r-- 1 root root 5611 Nov 22 10:36 edits_0000000000002787254-0000000000002787287
-rw-r--r-- 1 root root 1588 Nov 22 10:36 edits_0000000000002787288-0000000000002787301
-rw-r--r-- 1 root root 1251 Nov 22 10:36 edits_0000000000002787302-0000000000002787313
-rw-r--r-- 1 root root 4669 Nov 22 10:36 edits_0000000000002787314-0000000000002787338
-rw-r--r-- 1 root root 1587 Nov 22 10:36 edits_0000000000002787339-0000000000002787352
-rw-r--r-- 1 root root 4330 Nov 22 10:36 edits_0000000000002787353-0000000000002787375
-rw-r--r-- 1 root root 816 Nov 22 10:36 edits_0000000000002787376-0000000000002787383
-rw-r--r-- 1 root root 4330 Nov 22 10:36 edits_0000000000002787384-0000000000002787406
-rw-r--r-- 1 root root 1588 Nov 22 10:36 edits_0000000000002787407-0000000000002787420
-rw-r--r-- 1 root root 1588 Nov 22 10:36 edits_0000000000002787421-0000000000002787434
-rw-r--r-- 1 root root 4331 Nov 22 10:36 edits_0000000000002787435-0000000000002787457
-rw-r--r-- 1 root root 3695 Nov 22 10:36 edits_0000000000002787458-0000000000002787494
-rw-r--r-- 1 root root 13128 Nov 22 10:36 edits_0000000000002787495-0000000000002787575
-rw-r--r-- 1 root root 8425 Nov 22 10:36 edits_0000000000002787576-0000000000002787639
-rw-r--r-- 1 root root 1588 Nov 22 10:36 edits_0000000000002787640-0000000000002787653
-rw-r--r-- 1 root root 4333 Nov 22 10:36 edits_0000000000002787654-0000000000002787676
-rw-r--r-- 1 root root 2808 Nov 22 10:36 edits_0000000000002787677-0000000000002787698
-rw-r--r-- 1 root root 4332 Nov 22 10:36 edits_0000000000002787699-0000000000002787721
-rw-r--r-- 1 root root 2858 Nov 22 10:36 edits_0000000000002787722-0000000000002787744
-rw-r--r-- 1 root root 2855 Nov 22 10:36 edits_0000000000002787745-0000000000002787767
-rw-r--r-- 1 root root 10117 Nov 22 10:36 edits_0000000000002787768-0000000000002787846
-rw-r--r-- 1 root root 9826 Nov 22 10:36 edits_0000000000002787847-0000000000002787916
-rw-r--r-- 1 root root 4940 Nov 22 10:36 edits_0000000000002787917-0000000000002787944
-rw-r--r-- 1 root root 1960 Nov 22 10:36 edits_0000000000002787945-0000000000002787961
-rw-r--r-- 1 root root 813 Nov 22 10:36 edits_0000000000002787962-0000000000002787969
-rw-r--r-- 1 root root 4327 Nov 22 10:36 edits_0000000000002787970-0000000000002787992
-rw-r--r-- 1 root root 1644 Nov 22 10:36 edits_0000000000002787993-0000000000002788007
-rw-r--r-- 1 root root 4334 Nov 22 10:36 edits_0000000000002788008-0000000000002788030
-rw-r--r-- 1 root root 1588 Nov 22 10:36 edits_0000000000002788031-0000000000002788044
-rw-r--r-- 1 root root 1589 Nov 22 10:36 edits_0000000000002788045-0000000000002788058
-rw-r--r-- 1 root root 4331 Nov 22 10:36 edits_0000000000002788059-0000000000002788081
-rw-r--r-- 1 root root 1589 Nov 22 10:36 edits_0000000000002788082-0000000000002788095
-rw-r--r-- 1 root root 4332 Nov 22 10:36 edits_0000000000002788096-0000000000002788118
-rw-r--r-- 1 root root 1587 Nov 22 10:36 edits_0000000000002788119-0000000000002788132
-rw-r--r-- 1 root root 1587 Nov 22 10:36 edits_0000000000002788133-0000000000002788146
-rw-r--r-- 1 root root 17814 Nov 22 10:36 edits_0000000000002788147-0000000000002788278
-rw-r--r-- 1 root root 1584 Nov 22 10:36 edits_0000000000002788279-0000000000002788292
-rw-r--r-- 1 root root 4333 Nov 22 10:36 edits_0000000000002788293-0000000000002788315
-rw-r--r-- 1 root root 1588 Nov 22 10:36 edits_0000000000002788316-0000000000002788329
-rw-r--r-- 1 root root 1584 Nov 22 10:36 edits_0000000000002788330-0000000000002788343
-rw-r--r-- 1 root root 4390 Nov 22 10:36 edits_0000000000002788344-0000000000002788367
-rw-r--r-- 1 root root 1583 Nov 22 10:36 edits_0000000000002788368-0000000000002788381
-rw-r--r-- 1 root root 9140 Nov 22 10:36 edits_0000000000002788382-0000000000002788428
-rw-r--r-- 1 root root 4664 Nov 22 10:36 edits_0000000000002788429-0000000000002788458
-rw-r--r-- 1 root root 1590 Nov 22 10:36 edits_0000000000002788459-0000000000002788472
-rw-r--r-- 1 root root 4332 Nov 22 10:36 edits_0000000000002788473-0000000000002788495
-rw-r--r-- 1 root root 1761 Nov 22 10:36 edits_0000000000002788496-0000000000002788509
-rw-r--r-- 1 root root 4333 Nov 22 10:36 edits_0000000000002788510-0000000000002788532
-rw-r--r-- 1 root root 1586 Nov 22 10:36 edits_0000000000002788533-0000000000002788546
-rw-r--r-- 1 root root 1589 Nov 22 10:36 edits_0000000000002788547-0000000000002788560
-rw-r--r-- 1 root root 816 Nov 22 10:36 edits_0000000000002788561-0000000000002788568
-rw-r--r-- 1 root root 563666 Nov 22 10:36 fsimage_0000000000005559643
-rw-r--r-- 1 root root 62 Nov 22 10:36 fsimage_0000000000005559643.md5
-rw-r--r-- 1 root root 563666 Nov 22 10:36 fsimage_0000000000005560209
-rw-r--r-- 1 root root 62 Nov 22 10:36 fsimage_0000000000005560209.md5
-rw-r--r-- 1 root root 8 Nov 22 10:36 seen_txid

 and  hdfs-site.xml:

cat /etc/hadoop/conf.cloudera.hdfs/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>

<!--Autogenerated by Cloudera Manager-->
<configuration>
<property>
<name>dfs.nameservices</name>
<value>hdfs-cdp7</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.hdfs-cdp7</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled.hdfs-cdp7</name>
<value>true</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>ip-10-2-0-115.ec2.internal:2181,ip-10-2-0-171.ec2.internal:2181,ip-10-2-0-177.ec2.internal:2181</value>
</property>
<property>
<name>dfs.ha.namenodes.hdfs-cdp7</name>
<value>namenode1546334668,namenode1546333642</value>
</property>
<property>
<name>dfs.namenode.rpc-address.hdfs-cdp7.namenode1546334668</name>
<value>ip-10-2-0-224.ec2.internal:8020</value>
</property>
<property>
<name>dfs.namenode.servicerpc-address.hdfs-cdp7.namenode1546334668</name>
<value>ip-10-2-0-224.ec2.internal:8022</value>
</property>
<property>
<name>dfs.namenode.http-address.hdfs-cdp7.namenode1546334668</name>
<value>ip-10-2-0-224.ec2.internal:9870</value>
</property>
<property>
<name>dfs.namenode.https-address.hdfs-cdp7.namenode1546334668</name>
<value>ip-10-2-0-224.ec2.internal:9871</value>
</property>
<property>
<name>dfs.namenode.rpc-address.hdfs-cdp7.namenode1546333642</name>
<value>ip-10-2-0-49.ec2.internal:8020</value>
</property>
<property>
<name>dfs.namenode.servicerpc-address.hdfs-cdp7.namenode1546333642</name>
<value>ip-10-2-0-49.ec2.internal:8022</value>
</property>
<property>
<name>dfs.namenode.http-address.hdfs-cdp7.namenode1546333642</name>
<value>ip-10-2-0-49.ec2.internal:9870</value>
</property>
<property>
<name>dfs.namenode.https-address.hdfs-cdp7.namenode1546333642</name>
<value>ip-10-2-0-49.ec2.internal:9871</value>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
<property>
<name>dfs.blocksize</name>
<value>134217728</value>
</property>
<property>
<name>dfs.client.use.datanode.hostname</name>
<value>false</value>
</property>
<property>
<name>fs.permissions.umask-mode</name>
<value>022</value>
</property>
<property>
<name>dfs.client.block.write.locateFollowingBlock.retries</name>
<value>7</value>
</property>
<property>
<name>dfs.encrypt.data.transfer.algorithm</name>
<value>3des</value>
</property>
<property>
<name>dfs.encrypt.data.transfer.cipher.suites</name>
<value>AES/CTR/NoPadding</value>
</property>
<property>
<name>dfs.encrypt.data.transfer.cipher.key.bitlength</name>
<value>256</value>
</property>
<property>
<name>dfs.namenode.acls.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.client.read.shortcircuit</name>
<value>true</value>
</property>
<property>
<name>dfs.client.read.shortcircuit.streams.cache.size</name>
<value>4096</value>
</property>
<property>
<name>dfs.domain.socket.path</name>
<value>/var/run/hdfs-sockets/dn</value>
</property>
<property>
<name>dfs.client.read.shortcircuit.skip.checksum</name>
<value>false</value>
</property>
<property>
<name>dfs.client.domain.socket.data.traffic</name>
<value>false</value>
</property>
<property>
<name>dfs.datanode.hdfs-blocks-metadata.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/_HOST@CDP7.COM</value>
</property>
<property>
<name>dfs.namenode.kerberos.internal.spnego.principal</name>
<value>HTTP/_HOST@CDP7.COM</value>
</property>
<property>
<name>dfs.datanode.kerberos.principal</name>
<value>hdfs/_HOST@CDP7.COM</value>
</property>
</configuration>

 core-site.xml:

cat /etc/hadoop/conf.cloudera.hdfs/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>

<!--Autogenerated by Cloudera Manager-->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://hdfs-cdp7</value>
</property>
<property>
<name>ipc.client.connection.maxidletime</name>
<value>30000</value>
</property>
<property>
<name>ipc.client.connect.max.retries</name>
<value>50</value>
</property>
<property>
<name>fs.trash.interval</name>
<value>1</value>
</property>
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec</value>
</property>
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
<property>
<name>hadoop.rpc.protection</name>
<value>authentication</value>
</property>
<property>
<name>hadoop.ssl.require.client.cert</name>
<value>false</value>
<final>true</final>
</property>
<property>
<name>hadoop.ssl.keystores.factory.class</name>
<value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
<final>true</final>
</property>
<property>
<name>hadoop.ssl.server.conf</name>
<value>ssl-server.xml</value>
<final>true</final>
</property>
<property>
<name>hadoop.ssl.client.conf</name>
<value>ssl-client.xml</value>
<final>true</final>
</property>
<property>
<name>hadoop.security.auth_to_local</name>
<value>RULE:[2:$1@$0](rangeradmin@CDP7.COM)s/(.*)@CDP7.COM/ranger/
RULE:[2:$1@$0](rangertagsync@CDP7.COM)s/(.*)@CDP7.COM/rangertagsync/
RULE:[2:$1@$0](rangerusersync@CDP7.COM)s/(.*)@CDP7.COM/rangerusersync/
RULE:[1:$1@$0](HTTP.*@CDP7.COM$)s/@CDP7.COM$//
RULE:[1:$1@$0](.*@CDP7.COM$)s/@CDP7.COM$///L
RULE:[2:$1@$0](HTTP.*@CDP7.COM$)s/@CDP7.COM$//
RULE:[2:$1@$0](.*@CDP7.COM$)s/@CDP7.COM$///L
DEFAULT</value>
</property>
<property>
<name>hadoop.proxyuser.oozie.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.oozie.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.HTTP.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.HTTP.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hive.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hive.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hue.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hue.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.httpfs.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.httpfs.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.knox.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.knox.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.livy.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.livy.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.impala.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.impala.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hdfs.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hdfs.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.yarn.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.yarn.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.phoenix.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.phoenix.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.security.group.mapping</name>
<value>org.apache.hadoop.security.ShellBasedUnixGroupsMapping</value>
</property>
<property>
<name>hadoop.security.instrumentation.requires.admin</name>
<value>false</value>
</property>
</configuration>

 Thanks!

Translate
Majeti
Gold ⭐️
Majeti
Gold ⭐️
December 21, 2023

Hi @George-Megre , When namenode itself is down , who is writing into edits_inprogress_0000000000011353525 as of Dec 20th ?

Translate
George-Megre
George-MegreAuthor
Silver ⭐️⭐️⭐️
December 24, 2023

Hey @Majeti ,

So we have 2 NN that sitting on different machines.

One of them is writing to to the other probably.

how can we fix the issue? 🙁

Translate
Majeti
Gold ⭐️
Majeti
Gold ⭐️
December 24, 2023
Then you need to run bootstrap standby on non running Namenode or sync nn
datadir folder from running nn to down nn.
Translate
George-Megre
George-MegreAuthor
Silver ⭐️⭐️⭐️
January 8, 2024

Hello @Majeti ,

1. The previous error no longer appears, but we are now encountering a new error that is showing up in the logs, such as:

Request to collection [ranger_audits] failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry=5 commError=true errorCode=0

failed to log audit event: {"repoType":1,"repo":"cm_hdfs","reqUser":"mapred","evtTime":"2023-07-11 18:09:40.656","access":"listStatus","resource":"/user/history/done/2022/08","resType":"path","action":"read","result":1,"agent":"hdfs","policy":-1,"reason":"/user/history/done/2022/08","enforcer":"hadoop-acl","cliIP":"10.2.0.49","agentHost":"ip-10-2-0-49.ec2.internal","logType":"RangerAudit","id":"792dc152-89cf-425d-978d-b50af625b1fe-0","seq_num":1,"event_count":1,"event_dur_ms":1,"tags":[],"additional_info":"{\"remote-ip-address\":10.2.0.49, \"forwarded-ip-addresses\":[], \"accessTypes\":[execute, read]","cluster_name":"BigID_cdp7"}
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: No live SolrServers available to handle this request:[http://ip-10-2-0-224.ec2.internal:8993/solr/ranger_audits_shard1_replica_n1]
at org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:125)
at org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:46)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.directUpdate(BaseCloudSolrClient.java:559)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1046)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:906)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:971)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:971)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:971)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:971)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:971)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:838)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:213)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85)
at org.apache.ranger.audit.destination.SolrAuditDestination.lambda$addDocsToSolr$0(SolrAuditDestination.java:493)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.ranger.audit.utils.AbstractKerberosUser.doAs(AbstractKerberosUser.java:151)
at org.apache.ranger.audit.utils.KerberosAction.execute(KerberosAction.java:70)
at org.apache.ranger.audit.destination.SolrAuditDestination.addDocsToSolr(SolrAuditDestination.java:499)
at org.apache.ranger.audit.destination.SolrAuditDestination.log(SolrAuditDestination.java:282)
at org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:192)
at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:880)
at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827)
at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757)
at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://ip-10-2-0-224.ec2.internal:8993/solr/ranger_audits_shard1_replica_n1]
at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:345)
at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$directUpdate$0(BaseCloudSolrClient.java:533)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://ip-10-2-0-224.ec2.internal:8993/solr/ranger_audits_shard1_replica_n1
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:672)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:319)
... 6 more
Caused by: org.apache.http.conn.HttpHostConnectException: Connect to ip-10-2-0-224.ec2.internal:8993 [ip-10-2-0-224.ec2.internal/10.2.0.224] failed: Connection refused (Connection refused)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:564)
... 10 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
... 20 more

2. In addition, we have Health Test issue, which might be connected to issue #1 (?):

GeorgeMegre_0-1704720817353.png

We kind of lost, our cluster is not working and we must have it work 🙁🙏

Translate
George-Megre
George-MegreAuthor
Silver ⭐️⭐️⭐️
January 8, 2024

@Majeti can you direct me where is the SBNN and what should i verify there? 

Thanks!

Translate
Majeti
Gold ⭐️
Majeti
Gold ⭐️
January 8, 2024
Hi ,
That's you Standby Namenode (SBNN). Please verify if it's performing
checkpoint ing or not. Please perform one checkpoint from Cloudera manager
to get the health test clear.
Translate
Terms & ConditionsAccessibility statement