@Majeti can you direct me where is the SBNN and what should i verify there?  Thanks! ... View more

Hello @Majeti , 1. The previous error no longer appears, but we are now encountering a new error that is showing up in the logs, such as: Request to collection [ranger_audits] failed due to (0) java.net.ConnectException: Connection refused (Connection refused), retry=5 commError=true errorCode=0 failed to log audit event: {"repoType":1,"repo":"cm_hdfs","reqUser":"mapred","evtTime":"2023-07-11 18:09:40.656","access":"listStatus","resource":"/user/history/done/2022/08","resType":"path","action":"read","result":1,"agent":"hdfs","policy":-1,"reason":"/user/history/done/2022/08","enforcer":"hadoop-acl","cliIP":"10.2.0.49","agentHost":"ip-10-2-0-49.ec2.internal","logType":"RangerAudit","id":"792dc152-89cf-425d-978d-b50af625b1fe-0","seq_num":1,"event_count":1,"event_dur_ms":1,"tags":[],"additional_info":"{\"remote-ip-address\":10.2.0.49, \"forwarded-ip-addresses\":[], \"accessTypes\":[execute, read]","cluster_name":"BigID_cdp7"} org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: No live SolrServers available to handle this request:[http://ip-10-2-0-224.ec2.internal:8993/solr/ranger_audits_shard1_replica_n1] at org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:125) at org.apache.solr.client.solrj.impl.CloudSolrClient.getRouteException(CloudSolrClient.java:46) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.directUpdate(BaseCloudSolrClient.java:559) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1046) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:906) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:971) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:971) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:971) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:971) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:971) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:838) at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:213) at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106) at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71) at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:85) at org.apache.ranger.audit.destination.SolrAuditDestination.lambda$addDocsToSolr$0(SolrAuditDestination.java:493) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.ranger.audit.utils.AbstractKerberosUser.doAs(AbstractKerberosUser.java:151) at org.apache.ranger.audit.utils.KerberosAction.execute(KerberosAction.java:70) at org.apache.ranger.audit.destination.SolrAuditDestination.addDocsToSolr(SolrAuditDestination.java:499) at org.apache.ranger.audit.destination.SolrAuditDestination.log(SolrAuditDestination.java:282) at org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:192) at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:880) at org.apache.ranger.audit.queue.AuditFileSpool.runLogAudit(AuditFileSpool.java:827) at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:757) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://ip-10-2-0-224.ec2.internal:8993/solr/ranger_audits_shard1_replica_n1] at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:345) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.lambda$directUpdate$0(BaseCloudSolrClient.java:533) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more Caused by: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://ip-10-2-0-224.ec2.internal:8993/solr/ranger_audits_shard1_replica_n1 at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:672) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:265) at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248) at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368) at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:319) ... 6 more Caused by: org.apache.http.conn.HttpHostConnectException: Connect to ip-10-2-0-224.ec2.internal:8993 [ip-10-2-0-224.ec2.internal/10.2.0.224] failed: Connection refused (Connection refused) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:564) ... 10 more Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 20 more 2. In addition, we have Health Test issue, which might be connected to issue #1 (?): We kind of lost, our cluster is not working and we must have it work 🙁🙏 ... View more

Hey @Majeti , So we have 2 NN that sitting on different machines. One of them is writing to to the other probably. how can we fix the issue? 🙁 ... View more

Hey @Majeti , /data/disk1/dfs$ sudo lsof jn/hdfs-cdp7/current/edits_inprogress_0000000000011353525 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 26394 hdfs 295u REG 259,1 25165824 14417971 jn/hdfs-cdp7/current/edits_inprogress_0000000000011353525   /data/disk1/dfs$ sudo ps -h 26394 26394 ? Sl 41:01 /usr/lib/jvm/java-8-openjdk/bin/java -Dproc_journalnode -Djava.net.preferIPv4Stack=true -Xms536870912 -Xmx536870912 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/hdfs_hdfs-JOURNALNODE-4bdfe38e85923ad06e3472fcb152cc39_pid26394.hprof -XX:O Thanks.  ... View more

Hey @Majeti , we have Hive Metastore server that looks like probably writing, but this Hive is sitting in other machine that connected to our master NN. both, machines which are the masters have bad Health: Bad : NameNode summary: ip-10-2-X-X.ec2.internal (Availability: Active, Health: Bad), ip-10-2-X-X.ec2.internal (Availability: Unknown, Health: Bad). This health test reflects the health of the active NameNode. ... View more

Hey @Majeti , We have only 1 JN directory: /data/disk1/dfs$ ls jn nn which it's content was sent above  ... View more

Hey @Majeti , thanks for your comment, regrading listing of /data/disk1/dfs/nn/ you can find it above in my comments. here is  listing of /data/disk1/dfs/jn: /data/disk1/dfs$ sudo ls jn/hdfs-cdp7/current -rw-r--r-- 1 hdfs hdfs 7793 Jun 12 2022 edits_0000000000005559461-0000000000005559499 -rw-r--r-- 1 hdfs hdfs 1583 Jun 12 2022 edits_0000000000005559500-0000000000005559513 -rw-r--r-- 1 hdfs hdfs 2067 Jun 12 2022 edits_0000000000005559514-0000000000005559530 -rw-r--r-- 1 hdfs hdfs 4332 Jun 12 2022 edits_0000000000005559531-0000000000005559553 -rw-r--r-- 1 hdfs hdfs 4664 Jun 12 2022 edits_0000000000005559554-0000000000005559583 -rw-r--r-- 1 hdfs hdfs 1776 Jun 12 2022 edits_0000000000005559584-0000000000005559599 -rw-r--r-- 1 hdfs hdfs 4941 Jun 12 2022 edits_0000000000005559600-0000000000005559627 -rw-r--r-- 1 hdfs hdfs 1919 Jun 12 2022 edits_0000000000005559628-0000000000005559643 -rw-r--r-- 1 hdfs hdfs 4329 Jun 12 2022 edits_0000000000005559644-0000000000005559666 -rw-r--r-- 1 hdfs hdfs 1589 Jun 12 2022 edits_0000000000005559667-0000000000005559680 -rw-r--r-- 1 hdfs hdfs 815 Jun 12 2022 edits_0000000000005559681-0000000000005559688 -rw-r--r-- 1 hdfs hdfs 5106 Jun 12 2022 edits_0000000000005559689-0000000000005559717 -rw-r--r-- 1 hdfs hdfs 816 Jun 12 2022 edits_0000000000005559718-0000000000005559725 -rw-r--r-- 1 hdfs hdfs 4331 Jun 12 2022 edits_0000000000005559726-0000000000005559748 -rw-r--r-- 1 hdfs hdfs 1590 Jun 12 2022 edits_0000000000005559749-0000000000005559762 -rw-r--r-- 1 hdfs hdfs 1588 Jun 12 2022 edits_0000000000005559763-0000000000005559776 -rw-r--r-- 1 hdfs hdfs 4332 Jun 12 2022 edits_0000000000005559777-0000000000005559799 -rw-r--r-- 1 hdfs hdfs 1585 Jun 12 2022 edits_0000000000005559800-0000000000005559813 -rw-r--r-- 1 hdfs hdfs 4331 Jun 12 2022 edits_0000000000005559814-0000000000005559836 -rw-r--r-- 1 hdfs hdfs 1582 Jun 12 2022 edits_0000000000005559837-0000000000005559850 -rw-r--r-- 1 hdfs hdfs 1589 Jun 12 2022 edits_0000000000005559851-0000000000005559864 -rw-r--r-- 1 hdfs hdfs 4332 Jun 12 2022 edits_0000000000005559865-0000000000005559887 -rw-r--r-- 1 hdfs hdfs 1589 Jun 12 2022 edits_0000000000005559888-0000000000005559901 -rw-r--r-- 1 hdfs hdfs 4333 Jun 12 2022 edits_0000000000005559902-0000000000005559924 -rw-r--r-- 1 hdfs hdfs 1587 Jun 12 2022 edits_0000000000005559925-0000000000005559938 -rw-r--r-- 1 hdfs hdfs 1588 Jun 12 2022 edits_0000000000005559939-0000000000005559952 -rw-r--r-- 1 hdfs hdfs 4332 Jun 12 2022 edits_0000000000005559953-0000000000005559975 -rw-r--r-- 1 hdfs hdfs 1584 Jun 12 2022 edits_0000000000005559976-0000000000005559989 -rw-r--r-- 1 hdfs hdfs 4187 Jun 12 2022 edits_0000000000005559990-0000000000005560012 -rw-r--r-- 1 hdfs hdfs 1589 Jun 12 2022 edits_0000000000005560013-0000000000005560026 -rw-r--r-- 1 hdfs hdfs 5052 Jun 12 2022 edits_0000000000005560027-0000000000005560056 -rw-r--r-- 1 hdfs hdfs 4811 Jun 12 2022 edits_0000000000005560057-0000000000005560082 -rw-r--r-- 1 hdfs hdfs 1587 Jun 12 2022 edits_0000000000005560083-0000000000005560096 -rw-r--r-- 1 hdfs hdfs 7407 Jun 12 2022 edits_0000000000005560097-0000000000005560135 -rw-r--r-- 1 hdfs hdfs 1585 Jun 12 2022 edits_0000000000005560136-0000000000005560149 -rw-r--r-- 1 hdfs hdfs 1776 Jun 12 2022 edits_0000000000005560150-0000000000005560165 -rw-r--r-- 1 hdfs hdfs 5272 Jun 12 2022 edits_0000000000005560166-0000000000005560195 -rw-r--r-- 1 hdfs hdfs 1586 Jun 12 2022 edits_0000000000005560196-0000000000005560209 -rw-r--r-- 1 hdfs hdfs 4331 Jun 12 2022 edits_0000000000005560210-0000000000005560232 -rw-r--r-- 1 hdfs hdfs 1588 Jun 12 2022 edits_0000000000005560233-0000000000005560246 -rw-r--r-- 1 hdfs hdfs 1590 Jun 12 2022 edits_0000000000005560247-0000000000005560260 -rw-r--r-- 1 hdfs hdfs 4330 Jun 12 2022 edits_0000000000005560261-0000000000005560283 -rw-r--r-- 1 hdfs hdfs 1587 Jun 12 2022 edits_0000000000005560284-0000000000005560297 -rw-r--r-- 1 hdfs hdfs 4334 Jun 12 2022 edits_0000000000005560298-0000000000005560320 -rw-r--r-- 1 hdfs hdfs 1587 Jun 12 2022 edits_0000000000005560321-0000000000005560334 -rw-r--r-- 1 hdfs hdfs 1589 Jun 12 2022 edits_0000000000005560335-0000000000005560348 -rw-r--r-- 1 hdfs hdfs 4332 Jun 12 2022 edits_0000000000005560349-0000000000005560371 -rw-r--r-- 1 hdfs hdfs 1587 Jun 12 2022 edits_0000000000005560372-0000000000005560385 -rw-r--r-- 1 hdfs hdfs 4330 Jun 12 2022 edits_0000000000005560386-0000000000005560408 -rw-r--r-- 1 hdfs hdfs 1588 Jun 12 2022 edits_0000000000005560409-0000000000005560422 -rw-r--r-- 1 hdfs hdfs 814 Jun 12 2022 edits_0000000000005560423-0000000000005560430 -rw-r--r-- 1 hdfs hdfs 4328 Jun 12 2022 edits_0000000000005560431-0000000000005560453 -rw-r--r-- 1 hdfs hdfs 1587 Jun 12 2022 edits_0000000000005560454-0000000000005560467 -rw-r--r-- 1 hdfs hdfs 4329 Jun 12 2022 edits_0000000000005560468-0000000000005560490 -rw-r--r-- 1 hdfs hdfs 1048576 Jul 5 2022 edits_0000000000005880343-0000000000005880373 -rw-r--r-- 1 hdfs hdfs 1048576 Jul 5 2022 edits_0000000000005880374-0000000000005880666 -rw-r--r-- 1 hdfs hdfs 1048576 Jul 5 2022 edits_0000000000005880667-0000000000005880970 -rw-r--r-- 1 hdfs hdfs 1048576 Jul 5 2022 edits_0000000000005880971-0000000000005881003 -rw-r--r-- 1 hdfs hdfs 77594624 Aug 11 2022 edits_0000000000005881004-0000000000006333331 -rw-r--r-- 1 hdfs hdfs 6291456 Aug 14 2022 edits_0000000000006333332-0000000000006368408 -rw-r--r-- 1 hdfs hdfs 99919013 Sep 8 2022 edits_0000000000006368409-0000000000006868412 -rw-r--r-- 1 hdfs hdfs 76350515 Oct 22 2022 edits_0000000000006868413-0000000000007368417 -rw-r--r-- 1 hdfs hdfs 67517391 Nov 2 2022 edits_0000000000007368418-0000000000007868427 -rw-r--r-- 1 hdfs hdfs 65267739 Dec 15 2022 edits_0000000000007868428-0000000000008369431 -rw-r--r-- 1 hdfs hdfs 64536361 Feb 7 2023 edits_0000000000008369432-0000000000008869452 -rw-r--r-- 1 hdfs hdfs 58720256 Apr 2 2023 edits_0000000000008869453-0000000000009321288 -rw-r--r-- 1 hdfs hdfs 64295590 Jul 7 21:15 edits_0000000000009321289-0000000000009821291 -rw-r--r-- 1 hdfs hdfs 4194304 Jul 11 15:17 edits_0000000000009821292-0000000000009852912 -rw-r--r-- 1 hdfs hdfs 70512684 Aug 29 03:24 edits_0000000000009852913-0000000000010352938 -rw-r--r-- 1 hdfs hdfs 66622409 Oct 23 07:09 edits_0000000000010352939-0000000000010852967 -rw-r--r-- 1 hdfs hdfs 28311552 Nov 14 15:42 edits_0000000000010852968-0000000000011059775 -rw-r--r-- 1 hdfs hdfs 14680064 Nov 22 11:53 edits_0000000000011059776-0000000000011140505 -rw-r--r-- 1 hdfs hdfs 1048576 Nov 22 11:59 edits_0000000000011140506-0000000000011140559 -rw-r--r-- 1 hdfs hdfs 1048576 Nov 22 12:01 edits_0000000000011140560-0000000000011140589 -rw-r--r-- 1 hdfs hdfs 3145728 Nov 23 16:17 edits_0000000000011140590-0000000000011153933 -rw-r--r-- 1 hdfs hdfs 11534336 Nov 29 09:32 edits_0000000000011153934-0000000000011218580 -rw-r--r-- 1 hdfs hdfs 1048576 Nov 29 10:21 edits_0000000000011218581-0000000000011219009 -rw-r--r-- 1 hdfs hdfs 1048576 Nov 29 12:00 edits_0000000000011219010-0000000000011219781 -rw-r--r-- 1 hdfs hdfs 23068672 Dec 11 09:13 edits_0000000000011219782-0000000000011353482 -rw-r--r-- 1 hdfs hdfs 1048576 Dec 11 09:17 edits_0000000000011353483-0000000000011353524 -rw-r--r-- 1 hdfs hdfs 1048576 Jun 12 2022 edits_inprogress_0000000000005560491 -rw-r--r-- 1 hdfs hdfs 17825792 Dec 20 12:21 edits_inprogress_0000000000011353525 -rw-r--r-- 1 hdfs hdfs 3 Dec 11 09:19 last-promised-epoch -rw-r--r-- 1 hdfs hdfs 3 Dec 11 09:19 last-writer-epoch drwxr-xr-x 2 hdfs hdfs 4096 Dec 11 09:19 paxos ... View more

Hello @Majeti , we urgently need your assistance. Our environment is currently down. Could you please respond at your earliest convenience? Thank you! 🙏 ... View more

Hey @Majeti , After executing the 'chown' command on the current directory, the previous error disappeared; however, a new error has emerged:  Encountered exception loading fsimage java.io.IOException: There appears to be a gap in the edit log. We expected txid 5560210, but got txid 5687613. at org.apache.hadoop.hdfs.server.namenode.MetaRecoveryContext.editLogLoaderPrompt(MetaRecoveryContext.java:95) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:268) at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:182) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:912) at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:760) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:337) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1166) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:757) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:642) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:713) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:956) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:931) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1666) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1733)  Thanks! ... View more

Hey @Majeti, Unfortunately the offered solution didn't help,  VERSION file is present with valid content and there is no Namenode process is running (at all 🤔) in the machine after executing ps command. adding recursive directory listing of /data/disk1/dfs/nn/:  /data/disk1/dfs$ sudo ls -la nn/current total 1588 drwx------ 2 root root 4096 Nov 22 10:36 . drwx------ 3 hdfs hadoop 4096 Dec 11 09:19 .. -rw-r--r-- 1 root root 191 Nov 22 10:36 VERSION -rw-r--r-- 1 root root 2894 Nov 22 10:36 edits_0000000000002786579-0000000000002786605 -rw-r--r-- 1 root root 8466 Nov 22 10:36 edits_0000000000002786606-0000000000002786671 -rw-r--r-- 1 root root 8216 Nov 22 10:36 edits_0000000000002786672-0000000000002786743 -rw-r--r-- 1 root root 4767 Nov 22 10:36 edits_0000000000002786744-0000000000002786772 -rw-r--r-- 1 root root 8207 Nov 22 10:36 edits_0000000000002786773-0000000000002786818 -rw-r--r-- 1 root root 1965 Nov 22 10:36 edits_0000000000002786819-0000000000002786837 -rw-r--r-- 1 root root 15365 Nov 22 10:36 edits_0000000000002786838-0000000000002786940 -rw-r--r-- 1 root root 15786 Nov 22 10:36 edits_0000000000002786941-0000000000002787070 -rw-r--r-- 1 root root 2162 Nov 22 10:36 edits_0000000000002787071-0000000000002787088 -rw-r--r-- 1 root root 4940 Nov 22 10:36 edits_0000000000002787089-0000000000002787116 -rw-r--r-- 1 root root 3316 Nov 22 10:36 edits_0000000000002787117-0000000000002787144 -rw-r--r-- 1 root root 7246 Nov 22 10:36 edits_0000000000002787145-0000000000002787187 -rw-r--r-- 1 root root 1646 Nov 22 10:36 edits_0000000000002787188-0000000000002787202 -rw-r--r-- 1 root root 1589 Nov 22 10:36 edits_0000000000002787203-0000000000002787216 -rw-r--r-- 1 root root 4332 Nov 22 10:36 edits_0000000000002787217-0000000000002787239 -rw-r--r-- 1 root root 1587 Nov 22 10:36 edits_0000000000002787240-0000000000002787253 -rw-r--r-- 1 root root 5611 Nov 22 10:36 edits_0000000000002787254-0000000000002787287 -rw-r--r-- 1 root root 1588 Nov 22 10:36 edits_0000000000002787288-0000000000002787301 -rw-r--r-- 1 root root 1251 Nov 22 10:36 edits_0000000000002787302-0000000000002787313 -rw-r--r-- 1 root root 4669 Nov 22 10:36 edits_0000000000002787314-0000000000002787338 -rw-r--r-- 1 root root 1587 Nov 22 10:36 edits_0000000000002787339-0000000000002787352 -rw-r--r-- 1 root root 4330 Nov 22 10:36 edits_0000000000002787353-0000000000002787375 -rw-r--r-- 1 root root 816 Nov 22 10:36 edits_0000000000002787376-0000000000002787383 -rw-r--r-- 1 root root 4330 Nov 22 10:36 edits_0000000000002787384-0000000000002787406 -rw-r--r-- 1 root root 1588 Nov 22 10:36 edits_0000000000002787407-0000000000002787420 -rw-r--r-- 1 root root 1588 Nov 22 10:36 edits_0000000000002787421-0000000000002787434 -rw-r--r-- 1 root root 4331 Nov 22 10:36 edits_0000000000002787435-0000000000002787457 -rw-r--r-- 1 root root 3695 Nov 22 10:36 edits_0000000000002787458-0000000000002787494 -rw-r--r-- 1 root root 13128 Nov 22 10:36 edits_0000000000002787495-0000000000002787575 -rw-r--r-- 1 root root 8425 Nov 22 10:36 edits_0000000000002787576-0000000000002787639 -rw-r--r-- 1 root root 1588 Nov 22 10:36 edits_0000000000002787640-0000000000002787653 -rw-r--r-- 1 root root 4333 Nov 22 10:36 edits_0000000000002787654-0000000000002787676 -rw-r--r-- 1 root root 2808 Nov 22 10:36 edits_0000000000002787677-0000000000002787698 -rw-r--r-- 1 root root 4332 Nov 22 10:36 edits_0000000000002787699-0000000000002787721 -rw-r--r-- 1 root root 2858 Nov 22 10:36 edits_0000000000002787722-0000000000002787744 -rw-r--r-- 1 root root 2855 Nov 22 10:36 edits_0000000000002787745-0000000000002787767 -rw-r--r-- 1 root root 10117 Nov 22 10:36 edits_0000000000002787768-0000000000002787846 -rw-r--r-- 1 root root 9826 Nov 22 10:36 edits_0000000000002787847-0000000000002787916 -rw-r--r-- 1 root root 4940 Nov 22 10:36 edits_0000000000002787917-0000000000002787944 -rw-r--r-- 1 root root 1960 Nov 22 10:36 edits_0000000000002787945-0000000000002787961 -rw-r--r-- 1 root root 813 Nov 22 10:36 edits_0000000000002787962-0000000000002787969 -rw-r--r-- 1 root root 4327 Nov 22 10:36 edits_0000000000002787970-0000000000002787992 -rw-r--r-- 1 root root 1644 Nov 22 10:36 edits_0000000000002787993-0000000000002788007 -rw-r--r-- 1 root root 4334 Nov 22 10:36 edits_0000000000002788008-0000000000002788030 -rw-r--r-- 1 root root 1588 Nov 22 10:36 edits_0000000000002788031-0000000000002788044 -rw-r--r-- 1 root root 1589 Nov 22 10:36 edits_0000000000002788045-0000000000002788058 -rw-r--r-- 1 root root 4331 Nov 22 10:36 edits_0000000000002788059-0000000000002788081 -rw-r--r-- 1 root root 1589 Nov 22 10:36 edits_0000000000002788082-0000000000002788095 -rw-r--r-- 1 root root 4332 Nov 22 10:36 edits_0000000000002788096-0000000000002788118 -rw-r--r-- 1 root root 1587 Nov 22 10:36 edits_0000000000002788119-0000000000002788132 -rw-r--r-- 1 root root 1587 Nov 22 10:36 edits_0000000000002788133-0000000000002788146 -rw-r--r-- 1 root root 17814 Nov 22 10:36 edits_0000000000002788147-0000000000002788278 -rw-r--r-- 1 root root 1584 Nov 22 10:36 edits_0000000000002788279-0000000000002788292 -rw-r--r-- 1 root root 4333 Nov 22 10:36 edits_0000000000002788293-0000000000002788315 -rw-r--r-- 1 root root 1588 Nov 22 10:36 edits_0000000000002788316-0000000000002788329 -rw-r--r-- 1 root root 1584 Nov 22 10:36 edits_0000000000002788330-0000000000002788343 -rw-r--r-- 1 root root 4390 Nov 22 10:36 edits_0000000000002788344-0000000000002788367 -rw-r--r-- 1 root root 1583 Nov 22 10:36 edits_0000000000002788368-0000000000002788381 -rw-r--r-- 1 root root 9140 Nov 22 10:36 edits_0000000000002788382-0000000000002788428 -rw-r--r-- 1 root root 4664 Nov 22 10:36 edits_0000000000002788429-0000000000002788458 -rw-r--r-- 1 root root 1590 Nov 22 10:36 edits_0000000000002788459-0000000000002788472 -rw-r--r-- 1 root root 4332 Nov 22 10:36 edits_0000000000002788473-0000000000002788495 -rw-r--r-- 1 root root 1761 Nov 22 10:36 edits_0000000000002788496-0000000000002788509 -rw-r--r-- 1 root root 4333 Nov 22 10:36 edits_0000000000002788510-0000000000002788532 -rw-r--r-- 1 root root 1586 Nov 22 10:36 edits_0000000000002788533-0000000000002788546 -rw-r--r-- 1 root root 1589 Nov 22 10:36 edits_0000000000002788547-0000000000002788560 -rw-r--r-- 1 root root 816 Nov 22 10:36 edits_0000000000002788561-0000000000002788568 -rw-r--r-- 1 root root 563666 Nov 22 10:36 fsimage_0000000000005559643 -rw-r--r-- 1 root root 62 Nov 22 10:36 fsimage_0000000000005559643.md5 -rw-r--r-- 1 root root 563666 Nov 22 10:36 fsimage_0000000000005560209 -rw-r--r-- 1 root root 62 Nov 22 10:36 fsimage_0000000000005560209.md5 -rw-r--r-- 1 root root 8 Nov 22 10:36 seen_txid  and  hdfs-site.xml: cat /etc/hadoop/conf.cloudera.hdfs/hdfs-site.xml <?xml version="1.0" encoding="UTF-8"?> <!--Autogenerated by Cloudera Manager--> <configuration> <property> <name>dfs.nameservices</name> <value>hdfs-cdp7</value> </property> <property> <name>dfs.client.failover.proxy.provider.hdfs-cdp7</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <property> <name>dfs.ha.automatic-failover.enabled.hdfs-cdp7</name> <value>true</value> </property> <property> <name>ha.zookeeper.quorum</name> <value>ip-10-2-0-115.ec2.internal:2181,ip-10-2-0-171.ec2.internal:2181,ip-10-2-0-177.ec2.internal:2181</value> </property> <property> <name>dfs.ha.namenodes.hdfs-cdp7</name> <value>namenode1546334668,namenode1546333642</value> </property> <property> <name>dfs.namenode.rpc-address.hdfs-cdp7.namenode1546334668</name> <value>ip-10-2-0-224.ec2.internal:8020</value> </property> <property> <name>dfs.namenode.servicerpc-address.hdfs-cdp7.namenode1546334668</name> <value>ip-10-2-0-224.ec2.internal:8022</value> </property> <property> <name>dfs.namenode.http-address.hdfs-cdp7.namenode1546334668</name> <value>ip-10-2-0-224.ec2.internal:9870</value> </property> <property> <name>dfs.namenode.https-address.hdfs-cdp7.namenode1546334668</name> <value>ip-10-2-0-224.ec2.internal:9871</value> </property> <property> <name>dfs.namenode.rpc-address.hdfs-cdp7.namenode1546333642</name> <value>ip-10-2-0-49.ec2.internal:8020</value> </property> <property> <name>dfs.namenode.servicerpc-address.hdfs-cdp7.namenode1546333642</name> <value>ip-10-2-0-49.ec2.internal:8022</value> </property> <property> <name>dfs.namenode.http-address.hdfs-cdp7.namenode1546333642</name> <value>ip-10-2-0-49.ec2.internal:9870</value> </property> <property> <name>dfs.namenode.https-address.hdfs-cdp7.namenode1546333642</name> <value>ip-10-2-0-49.ec2.internal:9871</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.blocksize</name> <value>134217728</value> </property> <property> <name>dfs.client.use.datanode.hostname</name> <value>false</value> </property> <property> <name>fs.permissions.umask-mode</name> <value>022</value> </property> <property> <name>dfs.client.block.write.locateFollowingBlock.retries</name> <value>7</value> </property> <property> <name>dfs.encrypt.data.transfer.algorithm</name> <value>3des</value> </property> <property> <name>dfs.encrypt.data.transfer.cipher.suites</name> <value>AES/CTR/NoPadding</value> </property> <property> <name>dfs.encrypt.data.transfer.cipher.key.bitlength</name> <value>256</value> </property> <property> <name>dfs.namenode.acls.enabled</name> <value>true</value> </property> <property> <name>dfs.client.read.shortcircuit</name> <value>true</value> </property> <property> <name>dfs.client.read.shortcircuit.streams.cache.size</name> <value>4096</value> </property> <property> <name>dfs.domain.socket.path</name> <value>/var/run/hdfs-sockets/dn</value> </property> <property> <name>dfs.client.read.shortcircuit.skip.checksum</name> <value>false</value> </property> <property> <name>dfs.client.domain.socket.data.traffic</name> <value>false</value> </property> <property> <name>dfs.datanode.hdfs-blocks-metadata.enabled</name> <value>true</value> </property> <property> <name>dfs.block.access.token.enable</name> <value>true</value> </property> <property> <name>dfs.namenode.kerberos.principal</name> <value>hdfs/_HOST@CDP7.COM</value> </property> <property> <name>dfs.namenode.kerberos.internal.spnego.principal</name> <value>HTTP/_HOST@CDP7.COM</value> </property> <property> <name>dfs.datanode.kerberos.principal</name> <value>hdfs/_HOST@CDP7.COM</value> </property> </configuration>  core-site.xml: cat /etc/hadoop/conf.cloudera.hdfs/core-site.xml <?xml version="1.0" encoding="UTF-8"?> <!--Autogenerated by Cloudera Manager--> <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hdfs-cdp7</value> </property> <property> <name>ipc.client.connection.maxidletime</name> <value>30000</value> </property> <property> <name>ipc.client.connect.max.retries</name> <value>50</value> </property> <property> <name>fs.trash.interval</name> <value>1</value> </property> <property> <name>io.compression.codecs</name> <value>org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.DeflateCodec,org.apache.hadoop.io.compress.SnappyCodec,org.apache.hadoop.io.compress.Lz4Codec</value> </property> <property> <name>hadoop.security.authentication</name> <value>kerberos</value> </property> <property> <name>hadoop.security.authorization</name> <value>true</value> </property> <property> <name>hadoop.rpc.protection</name> <value>authentication</value> </property> <property> <name>hadoop.ssl.require.client.cert</name> <value>false</value> <final>true</final> </property> <property> <name>hadoop.ssl.keystores.factory.class</name> <value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value> <final>true</final> </property> <property> <name>hadoop.ssl.server.conf</name> <value>ssl-server.xml</value> <final>true</final> </property> <property> <name>hadoop.ssl.client.conf</name> <value>ssl-client.xml</value> <final>true</final> </property> <property> <name>hadoop.security.auth_to_local</name> <value>RULE:[2:$1@$0](rangeradmin@CDP7.COM)s/(.*)@CDP7.COM/ranger/ RULE:[2:$1@$0](rangertagsync@CDP7.COM)s/(.*)@CDP7.COM/rangertagsync/ RULE:[2:$1@$0](rangerusersync@CDP7.COM)s/(.*)@CDP7.COM/rangerusersync/ RULE:[1:$1@$0](HTTP.*@CDP7.COM$)s/@CDP7.COM$// RULE:[1:$1@$0](.*@CDP7.COM$)s/@CDP7.COM$///L RULE:[2:$1@$0](HTTP.*@CDP7.COM$)s/@CDP7.COM$// RULE:[2:$1@$0](.*@CDP7.COM$)s/@CDP7.COM$///L DEFAULT</value> </property> <property> <name>hadoop.proxyuser.oozie.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.oozie.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.HTTP.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.HTTP.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hive.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hive.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hue.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hue.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.httpfs.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.httpfs.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.knox.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.knox.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.livy.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.livy.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.impala.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.impala.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hdfs.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.hdfs.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.yarn.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.yarn.groups</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.phoenix.hosts</name> <value>*</value> </property> <property> <name>hadoop.proxyuser.phoenix.groups</name> <value>*</value> </property> <property> <name>hadoop.security.group.mapping</name> <value>org.apache.hadoop.security.ShellBasedUnixGroupsMapping</value> </property> <property> <name>hadoop.security.instrumentation.requires.admin</name> <value>false</value> </property> </configuration>  Thanks! ... View more