Support Questions
Find answers, ask questions, and share your expertise

Solr for Ranger Audits NoSuchFileException errors

Contributor

Hi All,

We have installed HDP-Search (Solr 5.5.0) on SUSE 11 SP3 to store Ranger audit logs. SolrCloud installation/configuration (1 shard, 2 replicas) done according to http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_Security_Guide/content/using_apache_solr_....

HDP 2.4.0 in use. We're seeing a lot of the below errors:

2016-08-29 11:20:24,434 [qtp94143590-73] ERROR [   ] org.apache.solr.common.SolrException (SolrException.java:148) - org.apache.solr.common.SolrException: Error handling 'status' action
        at org.apache.solr.handler.admin.CoreAdminOperation$4.call(CoreAdminOperation.java:192)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:354)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:153)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:676)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:439)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.NoSuchFileException: /opt/solr/ranger_audit_server/ranger_audits_shard1_replica1/data/index/segments_f
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
        at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
        at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:97)
        at java.nio.file.Files.readAttributes(Files.java:1686)
        at java.nio.file.Files.size(Files.java:2275)
        at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:210)
        at org.apache.lucene.store.NRTCachingDirectory.fileLength(NRTCachingDirectory.java:127)
        at org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:592)
        at org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:886)
        at org.apache.solr.handler.admin.CoreAdminOperation$4.call(CoreAdminOperation.java:180)
        ... 27 more.

These errors do not seem to actually interfere with Solr operation, but they do concern me.

Thank you and regards, Christian

7 REPLIES 7

Re: Solr for Ranger Audits NoSuchFileException errors

@Christian Guegi

Can you confirm if your solr updates are working fine ?

From the log definately it seems to be issue, not sure if its part of the log which needs to be captured.

Re: Solr for Ranger Audits NoSuchFileException errors

Contributor

I don't see any update errors in solr.log and the index is growing. So the updates are working fine

Re: Solr for Ranger Audits NoSuchFileException errors

are you see this error in ranger xa_portal.logs ?

Re: Solr for Ranger Audits NoSuchFileException errors

Contributor

No. These are Solr errors and are logged in /var/log/solr/ranger_audit/solr.log

Re: Solr for Ranger Audits NoSuchFileException errors

can you please check this particular dir is present /opt/solr/ranger_audit_server/ranger_audits_shard1_replica1/data/index/segments_f

Re: Solr for Ranger Audits NoSuchFileException errors

Contributor

No, this particular segment file is not present. However, I can reproduce the error with the following request:

http://<solr_host>:6083/solr/ranger_audits_shard1_replica2/admin/luke?numTerms=0&wt=json

Which occasionally returns

{"responseHeader":{"status":500,"QTime":1},"error":{"msg":"/opt/solr/ranger_audit_server/ranger_audits_shard1_replica2/data/index/segments_voa","trace":"java.nio.file.NoSuchFileException: /opt/solr/ranger_audit_server/ranger_audits_shard1_replica2/data/index/segments_voa\n\tat sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)\n\tat sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)\n\tat sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)\n\tat sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:97)\n\tat java.nio.file.Files.readAttributes(Files.java:1686)\n\tat java.nio.file.Files.size(Files.java:2275)\n\tat org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:210)\n\tat org.apache.lucene.store.NRTCachingDirectory.fileLength(NRTCachingDirectory.java:127)\n\tat org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:592)\n\tat org.apache.solr.handler.admin.LukeRequestHandler.handleRequestBody(LukeRequestHandler.java:137)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:499)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)\n\tat org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)\n\tat java.lang.Thread.run(Thread.java:745)\n","code":500}}

Checking the index shows a different segment file:

ll /opt/solr/ranger_audit_server/ranger_audits_shard1_replica2/data/index/
-rw-r--r-- 1 solr users      1795 Sep  6 08:23 segments_vob

Any idea why Solr tries to fetch a already replaced segment file?

Re: Solr for Ranger Audits NoSuchFileException errors

New Contributor

Are you able to resolved this issue ?