Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Solr for Ranger Audits NoSuchFileException errors

Solr for Ranger Audits NoSuchFileException errors

Contributor

Hi All,

We have installed HDP-Search (Solr 5.5.0) on SUSE 11 SP3 to store Ranger audit logs. SolrCloud installation/configuration (1 shard, 2 replicas) done according to http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_Security_Guide/content/using_apache_solr_....

HDP 2.4.0 in use. We're seeing a lot of the below errors:

2016-08-29 11:20:24,434 [qtp94143590-73] ERROR [   ] org.apache.solr.common.SolrException (SolrException.java:148) - org.apache.solr.common.SolrException: Error handling 'status' action
        at org.apache.solr.handler.admin.CoreAdminOperation$4.call(CoreAdminOperation.java:192)
        at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:354)
        at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:153)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
        at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:676)
        at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:439)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
        at org.eclipse.jetty.server.Server.handle(Server.java:499)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
        at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.NoSuchFileException: /opt/solr/ranger_audit_server/ranger_audits_shard1_replica1/data/index/segments_f
        at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
        at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
        at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
        at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
        at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:97)
        at java.nio.file.Files.readAttributes(Files.java:1686)
        at java.nio.file.Files.size(Files.java:2275)
        at org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:210)
        at org.apache.lucene.store.NRTCachingDirectory.fileLength(NRTCachingDirectory.java:127)
        at org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:592)
        at org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:886)
        at org.apache.solr.handler.admin.CoreAdminOperation$4.call(CoreAdminOperation.java:180)
        ... 27 more.

These errors do not seem to actually interfere with Solr operation, but they do concern me.

Thank you and regards, Christian

7 REPLIES 7

Re: Solr for Ranger Audits NoSuchFileException errors

@Christian Guegi

Can you confirm if your solr updates are working fine ?

From the log definately it seems to be issue, not sure if its part of the log which needs to be captured.

Re: Solr for Ranger Audits NoSuchFileException errors

Contributor

I don't see any update errors in solr.log and the index is growing. So the updates are working fine

Re: Solr for Ranger Audits NoSuchFileException errors

are you see this error in ranger xa_portal.logs ?

Re: Solr for Ranger Audits NoSuchFileException errors

Contributor

No. These are Solr errors and are logged in /var/log/solr/ranger_audit/solr.log

Re: Solr for Ranger Audits NoSuchFileException errors

can you please check this particular dir is present /opt/solr/ranger_audit_server/ranger_audits_shard1_replica1/data/index/segments_f

Re: Solr for Ranger Audits NoSuchFileException errors

Contributor

No, this particular segment file is not present. However, I can reproduce the error with the following request:

http://<solr_host>:6083/solr/ranger_audits_shard1_replica2/admin/luke?numTerms=0&wt=json

Which occasionally returns

{"responseHeader":{"status":500,"QTime":1},"error":{"msg":"/opt/solr/ranger_audit_server/ranger_audits_shard1_replica2/data/index/segments_voa","trace":"java.nio.file.NoSuchFileException: /opt/solr/ranger_audit_server/ranger_audits_shard1_replica2/data/index/segments_voa\n\tat sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)\n\tat sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)\n\tat sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)\n\tat sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)\n\tat sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:97)\n\tat java.nio.file.Files.readAttributes(Files.java:1686)\n\tat java.nio.file.Files.size(Files.java:2275)\n\tat org.apache.lucene.store.FSDirectory.fileLength(FSDirectory.java:210)\n\tat org.apache.lucene.store.NRTCachingDirectory.fileLength(NRTCachingDirectory.java:127)\n\tat org.apache.solr.handler.admin.LukeRequestHandler.getIndexInfo(LukeRequestHandler.java:592)\n\tat org.apache.solr.handler.admin.LukeRequestHandler.handleRequestBody(LukeRequestHandler.java:137)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2082)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:670)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:458)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:225)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:183)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:499)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)\n\tat org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)\n\tat java.lang.Thread.run(Thread.java:745)\n","code":500}}

Checking the index shows a different segment file:

ll /opt/solr/ranger_audit_server/ranger_audits_shard1_replica2/data/index/
-rw-r--r-- 1 solr users      1795 Sep  6 08:23 segments_vob

Any idea why Solr tries to fetch a already replaced segment file?

Re: Solr for Ranger Audits NoSuchFileException errors

New Contributor

Are you able to resolved this issue ?

Don't have an account?
Coming from Hortonworks? Activate your account here