Error sending message to Solr : o collection param specified on request and no default collection has been set.

[#1]

I built Hadoop cluster with HDP 2.3.0. This cluster is kerberized.

I'm trying Apache Ranger 0.5.0.2.3 for managing the access control and running into the following error.

2015-12-08 12:49:10,427 ERROR [hiveServer2.async.summary.multi_dest.batch_destWriter]: provider.BaseAuditHandler (BaseAuditHandler.java:logError(318)) - Error sending message to Solr
org.apache.solr.client.solrj.SolrServerException: No collection param specified on request and no default collection has been set.
        at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:519)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:918)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:809)
        at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:752)
        at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:131)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:76)
        at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:62)
        at org.apache.ranger.audit.destination.SolrAuditDestination.log(SolrAuditDestination.java:132)
        at org.apache.ranger.audit.provider.BaseAuditHandler.logJSON(BaseAuditHandler.java:161)
        at org.apache.ranger.audit.queue.AuditFileSpool.sendEvent(AuditFileSpool.java:882)
        at org.apache.ranger.audit.queue.AuditFileSpool.runDoAs(AuditFileSpool.java:830)
        at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:759)
        at org.apache.ranger.audit.queue.AuditFileSpool$2.run(AuditFileSpool.java:757)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:360)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1637)
        at org.apache.ranger.audit.queue.AuditFileSpool.run(AuditFileSpool.java:765)
        at java.lang.Thread.run(Thread.java:745)
2015-12-08 12:49:10,427 ERROR [hiveServer2.async.summary.multi_dest.batch_destWriter]: queue.AuditFileSpool (AuditFileSpool.java:logError(710)) - Error sending logs to consumer. provider=hiveServer2.async.summary.multi_dest.batch, consumer=hiveServer2.async.summary.multi_dest.batch.solr

I find this error in the hiveserver log file, /var/log/hive/hiveserver2.log, and audit logs for hive access don't appear on Audit tab page in Ranger UI.

I'm trying to solve the error, but I can't solve yet.

Please let me know what I should check.

Version

HDP 2.3.0.0

Ranger 0.5.0.2.3

Hive 1.2.1.2.3

Solr 5.2.1

I have bad English, so I apologize if I say something strange.

Thanks.

[#2 · Accepted solution]

Thanks for you question 🙂

I am seeing the same error message with HDFS SolrCloud Audit. Basically the Audit client is not picking up the collection name from the configuration and setting "ranger.audit.solr.collection.name" to "ranger_audits" has no effect. I will follow up on this issue and see what I can find out.

Just to make sure, did you follow this guide (http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Ranger_Install_Guide/content/ch_install_s...) to setup your SolrCloud for Ranger Audits?

Does your Audit configuration for Hive look something like this:

xasecure.audit.destination.solr.zookeepersmaster01.example.com:2181/solrxasecure.audit.destination.solr.urls{{ranger_audit_solr_urls}}Audit to SOLRtrue

You can work around this issue by setting the following configuration:

HDFS (ranger-hdfs-audit): xasecure.audit.destination.solr.zookeepers=NONE

Ranger (ranger-admin-site): ranger.audit.solr.urls=http://solrNode01.example.com:8983/solr/ranger_audits

This way you're working around Zookeeper and write your audit log directly to one of the Solr nodes.

[#3]

Thank you for your prompt reply!!

I had installed in according to the guide you told me.

However, some settings were wrong.

xasecure.audit.destination.solr.zookeepers for HDFS(ranger-hdfs-audit) was not "none"

xasecure.audit.destination.solr.zookeepers for Hive(ranger-hive-audit) was "master:2181/ranger_audits"

After fixing these settings, that problem disappeared and I can look audit logs for hive.

Thank you so much, Jonas.

[#4]

Awesome, I am glad you were able to fix it 🙂

I guess you are not using a separate znode for Solr in your Zookeeper environment, right? So basically all you solr content is placed in the root directory of Zookeeper.

[#5]

You are right. My Zookeeper environment is not using a separate znode.

I learned much from you. Thank you.

[#6]

Thanks for confirming! The Znode makes a huge difference 🙂

[#7]

The upcoming patch for Ranger should support giving Zookeeper quorum used by Ranger as the property.

[#8]

Thank you for telling me that.

I'm looking forward to it.