Community Articles

dvillarreal · ‎12-25-2016

ERRORS:

from HDFS log:

016-10-30 17:44:04,226 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(903)) - Request to collection ranger_audits failed due to (401) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://hwx.com:8886/solr/ranger_audits_shard1_replica1: Expected mime type application/octet-stream but got text/html. <html> 
<head> 
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> 
<title>Error 401 Authentication required</title> 
</head> 
<body><h2>HTTP ERROR 401</h2> 
<p>Problem accessing /solr/ranger_audits_shard1_replica1/update. Reason: 
<pre> Authentication required</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

From Hive log:

016-10-30 17:50:46,189 WARN [org.apache.ranger.audit.queue.AuditBatchQueue1]: provider.BaseAuditHandler (BaseAuditHandler.java:logFailedEvent(374)) - failed 
to log audit event: {"repoType":3,"repo":"AA_Prod_hive","reqUser":"alex","evtTime":"2016-10-30 17:50:43.587","access":"USE","resource":"default","resType":"@ 
database","action":"_any","result":0,"policy":-1,"enforcer":"ranger-acl","sess":"cf4d0c81-c4df-483b-ab51-aa7bb5cb1633","cliType":"HIVESERVER2","cliIP":"172.26 
.205.88","reqData":"show tables","agentHost":"hwx.com","logType":"RangerAudit","id":"d41d25ee-d198-475d-a288-11d6cc76535c-0","seq_num":1 
,"event_count":1,"event_dur_ms":0,"tags":[],"additional_info":"{\"remote-ip-address\":172.26.1.1, \"forwarded-ip-addresses\":[]"} 
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://hwx:8886/solr/ranger_audits_shard1_re 
plica1: Expected mime type application/octet-stream but got text/html. <html> 
<head> 
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> 
<title>Error 401 Authentication required</title>

Debug: Once DEBUG log for krb5 is enabled (-Dsun.security.krb5.debug=true) we can see in both hdfs (log is hadoop-hdfs-namenode-hwx.com.out) and hive (log is hive-server2.out) the same issue as from the Knox ticket (it tries to use the HTTPS/_HOST principal instead of HTTP/_HOST as it's standard with spnego):

>>KRBError: 
sTime is Sun Oct 30 17:50:08 PDT 2016 1477875008000 
suSec is 518135 
error code is 7 
error Message is Server not found in Kerberos database 
sname is HTTPS/host@HWX.COM 
msgType is 30

ROOT CAUSE: There is a defect in httpclient 4.5.2 that got introduced in HDP 2.5. WORKAROUND: Downgrade all the httpclient at 4.5.2 for ranger to 4.5.1

This will be fixed in a future release.

dvillarreal · ‎12-25-2016

httpclient-451jar.zip

dvillarreal · ‎12-25-2016

Also, see https://issues.apache.org/jira/browse/KNOX-762

ChrisV · ‎03-16-2017

Hi,

This bug can have consequences on Spark / Yarn as well. We were encountering Out of Memory conditions running Spark job, not matter how much memory we assigned, we kept ending up exhausting it completely.

This behaviour actually disappeared when we applied the fix listed here.

I'll post back when I know more about the root cause & link between issues.

Regards,

Christophe

Cloudera Community

Community Articles

HDP 2.5 Ranger service plugins fail to upload audits to SSL enabled Ambari Infa Solr instance and kerberos enabled cluster

Apache Ambari

Apache Ranger

Apache Solr

Re: HDP 2.5 Ranger service plugins fail to upload audits to SSL enabled Ambari Infa Solr instance and kerberos enabled cluster

Re: HDP 2.5 Ranger service plugins fail to upload audits to SSL enabled Ambari Infa Solr instance and kerberos enabled cluster

Re: HDP 2.5 Ranger service plugins fail to upload audits to SSL enabled Ambari Infa Solr instance and kerberos enabled cluster