ERRORS: from HDFS log: 016-10-30 17:44:04,226 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(903)) - Request to collection ranger_audits failed due to (401) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://hwx.com:8886/solr/ranger_audits_shard1_replica1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 401 Authentication required</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /solr/ranger_audits_shard1_replica1/update. Reason:
<pre> Authentication required</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
From Hive log: 016-10-30 17:50:46,189 WARN [org.apache.ranger.audit.queue.AuditBatchQueue1]: provider.BaseAuditHandler (BaseAuditHandler.java:logFailedEvent(374)) - failed
to log audit event: {"repoType":3,"repo":"AA_Prod_hive","reqUser":"alex","evtTime":"2016-10-30 17:50:43.587","access":"USE","resource":"default","resType":"@
database","action":"_any","result":0,"policy":-1,"enforcer":"ranger-acl","sess":"cf4d0c81-c4df-483b-ab51-aa7bb5cb1633","cliType":"HIVESERVER2","cliIP":"172.26
.205.88","reqData":"show tables","agentHost":"hwx.com","logType":"RangerAudit","id":"d41d25ee-d198-475d-a288-11d6cc76535c-0","seq_num":1
,"event_count":1,"event_dur_ms":0,"tags":[],"additional_info":"{\"remote-ip-address\":172.26.1.1, \"forwarded-ip-addresses\":[]"}
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://hwx:8886/solr/ranger_audits_shard1_re
plica1: Expected mime type application/octet-stream but got text/html. <html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 401 Authentication required</title>
Debug:
Once DEBUG log for krb5 is enabled (-Dsun.security.krb5.debug=true) we can see in both hdfs (log is hadoop-hdfs-namenode-hwx.com.out) and hive (log is hive-server2.out) the same issue as from the Knox ticket (it tries to use the HTTPS/_HOST principal instead of HTTP/_HOST as it's standard with spnego):
>>KRBError:
sTime is Sun Oct 30 17:50:08 PDT 2016 1477875008000
suSec is 518135
error code is 7
error Message is Server not found in Kerberos database
sname is HTTPS/host@HWX.COM
msgType is 30
ROOT CAUSE:
There is a defect in httpclient 4.5.2 that got introduced in HDP 2.5.
WORKAROUND:
Downgrade all the httpclient at 4.5.2 for ranger to 4.5.1 This will be fixed in a future release.
... View more
