Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
avatar

ERRORS:

from HDFS log:
016-10-30 17:44:04,226 ERROR impl.CloudSolrClient (CloudSolrClient.java:requestWithRetryOnStaleState(903)) - Request to collection ranger_audits failed due to (401) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://hwx.com:8886/solr/ranger_audits_shard1_replica1: Expected mime type application/octet-stream but got text/html. <html> 
<head> 
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> 
<title>Error 401 Authentication required</title> 
</head> 
<body><h2>HTTP ERROR 401</h2> 
<p>Problem accessing /solr/ranger_audits_shard1_replica1/update. Reason: 
<pre> Authentication required</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>
From Hive log:
016-10-30 17:50:46,189 WARN [org.apache.ranger.audit.queue.AuditBatchQueue1]: provider.BaseAuditHandler (BaseAuditHandler.java:logFailedEvent(374)) - failed 
to log audit event: {"repoType":3,"repo":"AA_Prod_hive","reqUser":"alex","evtTime":"2016-10-30 17:50:43.587","access":"USE","resource":"default","resType":"@ 
database","action":"_any","result":0,"policy":-1,"enforcer":"ranger-acl","sess":"cf4d0c81-c4df-483b-ab51-aa7bb5cb1633","cliType":"HIVESERVER2","cliIP":"172.26 
.205.88","reqData":"show tables","agentHost":"hwx.com","logType":"RangerAudit","id":"d41d25ee-d198-475d-a288-11d6cc76535c-0","seq_num":1 
,"event_count":1,"event_dur_ms":0,"tags":[],"additional_info":"{\"remote-ip-address\":172.26.1.1, \"forwarded-ip-addresses\":[]"} 
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from server at https://hwx:8886/solr/ranger_audits_shard1_re 
plica1: Expected mime type application/octet-stream but got text/html. <html> 
<head> 
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> 
<title>Error 401 Authentication required</title>
Debug: Once DEBUG log for krb5 is enabled (-Dsun.security.krb5.debug=true) we can see in both hdfs (log is hadoop-hdfs-namenode-hwx.com.out) and hive (log is hive-server2.out) the same issue as from the Knox ticket (it tries to use the HTTPS/_HOST principal instead of HTTP/_HOST as it's standard with spnego):
>>KRBError: 
sTime is Sun Oct 30 17:50:08 PDT 2016 1477875008000 
suSec is 518135 
error code is 7 
error Message is Server not found in Kerberos database 
sname is HTTPS/host@HWX.COM 
msgType is 30
ROOT CAUSE: There is a defect in httpclient 4.5.2 that got introduced in HDP 2.5. WORKAROUND: Downgrade all the httpclient at 4.5.2 for ranger to 4.5.1

This will be fixed in a future release.

2,625 Views
0 Kudos
Comments

Hi,

This bug can have consequences on Spark / Yarn as well. We were encountering Out of Memory conditions running Spark job, not matter how much memory we assigned, we kept ending up exhausting it completely.

This behaviour actually disappeared when we applied the fix listed here.

I'll post back when I know more about the root cause & link between issues.

Regards,

Christophe

Version history
Last update:
‎12-25-2016 07:24 PM
Updated by:
Contributors