Problem: Cloudera Agent HTTP error 401. Local Kerberos: Active Version: CDH 6.0.0 HDFS and YARN, both with HA on, running perfectly. Disabling/Enabling ResourceManager works perfectly. One RM goes active after the other goes down (example). Still, whenever one RM goes in Standby mode, the Cloudera-SCM-Agent starts showing the following error. [21/Nov/2018 15:00:31 +0000] 30488 Monitor-GenericMonitor url ERROR Autentication error on attempt 1. Retrying after sleeping 1.000000 seconds.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/util/url.py", line 241, in urlopen_with_retry_on_authentication_errors
return function()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/generic/metric_collectors.py", line 220, in _open_url
password=self._password_value)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/util/url.py", line 82, in urlopen_with_timeout
return opener.open(url, data, timeout)
File "/usr/lib64/python2.7/urllib2.py", line 437, in open
response = meth(req, response)
File "/usr/lib64/python2.7/urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib64/python2.7/urllib2.py", line 469, in error
result = self._call_chain(*args)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/urllib2_kerberos.py", line 203, in http_error_401
retry = self.http_error_auth_reqed(host, req, headers)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/urllib2_kerberos.py", line 127, in http_error_auth_reqed
return self.retry_http_kerberos_auth(req, headers, neg_value)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/urllib2_kerberos.py", line 143, in retry_http_kerberos_auth
resp = self.parent.open(req)
File "/usr/lib64/python2.7/urllib2.py", line 437, in open
response = meth(req, response)
File "/usr/lib64/python2.7/urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib64/python2.7/urllib2.py", line 469, in error
result = self._call_chain(*args)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib64/python2.7/urllib2.py", line 656, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/lib64/python2.7/urllib2.py", line 437, in open
response = meth(req, response)
File "/usr/lib64/python2.7/urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib64/python2.7/urllib2.py", line 475, in error
return self._call_chain(*args)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 219, in http_error_default
raise e
HTTPError: HTTP Error 401: Authentication required If, we disable the property " Enable Kerberos Authentication for HTTP Web-Consoles" this problem goes away. At the moment this is the only problematic situation we have with Kerberos. We've tried to Regenerate Keytabs, but the problem remains. This only happens in the Agent that has the Standby Resource Manager. On the Active there is no problem. If we shutdown the Active, the Standby goes up (to Active as it should), and then the error starts appearing in that Agent (in the "new" Standby). Also, if we remove the HA from YARN, (only one RM "active") the problem goes away... Any ideas? [Update]: we found out the following (fake names for example). klist /var/run/cloudera-scm-agent/krb5cc_cm_agent_0
Ticket cache: FILE:/var/run/cloudera-scm-agent/krb5cc_cm_agent_0
Default principal: HTTP/sl000060.besp.dsp.gbes@CLBGDXD01.BESP.DSP.GBES
Valid starting Expires Service principal
11/21/2018 16:44:55 11/22/2018 16:44:55 krbtgt/LOCAL.REALM@LOCAL.REALM
renew until 11/26/2018 16:44:55
11/21/2018 16:45:25 11/22/2018 16:44:55 HTTP/sl000060.domain.stuff@
renew until 11/26/2018 16:44:55
11/21/2018 16:45:25 11/22/2018 16:44:55 HTTP/sl000060.domain.stuff@LOCAL.REALM
renew until 11/26/2018 16:44:55 Where is this coming from stuff@, when we only have HTTP principals in kadmin.local with the format of HTTP/node.domain.stuff@LOCAL.REALM ???
... View more
