Missing HDFS, Yarn metrics after enable kerberos HTTP authentication

[#1]

Hello!

I have a kerberos enabled cluster with:

Ambari 2.4.1

HDP stack 2.4.0

One way trust between Kerberos and AD

After I have enabled HTTP authentication as described at http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_Security_Guide/content/_configuring_http_...

I have lost information in ambari as shows the following images:

If I check the logs of ambari-server.log I find a lot of entries with this message:

01 Feb 2017 12:41:47,167  WARN [ambari-metrics-retrieval-service-thread-1] RequestTargetAuthentication:88 - NEGOTIATE authentication error: No valid credentials provided (Mechanism level: No valid credentials provided (Mechanism level: Attempt to obtain new INITIATE credentials failed! (null)))
01 Feb 2017 12:41:47,169 ERROR [ambari-metrics-retrieval-service-thread-1] AppCookieManager:122 - SPNego authentication failed, can not get hadoop.auth cookie for URL: http://hadoop01.int:50070/jmx?get=Hadoop:service=NameNode,name=FSNamesystem::tag.HAState
01 Feb 2017 12:41:47,168  WARN [ambari-metrics-retrieval-service-thread-0] RequestTargetAuthentication:88 - NEGOTIATE authentication error: No valid credentials provided (Mechanism level: No valid credentials provided (Mechanism level: Attempt to obtain new INITIATE credentials failed! (null)))
01 Feb 2017 12:41:47,170 ERROR [ambari-metrics-retrieval-service-thread-0] AppCookieManager:122 - SPNego authentication failed, can not get hadoop.auth cookie for URL: http://hadoop01.int:8088/jmx

I have tried to regenerate the keytab but the problem persists.

I can access these url with my browser using both, kerberos ticket or AD windows ticket without any problem.

Any clue about what maybe happening?

Thank you in advance

[#2 · Accepted solution]

Did you configure Ambari for kerberos, then restart Ambari as well? http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/ch_enable_spnego_auth_fo...

Also run service check on Ambari Metrics. I also recommend upgrading Ambari to 2.4.2 but resolve the immediate issue first.

[#3]

@Artem Ervits

I also have read that part.. it's supossed to be configured automatically when I used Ambari for kerberos setup...

And yes I've restarted ambari-server.

Now what I have find is that I can curl from servers using @HADOOP.INT Realm users (krb5) but cannot with @TEST.INT AD realm... but If I go to a windows machine with an user logged in TEST.INT I can access to this websites using any webbrowser.. even after remove the hadoop.auth cookie.

output of curl error:

[root@hadoop02 lib]# curl --negotiate -u admin:admin <a href="http://hadoop01.int:8088/ws/v1/cluster/info">http://hadoop01.int:8088/ws/v1/cluster/info</a>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 403 GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)</title>
</head>
<body><h2>HTTP ERROR 403</h2>
<p>Problem accessing /ws/v1/cluster/info. Reason:
<pre>    GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/> 
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>

</body>
</html>

Ambari is suppose to use @HADOOP.INT realm user..

edited:

Ambari-metrics service check went fine.

[#4]

I finally configured ambari for kerberos manually even when documentation says that ambari performs that setup automatically and finally worked ... I had to do this optional stept http://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-security/content/set_up_kerberos_f...

Looks like it's not optional at all. thank you very much

[#5]

Excellent, I will review the language in documentation and issue a pull request to change. Sorry for confusion.