Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

SPNEGO issue after setting up MIT KDC one-way trust to AD

Solved Go to solution

SPNEGO issue after setting up MIT KDC one-way trust to AD

Super Collaborator

Hi,

I have set up my Kerberos conf on a MIT KDC kerberized cluster for trusting tickets from another AD domain/REALM with the steps described here. All the steps are implemented.

Now almost everything is working as expected:

-I can authenticate (to webHDFS) on any HDP node with tickets from both REALMS using 'curl --negotiate'

-I can authenticate (to webHDFS) from my own laptop (which is set up like just one of many computers in the AD network/REALM) with a ticket/keytab from the MIT REALM

but one vital part is not working:

-I can't authenticate to webHDFS from my own laptop having a ticket from AD. This is arguably the thing you want to have working is this setup, allowing anyone from the AD REALM to authenticate to HDP via TCP IP / browser

This is when authenticating with a ticket from the AD REALM (outside the cluster)

$ kdestroy
$ klist
klist: krb5_cc_get_principal: No credentials cache file found
$ kinit jk@FIELD.HORTONWORKS.COM
jk@FIELD.HORTONWORKS.COM's password:
$ klist
Credentials cache: API:42960B54-7745-4A95-B397-8FDE981283E4
        Principal: jk@FIELD.HORTONWORKS.COM


  Issued                Expires               Principal
Dec 22 00:24:01 2016  Dec 22 10:24:01 2016  krbtgt/FIELD.HORTONWORKS.COM@FIELD.HORTONWORKS.COM
$ curl -i -v --negotiate -u: sg-hdp24-mst7.field.hortonworks.com:50070/webhdfs/v1/?op=LISTSTATUS
*   Trying 172.26.233.221...
* TCP_NODELAY set
* Connected to sg-hdp24-mst7.field.hortonworks.com (172.26.233.221) port 50070 (#0)
> GET /webhdfs/v1/?op=LISTSTATUS HTTP/1.1
> Host: sg-hdp24-mst7.field.hortonworks.com:50070
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 401 Authentication required
HTTP/1.1 401 Authentication required
< Cache-Control: must-revalidate,no-cache,no-store
Cache-Control: must-revalidate,no-cache,no-store
< Date: Wed, 21 Dec 2016 23:24:11 GMT
Date: Wed, 21 Dec 2016 23:24:11 GMT
< Pragma: no-cache
Pragma: no-cache
< Date: Wed, 21 Dec 2016 23:24:11 GMT
Date: Wed, 21 Dec 2016 23:24:11 GMT
< Pragma: no-cache
Pragma: no-cache
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1
* gss_init_sec_context() failed:  An unsupported mechanism was requested. unknown mech-code 0 for mech unknown.
< WWW-Authenticate: Negotiate
WWW-Authenticate: Negotiate
< Set-Cookie: hadoop.auth=; Path=/; HttpOnly
Set-Cookie: hadoop.auth=; Path=/; HttpOnly
< Content-Length: 1404
Content-Length: 1404
< Server: Jetty(6.1.26.hwx)
Server: Jetty(6.1.26.hwx)


<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 401 Authentication required</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /webhdfs/v1/. Reason:
<pre>    Authentication required</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>
<br/>

</body>
</html>
* Curl_http_done: called premature == 0
* Connection #0 to host sg-hdp24-mst7.field.hortonworks.com left intact

So the error seems to be "* gss_init_sec_context() failed: An unsupported mechanism was requested. unknown mech-code 0 for mech unknown."

I am also using Firefox to test the same and it is configured to take on the SPNEGO ticket negotiation, (worked fine before, so that seems not to be the issue) .

10703-screen-shot-2016-12-22-at-115354-pm.png

The authentication error is the same now, but after the Firefox roundtrip something clearly changed to the ticket cache:

$ klist
Credentials cache: API:4E28B24B-FC22-4FF4-A769-840D5B058C25
        Principal: jk@FIELD.HORTONWORKS.COM


  Issued                Expires               Principal
Dec 22 23:51:44 2016  Dec 23 09:51:44 2016  krbtgt/FIELD.HORTONWORKS.COM@FIELD.HORTONWORKS.COM
Dec 22 23:53:37 2016  Dec 23 09:51:44 2016  krbtgt/MIT.KDC.COM@FIELD.HORTONWORKS.COM
$


So some cross REALM stuff is actually attempted it seems, but not leading to successful access to webHDFS.

Anyone, any clue ?

How to debug this?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: SPNEGO issue after setting up MIT KDC one-way trust to AD

Super Collaborator

Well eventually I was able to solve all this. I did multiple things, don't know exactly what solved it.

-Installed Mac_OS_X_10.4_10.6_Kerberos_Extras.dmg

-Upgraded Firefox to 50.1.0 from 49.x

-Reset value of 'network.negotiate-auth.trusted-uris' in Firefox about:config to '.field.hortonworks.com'

-Mapped all cluster nodes short and long fqdn in local /etc/hosts like

1xx.2x.x3x.220 sg-hdp24-mst6b sg-hdp24-mst6b.field.hortonworks.com

The local Kerberos config at /etc/krb5.conf has to have both REALMS:

[libdefaults]
  default_realm = MIT.KDC.COM

[domain_realm]
  .field.hortonworks.com = MIT.KDC.COM
  field.hortonworks.com = MIT.KDC.COM

[realms]
  FIELD.HORTONWORKS.COM = {
    admin_server = xxxx.field.hortonworks.com
    kdc = ad01.field.hortonworks.com
  }
MIT.KDC.COM = {
    admin_server = sg-hdp24-mst6b.field.hortonworks.com
    kdc = sg-hdp24-mst6b.field.hortonworks.com
  }

Both curl and webhdfs calls from Firefox work now. After such a successful call local cache looks like this:

$ klist
Credentials cache: API:C1AAF010-41BB-4705-B4FB-239BC06DCF8E
        Principal: jk@FIELD.HORTONWORKS.COM


  Issued                Expires               Principal
Dec 30 20:34:42 2016  Dec 31 06:34:42 2016  krbtgt/FIELD.HORTONWORKS.COM@FIELD.HORTONWORKS.COM
Dec 30 20:34:49 2016  Dec 31 06:34:42 2016  krbtgt/MIT.KDC.COM@FIELD.HORTONWORKS.COM
Dec 30 20:34:49 2016  Dec 31 06:34:42 2016  HTTP/sg-hdp24-mst7@MIT.KDC.COM

So now the cross realm trust cluster MIT --> AD is fully functional. One peculiar thing was that in Firefox the SPNEGO auth works just as well now for the destination 'http://sg-hdp24-mst7:50070/webhdfs/v1/?op=LISTSTATUS' as it is for http://sg-hdp24-mst7.field.hortonworks.com:50070/webhdfs/v1/?op=LISTSTATUS. So somehow Firefox figured out it needed to use Kerberos to auth without the domain indicator ('network.negotiate-auth.trusted-uris')

3 REPLIES 3

Re: SPNEGO issue after setting up MIT KDC one-way trust to AD

@Jasper

It seems like you have taken many of the steps that I would have. Normally I would have pointed you to a site like https://ping.force.com/Support/PingFederate/Integrations/How-to-configure-supported-browsers-for-Ker... to verify that your web browser was configured properly. If that didn't have any success, I would have had you execute curl to see if there are any interesting messages. In fact, in your case there are... or rather there is a lack of interesting messages.

I would have expected to see curl send back a new request containing an "Authorization" header after receiving the 401 error and the "WWW-Authenticate: Negotiate" response header. However this is not happening.

On top of this, the error that curl shows you is interesting

gss_init_sec_context() failed:  An unsupported mechanism was requested. unknown mech-code 0 for mech unknown

I typically see "unknown mech-code 0 for mech unknown" on successful calls, but I haven't seen "An unsupported mechanism was requested.", so maybe a feature of curl has not been turned on.

Execute "curl --version" to ensure that Kerberos authentication is enabled. On my workstation I get:

$curl --version
curl 7.43.0 (x86_64-apple-darwin15.0) libcurl/7.43.0 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets

Notice in the "features" section we see "GSS-API Kerberos SPNEGO". If you do not see this, your version of curl does not support this type of authentication. This may or may not affect how FireFox works.

Re: SPNEGO issue after setting up MIT KDC one-way trust to AD

Super Collaborator

@Robert Levas

I have this version:

$ curl --version
curl 7.51.0 (x86_64-apple-darwin16.0) libcurl/7.51.0 SecureTransport zlib/1.2.8
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets

But I think it is not curl, since an earlier test, with a keytab exported from HDP and used locally on my laptop just works:

$ kinit -kt ~/Downloads/smokeuser.headless.keytab ambari-qa-socgen_shadow@MIT.KDC.COM
Encryption type arcfour-hmac-md5(23) used for authentication is weak and will be deprecated
$ klist
Credentials cache: API:B0F322D5-5C1F-4EBC-9936-224FF7374B53
        Principal: ambari-qa-socgen_shadow@MIT.KDC.COM
  Issued                Expires               Principal
Dec 21 18:39:23 2016  Dec 22 18:39:22 2016  krbtgt/MIT.KDC.COM@MIT.KDC.COM
$ curl -i -v --negotiate -u: sg-hdp24-mst7:50070/webhdfs/v1/?op=LISTSTATUS
*   Trying 172.26.233.221...
* TCP_NODELAY set
* Connected to sg-hdp24-mst7 (172.26.233.221) port 50070 (#0)
> GET /webhdfs/v1/?op=LISTSTATUS HTTP/1.1
> Host: sg-hdp24-mst7:50070
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 401 Authentication required
HTTP/1.1 401 Authentication required
< Cache-Control: must-revalidate,no-cache,no-store
Cache-Control: must-revalidate,no-cache,no-store
< Date: Wed, 21 Dec 2016 17:39:39 GMT
Date: Wed, 21 Dec 2016 17:39:39 GMT
< Pragma: no-cache
Pragma: no-cache
< Date: Wed, 21 Dec 2016 17:39:39 GMT
Date: Wed, 21 Dec 2016 17:39:39 GMT
< Pragma: no-cache
Pragma: no-cache
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1
< WWW-Authenticate: Negotiate
WWW-Authenticate: Negotiate
< Set-Cookie: hadoop.auth=; Path=/; HttpOnly
Set-Cookie: hadoop.auth=; Path=/; HttpOnly
< Content-Length: 1404
Content-Length: 1404
< Server: Jetty(6.1.26.hwx)
Server: Jetty(6.1.26.hwx)
<
* Ignoring the response-body
* Curl_http_done: called premature == 0
* Connection #0 to host sg-hdp24-mst7 left intact
* Issue another request to this URL: 'http://sg-hdp24-mst7:50070/webhdfs/v1/?op=LISTSTATUS'
* Found bundle for host sg-hdp24-mst7: 0x7ff4a8f01430 [can pipeline]
* Re-using existing connection! (#0) with host sg-hdp24-mst7
* Connected to sg-hdp24-mst7 (172.26.233.221) port 50070 (#0)
* Server auth using Negotiate with user ''
> GET /webhdfs/v1/?op=LISTSTATUS HTTP/1.1
> Host: sg-hdp24-mst7:50070
> Authorization: Negotiate YIIDhwYGKwYBBQUCoIIDezCCA3egFTATBgkqhkiG9xIBAgIGBiqFcCsOA6KCA1wE.........
.......
.....WP6BPmSnLg/JXzr+NpYRnOMvrCtXaFrVPKJ2qtiYc2nOAX1hTsEOnJGkL2WHFdKo6/P7OnRLGzYXLrtWHAeL3IbYNM3moXdJRnf23aItsAhk/r6O7H88eSRHtOzd7HFscaAtlmV8Goh8V2JvQ==
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Cache-Control: no-cache
Cache-Control: no-cache
< Expires: Wed, 21 Dec 2016 17:39:39 GMT
Expires: Wed, 21 Dec 2016 17:39:39 GMT
< Date: Wed, 21 Dec 2016 17:39:39 GMT
Date: Wed, 21 Dec 2016 17:39:39 GMT
< Pragma: no-cache
Pragma: no-cache
< Expires: Wed, 21 Dec 2016 17:39:39 GMT
Expires: Wed, 21 Dec 2016 17:39:39 GMT
< Date: Wed, 21 Dec 2016 17:39:39 GMT
Date: Wed, 21 Dec 2016 17:39:39 GMT
< Pragma: no-cache
Pragma: no-cache
< Content-Type: application/json
Content-Type: application/json
< WWW-Authenticate: Negotiate oYH1MIHyoAMKAQChCwYJKoZIhvcSAQICom4EbGBqBgkqhkiG9xIBAgICAG9bMFmgAwIBBaEDAgEPok0wS6ADAgESokQEQtoEqw8cPRBs2EiQdAiNPPzx2wLLLBDzLrwUKneExsT/OopV3GrnqmXPxWeF............
............
...............TRLeDF+OwJKZh2k=
< Set-Cookie: hadoop.auth="u=ambari-qa&p=ambari-qa-socgen_shadow@MIT.KDC.COM&t=kerberos&e=1482377979783&s=MCVcBnMN/AWeNEnqrTk/msgmRrA="; Path=/; HttpOnly
Set-Cookie: hadoop.auth="u=ambari-qa&p=ambari-qa-socgen_shadow@MIT.KDC.COM&t=kerberos&e=1482377979783&s=MCVcBnMN/AWeNEnqrTk/msgmRrA="; Path=/; HttpOnly
< Transfer-Encoding: chunked
Transfer-Encoding: chunked
< Server: Jetty(6.1.26.hwx)
Server: Jetty(6.1.26.hwx)
<
{"FileStatuses":{"FileStatus":[
{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16392,"group":"hadoop","length":0,"modificationTime":1480419441661,"owner":"yarn","pathSuffix":"app-logs","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16389,"group":"hadoop","length":0,"modificationTime":1480415609465,"owner":"yarn","pathSuffix":"ats","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16399,"group":"hdfs","length":0,"modificationTime":1480415614093,"owner":"hdfs","pathSuffix":"hdp","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16395,"group":"hdfs","length":0,"modificationTime":1480415613182,"owner":"mapred","pathSuffix":"mapred","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16397,"group":"hadoop","length":0,"modificationTime":1480415620607,"owner":"mapred","pathSuffix":"mr-history","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16606,"group":"hdfs","length":0,"modificationTime":1480428257125,"owner":"hdfs","pathSuffix":"system","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":5,"fileId":16386,"group":"hdfs","length":0,"modificationTime":1482186163272,"owner":"hdfs","pathSuffix":"tmp","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":3,"fileId":16387,"group":"hdfs","length":0,"modificationTime":1480419487982,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
]}}
* Curl_http_done: called premature == 0
* Closing connection 0
 

Could there be a clue in the warning "Encryption type arcfour-hmac-md5(23) used for authentication is weak and will be deprecated" ?

Re: SPNEGO issue after setting up MIT KDC one-way trust to AD

Super Collaborator

Well eventually I was able to solve all this. I did multiple things, don't know exactly what solved it.

-Installed Mac_OS_X_10.4_10.6_Kerberos_Extras.dmg

-Upgraded Firefox to 50.1.0 from 49.x

-Reset value of 'network.negotiate-auth.trusted-uris' in Firefox about:config to '.field.hortonworks.com'

-Mapped all cluster nodes short and long fqdn in local /etc/hosts like

1xx.2x.x3x.220 sg-hdp24-mst6b sg-hdp24-mst6b.field.hortonworks.com

The local Kerberos config at /etc/krb5.conf has to have both REALMS:

[libdefaults]
  default_realm = MIT.KDC.COM

[domain_realm]
  .field.hortonworks.com = MIT.KDC.COM
  field.hortonworks.com = MIT.KDC.COM

[realms]
  FIELD.HORTONWORKS.COM = {
    admin_server = xxxx.field.hortonworks.com
    kdc = ad01.field.hortonworks.com
  }
MIT.KDC.COM = {
    admin_server = sg-hdp24-mst6b.field.hortonworks.com
    kdc = sg-hdp24-mst6b.field.hortonworks.com
  }

Both curl and webhdfs calls from Firefox work now. After such a successful call local cache looks like this:

$ klist
Credentials cache: API:C1AAF010-41BB-4705-B4FB-239BC06DCF8E
        Principal: jk@FIELD.HORTONWORKS.COM


  Issued                Expires               Principal
Dec 30 20:34:42 2016  Dec 31 06:34:42 2016  krbtgt/FIELD.HORTONWORKS.COM@FIELD.HORTONWORKS.COM
Dec 30 20:34:49 2016  Dec 31 06:34:42 2016  krbtgt/MIT.KDC.COM@FIELD.HORTONWORKS.COM
Dec 30 20:34:49 2016  Dec 31 06:34:42 2016  HTTP/sg-hdp24-mst7@MIT.KDC.COM

So now the cross realm trust cluster MIT --> AD is fully functional. One peculiar thing was that in Firefox the SPNEGO auth works just as well now for the destination 'http://sg-hdp24-mst7:50070/webhdfs/v1/?op=LISTSTATUS' as it is for http://sg-hdp24-mst7.field.hortonworks.com:50070/webhdfs/v1/?op=LISTSTATUS. So somehow Firefox figured out it needed to use Kerberos to auth without the domain indicator ('network.negotiate-auth.trusted-uris')