Accessing webHDFS works but YARN Rest API doesn't with kerberos enabled

jknulst — Wed, 28 Sep 2016 06:12:07 GMT

Hi,

I can access webHDFS from cli just fine:

[root@sandbox ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: root@SANDBOX.HORTONWORKS.COM


Valid starting     Expires            Service principal
09/28/16 00:25:33  09/28/16 10:25:36  krbtgt/SANDBOX.HORTONWORKS.COM@SANDBOX.HORTONWORKS.COM
	renew until 10/05/16 00:25:33
09/28/16 00:25:40  09/28/16 10:25:36  HTTP/sandbox.hortonworks.com@SANDBOX.HORTONWORKS.COM
	renew until 10/05/16 00:25:33
[root@sandbox ~]# curl -s -i --negotiate -u:anyUser http://sandbox.hortonworks.com:50070/webhdfs/v1/?op=LISTSTATUS
HTTP/1.1 401 Authentication required
Cache-Control: must-revalidate,no-cache,no-store
Date: Tue, 27 Sep 2016 23:07:01 GMT
Pragma: no-cache
Date: Tue, 27 Sep 2016 23:07:01 GMT
Pragma: no-cache
Content-Type: text/html; charset=iso-8859-1
WWW-Authenticate: Negotiate
Set-Cookie: hadoop.auth=; Path=/; HttpOnly
Content-Length: 1404
Server: Jetty(6.1.26.hwx)


HTTP/1.1 200 OK
Cache-Control: no-cache
Expires: Tue, 27 Sep 2016 23:07:01 GMT
Date: Tue, 27 Sep 2016 23:07:01 GMT
Pragma: no-cache
Expires: Tue, 27 Sep 2016 23:07:01 GMT
Date: Tue, 27 Sep 2016 23:07:01 GMT
Pragma: no-cache
Content-Type: application/json
Set-Cookie: hadoop.auth="u=root&p=root@SANDBOX.HORTONWORKS.COM&t=kerberos&e=1475053621856&s=OmhtWeWb8vfQ2n1eb9GhlOTq/CA="; Path=/; HttpOnly
Transfer-Encoding: chunked
Server: Jetty(6.1.26.hwx)


{"FileStatuses":{"FileStatus":[
{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16396,"group":"hadoop","length":0,"modificationTime":1472134778352,"owner":"yarn","pathSuffix":"app-logs","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":4,"fileId":16392,"group":"hdfs","length":0,"modificationTime":1457965550121,"owner":"hdfs","pathSuffix":"apps","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16389,"group":"hadoop","length":0,"modificationTime":1457965143118,"owner":"yarn","pathSuffix":"ats","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":17246,"group":"hdfs","length":0,"modificationTime":1457967047371,"owner":"hdfs","pathSuffix":"demo","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16403,"group":"hdfs","length":0,"modificationTime":1457965151394,"owner":"hdfs","pathSuffix":"hdp","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":16399,"group":"hdfs","length":0,"modificationTime":1457965149964,"owner":"mapred","pathSuffix":"mapred","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":2,"fileId":16401,"group":"hadoop","length":0,"modificationTime":1457965161645,"owner":"mapred","pathSuffix":"mr-history","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":1,"fileId":17161,"group":"hdfs","length":0,"modificationTime":1457966562806,"owner":"hdfs","pathSuffix":"ranger","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":0,"fileId":16437,"group":"hadoop","length":0,"modificationTime":1474960367134,"owner":"spark","pathSuffix":"spark-history","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":8,"fileId":16386,"group":"hdfs","length":0,"modificationTime":1472158956829,"owner":"hdfs","pathSuffix":"tmp","permission":"777","replication":0,"storagePolicy":0,"type":"DIRECTORY"},
{"accessTime":0,"blockSize":0,"childrenNum":9,"fileId":16387,"group":"hdfs","length":0,"modificationTime":1457966006266,"owner":"hdfs","pathSuffix":"user","permission":"755","replication":0,"storagePolicy":0,"type":"DIRECTORY"}
]}}

But when I try the same for YARN webUI or REST API it fails:

[root@sandbox ~]# curl -s -ikv --negotiate -u:anyUser -X GET http://sandbox.hortonworks.com:8088/ws/v1/cluster/apps
* About to connect() to sandbox.hortonworks.com port 8088 (#0)
*   Trying 10.0.3.15... connected
* Connected to sandbox.hortonworks.com (10.0.3.15) port 8088 (#0)
> GET /ws/v1/cluster/apps HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: sandbox.hortonworks.com:8088
> Accept: */*
>
< HTTP/1.1 401 Authentication required
HTTP/1.1 401 Authentication required
< Cache-Control: must-revalidate,no-cache,no-store
Cache-Control: must-revalidate,no-cache,no-store
< Date: Tue, 27 Sep 2016 23:08:45 GMT
Date: Tue, 27 Sep 2016 23:08:45 GMT
< Pragma: no-cache
Pragma: no-cache
< Date: Tue, 27 Sep 2016 23:08:45 GMT
Date: Tue, 27 Sep 2016 23:08:45 GMT
< Pragma: no-cache
Pragma: no-cache
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1
< WWW-Authenticate: PseudoAuth
WWW-Authenticate: PseudoAuth
< Set-Cookie: hadoop.auth=; Path=/; HttpOnly
Set-Cookie: hadoop.auth=; Path=/; HttpOnly
< Content-Length: 1411
Content-Length: 1411
< Server: Jetty(6.1.26.hwx)
Server: Jetty(6.1.26.hwx)


<
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"/>
<title>Error 401 Authentication required</title>
</head>
<body><h2>HTTP ERROR 401</h2>
<p>Problem accessing /ws/v1/cluster/apps. Reason:
<pre>    Authentication required</pre></p><hr /><i><small>Powered by Jetty://</small></i><br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>
<br/>


</body>
</html>
* Connection #0 to host sandbox.hortonworks.com left intact
* Closing connection #0

What is the difference with these 2 calls ?

Re: Accessing webHDFS works but YARN Rest API doesn't with kerberos enabled

berry_osterlund — Wed, 28 Sep 2016 17:43:21 GMT

I'm running on a secured cluster and with the execution of

curl --negotiate -u: -X GET http://<HOSTNAME>/ws/v1/cluster/apps

I get a normal response back. Have you enabled HTTP authentication for the services as described in https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_security/content/_configuring_http_authentication_for_HDFS_YARN_MapReduce2_HBase_Oozie_Falcon_and_Storm.html?

Re: Accessing webHDFS works but YARN Rest API doesn't with kerberos enabled

jknulst — Wed, 28 Sep 2016 19:52:03 GMT

That was it.I still had to apply the following to make it work for YARN as well:

First generate a secret key and push it to all nodes. Instructions here

Then add to custom core-site.xml:

hadoop.http.authentication.simple.anonymous.allowed=false
hadoop.http.authentication.signature.secret.file=/etc/security/http_secret
hadoop.http.authentication.type=kerberos
hadoop.http.authentication.kerberos.keytab=/etc/security/keytabs/spnego.service.keytab
hadoop.http.authentication.kerberos.principal=HTTP/_HOST@LAB.HORTONWORKS.NET
hadoop.http.authentication.cookie.domain=lab.hortonworks.net
hadoop.http.filter.initializers=org.apache.hadoop.security.AuthenticationFilterInitializer

Restart ambari-server

question Re: Accessing webHDFS works but YARN Rest API doesn't with kerberos enabled in Archives of Support Questions (Read Only)

Accessing webHDFS works but YARN Rest API doesn't with kerberos enabled

Re: Accessing webHDFS works but YARN Rest API doesn't with kerberos enabled

Re: Accessing webHDFS works but YARN Rest API doesn't with kerberos enabled