KNOX Kerberos webHDFS HA

[#1]

Hi,

I have kerberos and HA enabled on my hadoop cluster.Now to enable HA over the webhdfs i did the following configuration:

<provider>

<role>ha</role>

<name>HaProvider</name>

<enabled>true</enabled>

<param>

<name>WEBHDFS</name> <value>maxFailoverAttempts=3;failoverSleep=1000;maxRetryAttempts=300;retrySleep=1000;enabled=true</value>

</param>

</provider>

<service>

<role>WEBHDFS</role>

<url>http://<nn1>:50070/webhdfs</url>

<url>http://<nn2>:50070/webhdfs</url>

</service>

But the curl command is still failing.

I am not using SSL.

Can someone point to correct curl command i should use assuming knox1 is my hostname of knox gateway.

I have used below command

$ curl -k -i -vvvv --negotiate -u : "http://<knox1>:50070/gateway/<cluster_name>/webhdfs/v1/user?=op=LISTSTATUS"

I have followed below tutorials but they cannot help:

https://community.hortonworks.com/questions/35125/knox-error-after-configuring-namenode-ha.html

https://community.hortonworks.com/content/supportkb/150585/how-to-configure-a-knox-topology-for-name...

[#2]

@Mudit Kumar

Your configuration looks good however the Curl command seems incorrect.

Your command as following:

# curl -k -i -vvvv --negotiate -u : "http://<knox1>:50070/gateway/<cluster_name>/webhdfs/v1/user?=op=LISTSTATUS"

.

In the above command we see that you are using "http://<knox1>:50070". (which is a direct port 50070 for HDFS namenode) ideally you should be hitting the Knox port. Also as knox is a trusted proxy hence you do not need to use "--negotiate" in the curl command (as long as you have not configured knox for SPNEGO authentication)

Hence you should try with the following kind of commands:

As knox by default runs on "https" (instead of "http" and it';s default port is 8443 (instead of 50070) hence polease try this. Also please pass knox LDAP credentials.

# curl -k -i -vvvv -u guest:guest-password  "https://<knox1>:8443/gateway/<topology_name>/webhdfs/v1/user?=op=LISTSTATUS"

Example:
-----------
# curl -k -i -vvvv  -u guest:guest-password  "https://<knox1>:8443/gateway/defaqult/webhdfs/v1/user?=op=LISTSTATUS"

.

Here the <topolofy_name> should be your knox topology name like "default"

[#3]

Why its using LDAP?LDAP is not setup on my cluster.I am using KDC. 

@JayKumarSharma

Also i have done the configuration in admin topology so i am using now admin instead of default in my URL.
[hdfs@<knox1> ~]$ curl -k -i -vvvv  -u guest:guest-password  "https://<knox>:8443/gateway/default/webhdfs/v1/user?=op=LISTSTATUS"
* About to connect() to <knox> port 8443 (#0)
*   Trying <knoxIP>... connected
* Connected to <knox> (<knoxIP>) port 8443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* warning: ignoring value of ssl.verifyhost
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*     subject: CN=<knox>,OU=Test,O=Hadoop,L=Test,ST=Test,C=US
*     start date: Oct 22 16:16:52 2018 GMT
*     expire date: Oct 22 16:16:52 2019 GMT
*     common name: <knox>
*     issuer: CN=<knox>,OU=Test,O=Hadoop,L=Test,ST=Test,C=US
* Server auth using Basic with user 'guest'
> GET /gateway/default/webhdfs/v1/user?=op=LISTSTATUS HTTP/1.1
> Authorization: Basic Z3Vlc3Q6Z3Vlc3QtcGFzc3dvcmQ=
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: <knox>:8443
> Accept: */*
>
< HTTP/1.1 401 Unauthorized
HTTP/1.1 401 Unauthorized
< Date: Wed, 24 Oct 2018 06:04:23 GMT
Date: Wed, 24 Oct 2018 06:04:23 GMT
< Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Tue, 23-Oct-2018 06:04:23 GMT
Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Tue, 23-Oct-2018 06:04:23 GMT
* Authentication problem. Ignoring this.
< WWW-Authenticate: BASIC realm="application"
WWW-Authenticate: BASIC realm="application"
< Content-Length: 0
Content-Length: 0
< Server: Jetty(9.2.15.v20160210)
Server: Jetty(9.2.15.v20160210)


<
* Connection #0 to host <knox> left intact
* Closing connection #0
[hdfs@dev-p76-app-01 ~]$
<br>