Knox to HS2 gives response 401

[#1]

Hi everyone,

I am facing an issue that bruns my brain for a couple of days; hope you will help me managing this.

I have a Hive Server 2 running in HTTP mode with Kerberos and I can connect well using beeline from another server of my cluster using bellow JDBC URi (after getting a Kerberos ticket with kinit):

jdbc:hive2://my_hive_server:10011/;principal=myprincipal/hostname@domain;transportMode=http;httpPath=cliservice

The problem comes when I try to connect to this Hive Server through Knox with LDAP user credentials; it gives me response 401 error. I have tried many configurations found on this community site and googling, but without success.

Same issue occurs using curl command:

curl -iv -k -u myuser:mypasswd -X GET 'https://my_knox_hostname:9443/gateway/default/hive/?op=LISTSTATUS'

* Server auth using Basic with user 'myuser'

> GET /gateway/default/hive/?op=LISTSTATUS HTTP/1.1

> Authorization: Basic WDExMTExNTpoYWhhaGE=

> User-Agent: curl/7.29.0 > Host: my_knox_hostname:9443

> Accept: */* >

< HTTP/1.1 401 Unauthorized

HTTP/1.1 401 Unauthorized

< Date: Mon, 04 Jun 2018 07:46:45 GMT Date: Mon, 04 Jun 2018 07:46:45 GMT

< Set-Cookie: JSESSIONID=5v2868pq8l6m1mc3lt5u6l156;Path=/gateway/default;Secure;HttpOnly

Set-Cookie: JSESSIONID=5v2868pq8l6m1mc3lt5u6l156;Path=/gateway/default;Secure;HttpOnly

< Expires: Thu, 01 Jan 1970 00:00:00 GMT

Expires: Thu, 01 Jan 1970 00:00:00 GMT

< Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Sun, 03-Jun-2018 07:46:45 GMT

Set-Cookie: rememberMe=deleteMe; Path=/gateway/default; Max-Age=0; Expires=Sun, 03-Jun-2018 07:46:45 GMT

< Server: Jetty(7.6.0.v20120127)

Server: Jetty(7.6.0.v20120127)

< Content-Length: 69

Content-Length: 69

<

Authentication Error: java.lang.reflect.UndeclaredThrowableException

* Connection #0 to host my_knox_hostname left intact

Some help would be appreciated; thnak you in advance.

Regards.

[#2]

@Felix Albani I don't have any logs on my hive server. My guess is that connection passes Knox gateway but don't go through Hive that asks some credentials that the POST statement don't deliver.

Here are my Knox gateway logs when trying to reach my Hive server with mentionned beeline:

2018-05-24 10:32:51,330 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: POST /hive
2018-05-24 10:32:51,382 DEBUG hadoop.gateway (KnoxLdapRealm.java:getUserDn(718)) - Searching from dc=domain,dc=realm where (&(objectclass=posixAccount)(uid=myuser)) scope subtree
2018-05-24 10:32:51,387 INFO  hadoop.gateway (KnoxLdapRealm.java:getUserDn(724)) - Computed userDn: uid=myuser,ou=Users,dc=domain,dc=realm using ldapSearch for principal: myuser
2018-05-24 10:32:51,441 INFO  hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
2018-05-24 10:32:51,442 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: https://my_knox_hostname:9443/gateway/default/hive, direction: IN via implicit rule: HIVE/hive/inbound to URL: http://my_http_hive_hostname:10011/cliservice
2018-05-24 10:32:51,443 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: POST http://my_http_hive_hostname:10011/cliservice?doAs=myuser
2018-05-24 10:32:51,461 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 401
2018-05-24 10:32:51,461 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(209)) - Inbound response entity content type not provided.
2018-05-24 10:32:51,471 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: POST /hive
2018-05-24 10:32:51,472 INFO  hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
2018-05-24 10:32:51,472 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: https://my_knox_hostname:9443/gateway/default/hive, direction: IN via implicit rule: HIVE/hive/inbound to URL: http://my_http_hive_hostname:10011/cliservice
2018-05-24 10:32:51,473 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: POST http://my_http_hive_hostname:10011/cliservice?doAs=myuser
2018-05-24 10:32:51,488 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 401
2018-05-24 10:32:51,488 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(209)) - Inbound response entity content type not provided.
2018-05-24 10:32:51,524 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: POST /hive
2018-05-24 10:32:51,575 DEBUG hadoop.gateway (KnoxLdapRealm.java:getUserDn(718)) - Searching from dc=domain,dc=realm where (&(objectclass=posixAccount)(uid=myuser)) scope subtree
2018-05-24 10:32:51,579 INFO  hadoop.gateway (KnoxLdapRealm.java:getUserDn(724)) - Computed userDn: uid=myuser,ou=Users,dc=domain,dc=realm using ldapSearch for principal: myuser
2018-05-24 10:32:51,631 INFO  hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
2018-05-24 10:32:51,632 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: https://my_knox_hostname:9443/gateway/default/hive, direction: IN via implicit rule: HIVE/hive/inbound to URL: http://my_http_hive_hostname:10011/cliservice
2018-05-24 10:32:51,633 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: POST http://my_http_hive_hostname:10011/cliservice?doAs=myuser
2018-05-24 10:32:51,648 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 401
2018-05-24 10:32:51,649 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(209)) - Inbound response entity content type not provided.
2018-05-24 10:32:51,658 DEBUG hadoop.gateway (GatewayFilter.java:doFilter(116)) - Received request: POST /hive
2018-05-24 10:32:51,659 INFO  hadoop.gateway (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
2018-05-24 10:32:51,660 DEBUG hadoop.gateway (UrlRewriteProcessor.java:rewrite(164)) - Rewrote URL: https://my_knox_hostname:9443/gateway/default/hive, direction: IN via implicit rule: HIVE/hive/inbound to URL: http://my_http_hive_hostname:10011/cliservice
2018-05-24 10:32:51,661 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: POST http://my_http_hive_hostname:10011/cliservice?doAs=myuser
2018-05-24 10:32:51,678 DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 401
2018-05-24 10:32:51,678 DEBUG hadoop.gateway (DefaultDispatch.java:getInboundResponseContentType(209)) - Inbound response entity content type not provided.
<br>

[#3]

@Pirlouis Pirlouis I can see 401 is coming from hiveserver2. Hive is not authorizing the access to /cliservice?doAs=myuser

DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: POST http://my_http_hive_hostname:10011/cliservice?doAs=myuser 
DEBUG hadoop.gateway (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response status: 401

Few things to check:

1. Check ambari > hdfs > configs> core-site.xml contains > hadoop.proxyuser.knox.groups=* and hadoop.proxyuser.knox.hosts=*

2. Check ambari > hive > configs > Custom webhcat-site > webhcat.proxyuser.knox.groups=* and webhcat.proxyuser.knox.hosts=*

3. Enable debug on hiveserver2, hopefully this will print the reason behind the 401 error being returned by hs2

4. Finally, in my experience this has helped. On knox host run the following command to see the html response coming back from hs2:

tcpdump -A port 10011

HTH

[#4]

@Felix Albani

1 - I confirm these settings; just to let you know that hadoop.proxyuser.knox.hosts is not set as star "*" but with some hostnames. But I confirm my Knox hostname is there.

2 - For a personal purpose, I actually don't manage hive-site content with Ambari as I am using a particular software that have its own hive-site.xml. As I can see, webhcat settings are not written into this file so I've asked the software editor how I can handle this.

3 - How can I activate debug for HS2, please ?

4 - Can you please tell me more on how tu use this ? When I type this command on my Knox hostname, I have:

$ tcpdump -A port 10011
tcpdump: WARNING: SIOCGIFADDR: nflog: No such device
tcpdump: NFLOG link-layer type filtering not implemented

Thank you.

[#5]

@Felix Albani any news regarding my last answer, please ?

[#6]

@Felix Albani is their a relation between the WebHcat server and Hive Server or Know server ? I still don't understand how these components are related to each other ?

Thank you.

[#7]

@Pirlouis Pirlouis As I mentioned you should check hiveserver2 to find more clues as why we are getting 401. Since you mentioned the hive-site.xml is not managed by ambari - I highly suspect the proxy users for webhcat could be missing and perhaps leading to this problem. To enable debug on hiveserver2 just set root logger to debug on the hive-log4j.properties