Acces Hive Via Knox Gatewy URL

[#1]

Hi, My environment is kerberized. So we would like to acces hive db via knox gateway setup. I did my nitial setup in sandbox to test the possibliity. But I am facing some issue.

Setup I did :

#1. hive --> ssl=true, sasl.qop=true,trasportmode=http;

#2. knox --> added hive services(it was there already)

and I tried following, I got the error :

jdbc:hive2://sandbox-hdp.hortonworks.com:8443/;ssl=true;sslTrustStore=/var/lib/knox/data/security/keystores/gateway.jks;trustStorePassword=knox;hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice

error :

[root@sandbox-hdp ~]# beeline -u "jdbc:hive2://sandbox-hdp.hortonworks.com:8443/;ssl=true;sslTrustStore=/var/lib/knox/data/security/keystores/gateway.jks;trustStorePassword=knox;hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice" 18/07/25 02:06:09 [main]: WARN jdbc.HiveConnection: Failed to connect to sandbox-hdp.hortonworks.com:8443 Error: Could not open client transport with JDBC Uri: jdbc:hive2://sandbox-hdp.hortonworks.com:8443/;ssl=true;sslTrustStore=/var/lib/knox/data/security/keystores/gateway.jks;trustStorePassword=knox;hive.server2.transport.mode=http;hive.server2.thrift.http.path=cliservice: Error creating the transport (state=08S01,code=0) Beeline version 1.2.1000.2.6.5.0-292 by Apache Hive 0: jdbc:hive2://sandbox-hdp.hortonworks.com:8 (closed)>

can you pls help me on this? where I am going wrong.

[#2]

I tried the following url now : Error: Could not establish connection to jdbc:hive2://sandbox-hdp.hortonworks.com:8443/default;ssl=true;sslTrustStore=/var/lib/knox/data-2.6.5.0-292/security/keystores/gateway.jks;trustStorePassword=knox;transportMode=http;httpPath=gateway/default/hive: HTTP Response code: 401 (state=08S01,code=0) Beeline version 1.2.1000.2.6.5.0-292 by Apache Hive 0: jdbc:hive2://sandbox-hdp.hortonworks.com:8 (closed)>

and its failing with response code error as 401. I think its authentication related issue. what I am doing wrong here. can anyone help me pls?

regads

Ashok

[#3]

What do you see in gateway.log ? Also, is your sandbox is kerberized ? if not check whether demo ldap is running, sandbox might be using demo ldap.

[#4]

Hi Sandeep,

Yes. My Sandbox is kerberized and start ldapdemo and which is running. My Gateway-audit log says its 'LDAP Authentication issue' when I tried to access hive via knox. Do you think I am missing LDAP sync with ambari? If so how do i do it? attached gateway-audit.log & gateway.log.

gateway-audit.log:

18/07/25 15:41:48 ||c3c9515a-9cf6-4d0e-bcc5-9432234fdabb|audit|<IP-ADDR>|HIVE|guest|||authentication|uri|/gateway/default/hive|success|

18/07/25 15:41:48 ||c3c9515a-9cf6-4d0e-bcc5-9432234fdabb|audit|<IP-ADDR>|HIVE|guest|||authentication|uri|/gateway/default/hive|success|Groups: []

18/07/25 15:41:48 ||c3c9515a-9cf6-4d0e-bcc5-9432234fdabb|audit|<IP-ADDR>|HIVE|guest|||authorization|uri|/gateway/default/hive|success|

18/07/25 15:41:48 ||c3c9515a-9cf6-4d0e-bcc5-9432234fdabb|audit|<IP-ADDR>|HIVE|guest|||dispatch|uri|http://ip-<IP-ADDR>.ec2.internal:10001/cliservice?doAs=guest|unavailable|Request method: POST

18/07/25 15:41:48 ||c3c9515a-9cf6-4d0e-bcc5-9432234fdabb|audit|<IP-ADDR>|HIVE|guest|||dispatch|uri|http://ip-<IP-ADDR>.ec2.internal:10001/cliservice?doAs=guest|failure|

Gateway :

2018-07-25 17:07:16,160 ERROR hadoop.gateway (KnoxLdapRealm.java:doGetAuthenticationInfo(205)) - Shiro unable to login: javax.naming.CommunicationException: ip-<IP-ADDR>.ec2.internal:33389 [Root exception is java.net.ConnectException: Connection refused (Connection refused)] 2018-07-25 20:34:10,369 INFO hadoop.gateway (KnoxLdapRealm.java:getUserDn(691)) - Computed userDn: uid=anonymous,ou=people,dc=hadoop,dc=apache,dc=org using dnTemplate for principal: anonymous 2018-07-25 20:34:10,370 INFO hadoop.gateway (KnoxLdapRealm.java:doGetAuthenticationInfo(203)) - Could not login: org.apache.shiro.authc.UsernamePasswordToken - anonymous, rememberMe=false (<IP-ADDR>) 2018-07-25 20:34:10,371 ERROR hadoop.gateway (KnoxLdapRealm.java:doGetAuthenticationInfo(205)) - Shiro unable to login: javax.naming.CommunicationException: ip-<IP-ADDR>.ec2.internal:33389 [Root exception is java.net.ConnectException: Connection refused (Connection refused)]

regards

Ashokkumar.R

[#5]

And One more question Sandeep. To access hive via knox, hive must syncup with LDAP ?

[#6]

If you are using kerberos then you do not need LDAP, this looks like an issue with your configuration.

This is an example

https://community.hortonworks.com/articles/192759/knox-with-kerberos-authentication-to-proxy-to-hive...

The topology file is not formatted properly but you can compare it with yours and add the necessary configuration.

[#7]

Thanks Sandeep. I followed steps, When I am validating the topolgy file I am facing below error :

cmd I run : knoxcli.sh --d system-user-auth-test --cluster <clustername>

error :

/usr/hdp/current/knox-server/bin/knoxcli.sh --d system-user-auth-test --cluster knoxpocsetup Warn: main.ldapRealm.contextFactory.systemUsername is not present in topology Warn: main.ldapRealm.contextFactory.systemUsername is not present in topology main.ldapRealm.userSearchAttributeName or main.ldapRealm.userObjectClass or main.ldapRealm.searchBase or main.ldapRealm.userSearchBase was found in the topology If any one of the above params is present then main.ldapRealm.userSearchAttributeName and main.ldapRealm.userObjectClass must both be present and either main.ldapRealm.searchBase or main.ldapRealm.userSearchBase must also be present. Topology warnings present. SystemUser may not bind. org.apache.shiro.authc.AuthenticationException: Authentication failed for token submission [org.apache.shiro.authc.UsernamePasswordToken - null, rememberMe=false]. Possible unexpected error? (Typical or expected login exceptions should extend from AuthenticationException). principal argument cannot be null. org.apache.shiro.authc.AuthenticationException: Authentication failed for token submission [org.apache.shiro.authc.UsernamePasswordToken - null, rememberMe=false]. Possible unexpected error? (Typical or expected login exceptions should extend from AuthenticationException).

regards

Ashokkumar.R

[#8]

Sandeep,

I didnt understand this part in your document? What doesit mean? adding principal

[root@groot1 hive]# kinit dvillarreal Password for dvillarreal@SUPPORT.COM:

I have only default.xml file, do I need to rename it validate it?

regards

Ashokkumar.R

[#9]

to add more info : this is my default.xml file under /usr/hdp/current/knox-server/conf/.

<topology>


            <gateway>


                <provider>
                    <role>authentication</role>
                    <name>ShiroProvider</name>
                    <enabled>true</enabled>
                    <param>
                        <name>sessionTimeout</name>
                        <value>30</value>
                    </param>
                    <param>
                        <name>main.ldapRealm</name>
                        <value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
                    </param>
                    <param>
                        <name>main.ldapRealm.userDnTemplate</name>
                        <value>uid={0},ou=people,dc=hadoop,dc=apache,dc=org</value>
                    </param>
                    <param>
                        <name>main.ldapRealm.contextFactory.url</name>
                        <value>ldap://sandbox-hdp.hortonworks.com:33389</value>
                    </param>
                    <param>
                        <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
                        <value>simple</value>
                    </param>
                    <param>
                        <name>urls./**</name>
                        <value>authcBasic</value>
                    </param>
                </provider>


                <provider>
                    <role>identity-assertion</role>
                    <name>Default</name>
                    <enabled>true</enabled>
                </provider>


                <provider>
                    <role>authorization</role>
                    <name>XASecurePDPKnox</name>
                    <enabled>true</enabled>
                </provider>


            </gateway>


            <service>
                <role>NAMENODE</role>
                <url>hdfs://sandbox-hdp.hortonworks.com:8020</url>
            </service>


            <service>
                <role>JOBTRACKER</role>
                <url>rpc://sandbox-hdp.hortonworks.com:8032</url>
            </service>


            <service>
                <role>WEBHDFS</role>
                <url>http://sandbox-hdp.hortonworks.com:50070/webhdfs</url>
   <service>
                <role>WEBHCAT</role>
                <url>http://sandbox-hdp.hortonworks.com:50111/templeton</url>
            </service>


            <service>
                <role>OOZIE</role>
                <url>http://sandbox-hdp.hortonworks.com:11000/oozie</url>
            </service>


            <service>
                <role>WEBHBASE</role>
                <url>http://sandbox-hdp.hortonworks.com:8080</url>
            </service>


            <service>
                <role>HIVE</role>
                <url>http://sandbox-hdp.hortonworks.com:10001/cliservice</url>
            </service>


            <service>
                <role>RESOURCEMANAGER</role>
                <url>http://sandbox-hdp.hortonworks.com:8088/ws</url>
            </service>
        </topology>

[#10]

Ashok, do you find any solution for this issue? I am getting the same error while connecting hive through Knox gateway