Support Questions

Meepoljd · ‎08-07-2023

I have a secure hadoop cluster with HDP3.1，I recently tried to interconnect this cluster with the Knox component to implement a secure proxy. This cluster has kerberos, Ldap, and https enabled, I create one config like this:

<topology>
    <gateway>
        <provider>
            <role>authentication</role>
            <name>ShiroProvider</name>
            <enabled>true</enabled>
            <param>
                <name>main.ldapRealm</name>
                <value>org.apache.knox.gateway.shirorealm.KnoxLdapRealm</value>
            </param>
            <param>
                <name>main.ldapContextFactory</name>
                <value>org.apache.knox.gateway.shirorealm.KnoxLdapContextFactory</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory</name>
                <value>$ldapContextFactory</value>
            </param>
            <param>
                <name>main.ldapRealm.userDnTemplate</name>
                <value>cn=admin,dc=datasw,dc=com</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory.url</name>
                <value>ldap://hdp001.datasw.com:389</value>
            </param>
            <param>
                <name>main.ldapRealm.contextFactory.authenticationMechanism</name>
                <value>simple</value>
            </param>
            <param>
                <name>urls./**</name>
                <value>authcBasic</value>
            </param>
        </provider>
        <provider>
            <role>authentication</role>
            <name>HadoopAuth</name>
            <enabled>true</enabled>
            <param>
                <name>config.prefix</name>
                <value>hadoop.auth.config</value>
            </param>
            <param>
                <name>hadoop.auth.config.type</name>
                <value>kerberos</value>
            </param>
            <param>
                <name>hadoop.auth.config.simple.anonymous.allowed</name>
                <value>false</value>
            </param>
            <param>
                <name>hadoop.auth.config.token.validity</name>
                <value>1800</value>
            </param>
            <param>
                <name>hadoop.auth.config.cookie.domain</name>
                <value>datasw.com</value>
            </param>
            <param>
                <name>hadoop.auth.config.cookie.path</name>
                <value>gateway/default</value>
            </param>
            <param>
                <name>hadoop.auth.config.kerberos.principal</name>
                <value>HTTP/hdp003.datasw@DATASW.COM</value>
            </param>
            <param>
                <name>hadoop.auth.config.kerberos.keytab</name>
                <value>/etc/security/keytabs/spnego.service.keytab</value>
            </param>
            <param>
                <name>hadoop.auth.config.kerberos.name.rules</name>
                <value>DEFAULT</value>
            </param>
            <param>
                <name>fs.defaultFS</name>
                <value>hdfs://hdfsCluster</value>
            </param>
            <param>
                <name>dfs.internal.nameservices</name>
                <value>hdfsCluster</value>
            </param>
            <param>
                <name>dfs.ha.namenodes.hdfsCluster</name>
                <value>nn1,nn2</value>
            </param>
            <param>
                <name>dfs.nameservices</name>
                <value>hdfsCluster</value>
            </param>
            <param>
                <name>dfs.namenode.https-address</name>
                <value>hdp001.datasw:50470</value>
            </param>
            <param>
                <name>dfs.namenode.https-address.hdfsCluster.nn1</name>
                <value>hdp001.datasw:50470</value>
            </param>
            <param>
                <name>dfs.namenode.https-address.hdfsCluster.nn2</name>
                <value>hdp002.datasw:50470</value>
            </param>
        </provider>
    </gateway>
    <service>
        <role>HDFSUI</role>
        <url>https://hdp002.datasw.com:50470</url>
    </service>
</topology>

and I copy the hadoop cluster's truststore.jks file to the $GATEWAY_HOME/data/security/keystores/ and set

gateway.httpclient.truststore.path param in gateway-stie.xml:

    <property>
        <name>gateway.httpclient.truststore.path</name>
        <value>/usr/local/knox/data/security/keystores/truststore.jks</value>
    </property>
    <property>
        <name>gateway.httpclient.truststore.type</name>
        <value>JKS</value>
    </property>
    <property>
        <name>gateway.httpclient.truststore.password.alias</name>
        <value>pthdp</value>
    </property>

Then I restart the Knox gateway，but when I access the NameNode webUi, I receive the following error message:

2023-08-08 11:14:38,050 58fc3dbf-4c6e-4684-860d-0a4e443f85d2 WARN  knox.gateway (DefaultDispatch.java:executeOutboundRequest(183)) - Connection exception dispatching request: https://hdp002.datasw.com:50470/?user.name=admin javax.net.ssl.SSLPeerUnverifiedException: Certificate for <hdp002.datasw.com> doesn't match any of the subject alternative names: []
javax.net.ssl.SSLPeerUnverifiedException: Certificate for <hdp002.datasw.com> doesn't match any of the subject alternative names: []
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.verifyHostname(SSLConnectionSocketFactory.java:507) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:437) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.13.jar:4.5.13]
        at org.apache.knox.gateway.dispatch.DefaultDispatch.executeOutboundRequest(DefaultDispatch.java:166) ~[gateway-spi-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.dispatch.DefaultDispatch.executeRequest(DefaultDispatch.java:152) ~[gateway-spi-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.dispatch.DefaultDispatch.executeRequestWrapper(DefaultDispatch.java:135) ~[gateway-spi-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.dispatch.DefaultDispatch.doGet(DefaultDispatch.java:300) ~[gateway-spi-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:183) ~[gateway-spi-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:127) ~[gateway-spi-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:58) ~[gateway-spi-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:377) ~[gateway-server-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:291) ~[gateway-server-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:193) ~[gateway-provider-identity-assertion-common-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.access$000(AbstractIdentityAssertionFilter.java:55) ~[gateway-provider-identity-assertion-common-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter$1.run(AbstractIdentityAssertionFilter.java:161) ~[gateway-provider-identity-assertion-common-2.0.0.jar:2.0.0]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_291]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_291]
        at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.doAs(AbstractIdentityAssertionFilter.java:156) ~[gateway-provider-identity-assertion-common-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:146) ~[gateway-provider-identity-assertion-common-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.identityasserter.common.filter.CommonIdentityAssertionFilter.doFilter(CommonIdentityAssertionFilter.java:241) ~[gateway-provider-identity-assertion-common-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:377) ~[gateway-server-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:291) ~[gateway-server-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:57) ~[gateway-provider-rewrite-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:58) ~[gateway-spi-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:377) ~[gateway-server-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:291) ~[gateway-server-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:93) ~[gateway-provider-security-shiro-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain$1.run(ShiroSubjectIdentityAdapter.java:90) ~[gateway-provider-security-shiro-2.0.0.jar:2.0.0]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_291]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_291]
        at org.apache.knox.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:146) ~[gateway-provider-security-shiro-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.filter.ShiroSubjectIdentityAdapter$CallableChain.call(ShiroSubjectIdentityAdapter.java:76) ~[gateway-provider-security-shiro-2.0.0.jar:2.0.0]
        at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) ~[shiro-core-1.10.0.jar:1.10.0]
        at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) ~[shiro-core-1.10.0.jar:1.10.0]
        at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:387) ~[shiro-core-1.10.0.jar:1.10.0]
        at org.apache.knox.gateway.filter.ShiroSubjectIdentityAdapter.doFilter(ShiroSubjectIdentityAdapter.java:73) ~[gateway-provider-security-shiro-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:377) ~[gateway-server-2.0.0.jar:2.0.0]
        at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:291) ~[gateway-server-2.0.0.jar:2.0.0]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:61) ~[shiro-web-1.10.0.jar:1.10.0]
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) ~[shiro-web-1.10.0.jar:1.10.0]
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) ~[shiro-web-1.10.0.jar:1.10.0]
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) ~[shiro-web-1.10.0.jar:1.10.0]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) ~[shiro-web-1.10.0.jar:1.10.0]
        at org.apache.shiro.web.servlet.AdviceFilter.executeChain(AdviceFilter.java:108) ~[shiro-web-1.10.0.jar:1.10.0]
        at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:137) ~[shiro-web-1.10.0.jar:1.10.0]
        at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:154) ~[shiro-web-1.10.0.jar:1.10.0]
        at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) ~[shiro-web-1.10.0.jar:1.10.0]
        at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:458) ~[shiro-web-1.10.0.jar:1.10.0]
        at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:373) ~[shiro-web-1.10.0.jar:1.10.0]

In order to achieve Knox proxy, What else do I need to do?

Scharan · ‎09-06-2023

Hi @Meepoljd

It seems like the issue is with the certificate. Make sure the certificate is created with fqdn.

The curl command is working because, as you passing --insecure, it ignores invalid and self-signed certificate errors.

View solution in original post

Scharan · ‎08-10-2023

Hello @Meepoljd

Are you able to access the Namnode UI without a Knox proxy?

Can you check the output of the below command to verify the hostname in the certificate?

# openssl s_client -connect hdp002.datasw.com:50470 -showcerts

Meepoljd · ‎08-10-2023

Hi, Scharan, thisi is the command's return:

[root@hdp002 ~]# openssl s_client -connect hdp002.datasw.com:50470 -showcerts
CONNECTED(00000003)
depth=1 C = CN, ST = ShenZhen, L = GuangDong, O = DATASW, OU = PlatformTeam, CN = datsw
verify error:num=19:self signed certificate in certificate chain
---
Certificate chain
 0 s:/C=CN/ST=GuangDong/L=ShenZhen/O=DATASW/OU=PlatformTeam/CN=hdp002.datasw
   i:/C=CN/ST=ShenZhen/L=GuangDong/O=DATASW/OU=PlatformTeam/CN=datsw
-----BEGIN CERTIFICATE-----
MIIDXDCCAkQCCQDNmxfKgcxaOjANBgkqhkiG9w0BAQsFADBsMQswCQYDVQQGEwJD
TjERMA8GA1UECAwIU2hlblpoZW4xEjAQBgNVBAcMCUd1YW5nRG9uZzEPMA0GA1UE
CgwGREFUQVNXMRUwEwYDVQQLDAxQbGF0Zm9ybVRlYW0xDjAMBgNVBAMMBWRhdHN3
MB4XDTIxMDkxNDE3MjcwM1oXDTMxMDkxMjE3MjcwM1owdDELMAkGA1UEBhMCQ04x
EjAQBgNVBAgTCUd1YW5nRG9uZzERMA8GA1UEBxMIU2hlblpoZW4xDzANBgNVBAoT
BkRBVEFTVzEVMBMGA1UECxMMUGxhdGZvcm1UZWFtMRYwFAYDVQQDEw1oZHAwMDIu
ZGF0YXN3MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAgUquEQOPlyZs
vqfNZCB6qXh+wDoQlOMTPWfoHAtk5c041LwDrtkjF3I+uXntqcV/aB0ufSLA3M/j
2iiPt+8sosjp0cOakWXpOGx3Vv7MXteF/c8HMav+9yheY8qcoHfrRwPqpq9v0Ysz
31v1l6zJiuubF9JJpYMMmBkfwd0RUyLD079VkKAHtq6zAlpBm+zKVc6B0Xiddvw6
La2PB/c+vqVGVKsI+0RqiMdM1IXCuV76CT47riQ8G0PPs1OSu4HVI+J1j6P3R6He
rpIG1GstmcGjbR10qo/MDg1svwxiGOJxw1sN+65LQPLB3cTw4JmUMIJq8F1REiAZ
azP8I1oTsQIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQA4B1NwbQ0jBBccOfX0zCoQ
mHiIQvpMiiUYJRTAfUqueIFMwRF5aw85u2PAMnkgxqUmRG87RTgOg5dx3hPswS83
FQOeG2kUPXIR9HRaJZirRZ0vZByzsHl77N4Bb0y0Z0m1svbSczxH/RCEY5L7NvMR
fUSZjYW8LFk8nl3tH60u4f7cDJ+bTiDA37B7tUqvwrRHsvSJY9Ndp52QRUHxST3o
iutBSEOWfhBvPkv7U/B3WAnT9Dp0aSur2jIr5BP99qs13cy4qK0h5OAB+lgCEBt5
L4JKqWOv3W33j4vtNeYOSzfHgoi2JgKDP+imoVgnGeK1GNJsHMxecw4ef0Eik9sW
-----END CERTIFICATE-----
 1 s:/C=CN/ST=ShenZhen/L=GuangDong/O=DATASW/OU=PlatformTeam/CN=datsw
   i:/C=CN/ST=ShenZhen/L=GuangDong/O=DATASW/OU=PlatformTeam/CN=datsw
-----BEGIN CERTIFICATE-----
MIIDqzCCApOgAwIBAgIJAMmdxf5CbH3BMA0GCSqGSIb3DQEBCwUAMGwxCzAJBgNV
BAYTAkNOMREwDwYDVQQIDAhTaGVuWmhlbjESMBAGA1UEBwwJR3VhbmdEb25nMQ8w
DQYDVQQKDAZEQVRBU1cxFTATBgNVBAsMDFBsYXRmb3JtVGVhbTEOMAwGA1UEAwwF
ZGF0c3cwHhcNMjEwOTE0MDE1MTQwWhcNMzEwOTEyMDE1MTQwWjBsMQswCQYDVQQG
EwJDTjERMA8GA1UECAwIU2hlblpoZW4xEjAQBgNVBAcMCUd1YW5nRG9uZzEPMA0G
A1UECgwGREFUQVNXMRUwEwYDVQQLDAxQbGF0Zm9ybVRlYW0xDjAMBgNVBAMMBWRh
dHN3MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAuDsAyDscHIGXRmTz
EKzWjenR8c2f6hpLEyRTqvlI9AzTd5gZRMMWg7ax4erC7BPwva+RHZoeug9kE2HC
UHoGCP4YIZdEux5phPqv1vP/CBvbXnYZ4olMRSJDuf57TpAZjMTy5FHgs7QDzpCk
9Ez7CQWeXaaAaqnGo8SUWLATLadudSgkPDLSJL/h2IGjhxKPMyaHODxXQRxUIRbr
tI+8+9+siRLi+3EIhMXLT1oEOnsB/BQmawbNjyLtuZZoH8pyGJ3ByoM06zLWWMGI
eujFCOlSRMMzpEr/xLhJQQDFBLEFJKYOD0Z5QbqgrFkQtLOFpEnDbTXasQvlk4on
nLVemQIDAQABo1AwTjAdBgNVHQ4EFgQUUkBUUrsXSVF7MQDnB0hKjtpiOt0wHwYD
VR0jBBgwFoAUUkBUUrsXSVF7MQDnB0hKjtpiOt0wDAYDVR0TBAUwAwEB/zANBgkq
hkiG9w0BAQsFAAOCAQEAdU7iAr/F5lCLvPMfo9LA7JhI7IQdic/EhxELLuUELF7c
UBOOlJbWFxLYaZ6SwZ9lGa4d+wjNWoX+QvLt02PGZV3h0aB6O8E0827jjgI61r0C
UNSD3N3KadbK52st5W34sIssXqBNIga1w9knfWouiqNcHBixyZdYfWOwGLPSAbpC
K4os4yi5QU4YSvNwLO9GAYgem0p0Uel9By3m0cFmyFr+GcA+VAWltk7xBKOsCxam
nnQJE+djbMekmXW6cmujbqh02Q6LF0/6wNDMRnRFkDvF5WnT1XxQ7O+HFkeQXPED
qCkcKcHLqMxhK72iVlLgCq6n+oYLDxeODfHEjvo3sg==
-----END CERTIFICATE-----
---
Server certificate
subject=/C=CN/ST=GuangDong/L=ShenZhen/O=DATASW/OU=PlatformTeam/CN=hdp002.datasw
issuer=/C=CN/ST=ShenZhen/L=GuangDong/O=DATASW/OU=PlatformTeam/CN=datsw
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2350 bytes and written 483 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-SHA384
    Session-ID: 64D5CE575ACD383C1B9BED92D5F2FDC1C63308098FD241173411E62C2E5E0395
    Session-ID-ctx: 
    Master-Key: D73511C7D981C1A2F7813E02F102BD23057A5A79C5E9E75C3BCE870AA40D7CE4F02E41115F28510CE7AF85C6F6675BE6
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1691733590
    Timeout   : 300 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---

KIT is not installed on my Windows machine, so I use curl on the Linux server to access the http request of the namenode:

curl -i --insecure --negotiate -u: "https://hdp002.datasw.com:50470/jmx?qry=Hadoop:service=NameNode,name=RpcActivityForPort8020"

Scharan · ‎09-06-2023

Hi @Meepoljd

It seems like the issue is with the certificate. Make sure the certificate is created with fqdn.

The curl command is working because, as you passing --insecure, it ignores invalid and self-signed certificate errors.

Cloudera Community

Support Questions

Use Knox proxy Secure Hadoop cluster has some question(doesn't match any of the subject alternative names)