Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

KDC noRealm error on KDC-AD setup

KDC noRealm error on KDC-AD setup

New Contributor

Hi.

I am triying to setup a kerberized cluster against a AD2012r2 and it fails with error due a simpel bind problem.

Ambari-server.log

19 Feb 2018 11:05:48,109  WARN [ambari-client-thread-42] ADKerberosOperationHandler:470 - Failed to communicate with the Active Directory at ldaps://windc12.domain.tld:636: simple bind failed: windc12.domain.tld:636
javax.naming.CommunicationException: simple bind failed: windc12.domain.tld:636 [Root exception is javax.net.ssl.SSLException: java.lang.RuntimeException:
 Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty]
        at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:219)
        at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2788)
        at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319)
        at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192)
        at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210)
        at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153)
        at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83)
        at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
        at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313)
        at javax.naming.InitialContext.init(InitialContext.java:244)
        at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154)
        at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createInitialLdapContext(ADKerberosOperationHandler.java:514)
        at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createLdapContext(ADKerberosOperationHandler.java:465)
        at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.open(ADKerberosOperationHandler.java:182)
        at org.apache.ambari.server.controller.KerberosHelperImpl.validateKDCCredentials(KerberosHelperImpl.java:1901)
        at org.apache.ambari.server.controller.KerberosHelperImpl.handle(KerberosHelperImpl.java:2027)
        at org.apache.ambari.server.controller.KerberosHelperImpl$$EnhancerByGuice$$1fa97c03.CGLIB$handle$0(<generated>)
        at org.apache.ambari.server.controller.KerberosHelperImpl$$EnhancerByGuice$$1fa97c03$$FastClassByGuice$$3f1a93b8.invoke(<generated>)
        at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:118)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
        at org.apache.ambari.server.controller.KerberosHelperImpl$$EnhancerByGuice$$1fa97c03.handle(<generated>)
        at org.apache.ambari.server.controller.KerberosHelperImpl.toggleKerberos(KerberosHelperImpl.java:228)
        at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateCluster(AmbariManagementControllerImpl.java:1949)
        at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateClusters(AmbariManagementControllerImpl.java:1521)
        at org.apache.ambari.server.controller.AmbariManagementControllerImpl$$EnhancerByGuice$$8e25cb18.CGLIB$updateClusters$47(<generated>)
        at org.apache.ambari.server.controller.AmbariManagementControllerImpl$$EnhancerByGuice$$8e25cb18$$FastClassByGuice$$16893f3a.invoke(<generated>)
        at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:128)
        at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
        at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
        at org.apache.ambari.server.controller.AmbariManagementControllerImpl$$EnhancerByGuice$$8e25cb18.updateClusters(<generated>)
        at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:313)
        at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:310)
        at org.apache.ambari.server.controller.internal.AbstractResourceProvider.invokeWithRetry(AbstractResourceProvider.java:455)
        at org.apache.ambari.server.controller.internal.AbstractResourceProvider.modifyResources(AbstractResourceProvider.java:336)
        at org.apache.ambari.server.controller.internal.ClusterResourceProvider.updateResourcesAuthorized(ClusterResourceProvider.java:310)
        at org.apache.ambari.server.controller.internal.AbstractAuthorizedResourceProvider.updateResources(AbstractAuthorizedResourceProvider.java:301)
        at org.apache.ambari.server.controller.internal.ClusterControllerImpl.updateResources(ClusterControllerImpl.java:319)
        at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.update(PersistenceManagerImpl.java:125)
        at org.apache.ambari.server.api.handlers.UpdateHandler.persist(UpdateHandler.java:45)
        at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:73)
        at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:126)
        at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:90)
        at org.apache.ambari.server.api.services.ClusterService.updateCluster(ClusterService.java:142)
        at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)

TCPDUMP betwen windc12 (10.25.8.24) and hadoop-1 (10.43.30.35)

1. Exchange of certificates

No.     Time           Source                Destination           Protocol Length Info
     27 369.395533     10.43.30.35           10.25.8.24            TLSv1.2  341    Client Hello
Frame 27: 341 bytes on wire (2728 bits), 341 bytes captured (2728 bits) on interface 0
Ethernet II, Src: Fortinet_09:00:06 (00:09:0f:09:00:06), Dst: Vmware_99:e7:af (00:50:56:99:e7:af)
Internet Protocol Version 4, Src: 10.43.30.35, Dst: 10.25.8.24
Transmission Control Protocol, Src Port: 33482, Dst Port: 636, Seq: 1, Ack: 1, Len: 275
    Source Port: 33482
    Destination Port: 636
    [Stream index: 2]
    [TCP Segment Len: 275]
    Sequence number: 1    (relative sequence number)
    [Next sequence number: 276    (relative sequence number)]
    Acknowledgment number: 1    (relative ack number)
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window size value: 229
    [Calculated window size: 29312]
    [Window size scaling factor: 128]
    Checksum: 0x8e5c [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
    TCP payload (275 bytes)
Secure Sockets Layer
    TLSv1.2 Record Layer: Handshake Protocol: Client Hello
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 270
        Handshake Protocol: Client Hello
            Handshake Type: Client Hello (1)
            Length: 266
            Version: TLS 1.2 (0x0303)
            Random: 5a87095c50d1d156ccb93b473da7f0580f40edfc3a53d42f...
                GMT Unix Time: Feb 16, 2018 17:39:56.000000000 Romance Standard Time
                Random Bytes: 50d1d156ccb93b473da7f0580f40edfc3a53d42f5ac230d7...
            Session ID Length: 0
            Cipher Suites Length: 100
            Cipher Suites (50 suites)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 (0xc026)
                Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 (0xc02a)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x006b)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 (0x006a)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
                Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA (0xc005)
                Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA (0xc00f)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 (0xc025)
                Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 (0xc029)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x0067)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 (0x0040)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
                Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA (0xc004)
                Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA (0xc00e)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
                Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02e)
                Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 (0xc032)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 (0x00a3)
                Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
                Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02d)
                Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 (0xc031)
                Cipher Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
                Cipher Suite: TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 (0x00a2)
                Cipher Suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc008)
                Cipher Suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (0xc012)
                Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
                Cipher Suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc003)
                Cipher Suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA (0xc00d)
                Cipher Suite: TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016)
                Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)
                Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
            Compression Methods Length: 1
            Compression Methods (1 method)
            Extensions Length: 125
            Extension: supported_groups (len=52)
                Type: supported_groups (10)
                Length: 52
                Supported Groups List Length: 50
                Supported Groups (25 groups)
            Extension: ec_point_formats (len=2)
                Type: ec_point_formats (11)
                Length: 2
                EC point formats Length: 1
                Elliptic curves point formats (1)
            Extension: signature_algorithms (len=28)
                Type: signature_algorithms (13)
                Length: 28
                Signature Hash Algorithms Length: 26
                Signature Hash Algorithms (13 algorithms)
            Extension: server_name (len=27)
                Type: server_name (0)
                Length: 27
                Server Name Indication extension
                    Server Name list length: 25
                    Server Name Type: host_name (0)
                    Server Name length: 22
                    Server Name: windc12.knockout.local
No.     Time           Source                Destination           Protocol Length Info
     28 369.399674     10.25.8.24            10.43.30.35           TLSv1.2  1993   Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
Frame 28: 1993 bytes on wire (15944 bits), 1993 bytes captured (15944 bits) on interface 0
Ethernet II, Src: Vmware_99:e7:af (00:50:56:99:e7:af), Dst: Fortinet_09:00:06 (00:09:0f:09:00:06)
Internet Protocol Version 4, Src: 10.25.8.24, Dst: 10.43.30.35
Transmission Control Protocol, Src Port: 636, Dst Port: 33482, Seq: 1, Ack: 276, Len: 1927
    Source Port: 636
    Destination Port: 33482
    [Stream index: 2]
    [TCP Segment Len: 1927]
    Sequence number: 1    (relative sequence number)
    [Next sequence number: 1928    (relative sequence number)]
    Acknowledgment number: 276    (relative ack number)
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window size value: 513
    [Calculated window size: 131328]
    [Window size scaling factor: 256]
    Checksum: 0x3a85 [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
    TCP payload (1927 bytes)
Secure Sockets Layer
    TLSv1.2 Record Layer: Handshake Protocol: Multiple Handshake Messages
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 1922
        Handshake Protocol: Server Hello
            Handshake Type: Server Hello (2)
            Length: 77
            Version: TLS 1.2 (0x0303)
            Random: 5a87095c08ab7ab8058b2d4454dfdd8fb292a021bb3c4cf7...
                GMT Unix Time: Feb 16, 2018 17:39:56.000000000 Romance Standard Time
                Random Bytes: 08ab7ab8058b2d4454dfdd8fb292a021bb3c4cf7aa0db8e2...
            Session ID Length: 32
            Session ID: 0b4200002bacc095c1fa5c4c54eb43a8ba2e34064517b1ea...
            Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
            Compression Method: null (0)
            Extensions Length: 5
            Extension: renegotiation_info (len=1)
                Type: renegotiation_info (65281)
                Length: 1
                Renegotiation Info extension
        Handshake Protocol: Certificate
            Handshake Type: Certificate (11)
            Length: 1470
            Certificates Length: 1467
            Certificates (1467 bytes)
        Handshake Protocol: Server Key Exchange
            Handshake Type: Server Key Exchange (12)
            Length: 329
            EC Diffie-Hellman Server Params
        Handshake Protocol: Certificate Request
            Handshake Type: Certificate Request (13)
            Length: 26
            Certificate types count: 3
            Certificate types (3 types)
            Signature Hash Algorithms Length: 18
            Signature Hash Algorithms (9 algorithms)
            Distinguished Names Length: 0
        Handshake Protocol: Server Hello Done
            Handshake Type: Server Hello Done (14)
            Length: 0

2. Then, weird fatal error

No.     Time           Source                Destination           Protocol Length Info
     31 369.530739     10.43.30.35           10.25.8.24            TLSv1.2  73     Alert (Level: Fatal, Description: Internal Error)
Frame 31: 73 bytes on wire (584 bits), 73 bytes captured (584 bits) on interface 0
Ethernet II, Src: Fortinet_09:00:06 (00:09:0f:09:00:06), Dst: Vmware_99:e7:af (00:50:56:99:e7:af)
Internet Protocol Version 4, Src: 10.43.30.35, Dst: 10.25.8.24
Transmission Control Protocol, Src Port: 33482, Dst Port: 636, Seq: 276, Ack: 1928, Len: 7
    Source Port: 33482
    Destination Port: 636
    [Stream index: 2]
    [TCP Segment Len: 7]
    Sequence number: 276    (relative sequence number)
    [Next sequence number: 283    (relative sequence number)]
    Acknowledgment number: 1928    (relative ack number)
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window size value: 273
    [Calculated window size: 34944]
    [Window size scaling factor: 128]
    Checksum: 0xe52a [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
    TCP payload (7 bytes)
Secure Sockets Layer
    TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Internal Error)
        Content Type: Alert (21)
        Version: TLS 1.2 (0x0303)
        Length: 2
        Alert Message
            Level: Fatal (2)
            Description: Internal Error (80)

3. This seems to be the culprit a "couple of tcp packets later" > noRealm send

No.     Time           Source                Destination           Protocol Length Info
     38 413.210817     10.43.30.35           10.25.8.24            KRB5     225    AS-REQ
Frame 38: 225 bytes on wire (1800 bits), 225 bytes captured (1800 bits) on interface 0
Ethernet II, Src: Fortinet_09:00:06 (00:09:0f:09:00:06), Dst: Vmware_99:e7:af (00:50:56:99:e7:af)
Internet Protocol Version 4, Src: 10.43.30.35, Dst: 10.25.8.24
Transmission Control Protocol, Src Port: 37748, Dst Port: 88, Seq: 1, Ack: 1, Len: 159
    Source Port: 37748
    Destination Port: 88
    [Stream index: 3]
    [TCP Segment Len: 159]
    Sequence number: 1    (relative sequence number)
    [Next sequence number: 160    (relative sequence number)]
    Acknowledgment number: 1    (relative ack number)
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window size value: 229
    [Calculated window size: 29312]
    [Window size scaling factor: 128]
    Checksum: 0xcd8e [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
    TCP payload (159 bytes)
    [PDU Size: 159]
Kerberos
    Record Mark: 155 bytes
        0... .... .... .... .... .... .... .... = Reserved: Not set
        .000 0000 0000 0000 0000 0000 1001 1011 = Record Length: 155
    as-req
        pvno: 5
        msg-type: krb-as-req (10)
        req-body
            Padding: 0
            kdc-options: 00000020 (disable-transited-check)
            cname
                name-type: kRB5-NT-PRINCIPAL (1)
                cname-string: 1 item
                    CNameString: noUser
            realm: noRealm
            sname
                name-type: kRB5-NT-SRV-INST (2)
                sname-string: 2 items
                    SNameString: krbtgt
                    SNameString: noRealm
            from: 2018-02-16 16:40:40 (UTC)
            till: 2018-02-17 00:40:40 (UTC)
            nonce: 3125110255
            etype: 4 items
                ENCTYPE: eTYPE-AES256-CTS-HMAC-SHA1-96 (18)
                ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17)
                ENCTYPE: eTYPE-DES3-CBC-SHA1 (16)
                ENCTYPE: eTYPE-DES-CBC-MD5 (3)
No.     Time           Source                Destination           Protocol Length Info
     39 413.210966     10.25.8.24            10.43.30.35           KRB5     156    KRB Error: KDC_ERR_WRONG_REALM
Frame 39: 156 bytes on wire (1248 bits), 156 bytes captured (1248 bits) on interface 0
Ethernet II, Src: Vmware_99:e7:af (00:50:56:99:e7:af), Dst: Fortinet_09:00:06 (00:09:0f:09:00:06)
Internet Protocol Version 4, Src: 10.25.8.24, Dst: 10.43.30.35
Transmission Control Protocol, Src Port: 88, Dst Port: 37748, Seq: 1, Ack: 160, Len: 90
    Source Port: 88
    Destination Port: 37748
    [Stream index: 3]
    [TCP Segment Len: 90]
    Sequence number: 1    (relative sequence number)
    [Next sequence number: 91    (relative sequence number)]
    Acknowledgment number: 160    (relative ack number)
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window size value: 513
    [Calculated window size: 131328]
    [Window size scaling factor: 256]
    Checksum: 0x3aff [unverified]
    [Checksum Status: Unverified]
    Urgent pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
    TCP payload (90 bytes)
    [PDU Size: 90]
Kerberos
    Record Mark: 86 bytes
        0... .... .... .... .... .... .... .... = Reserved: Not set
        .000 0000 0000 0000 0000 0000 0101 0110 = Record Length: 86
    krb-error
        pvno: 5
        msg-type: krb-error (30)
        stime: 2018-02-16 16:40:40 (UTC)
        susec: 71498
        error-code: eRR-WRONG-REALM (68)
        realm: noRealm
        sname
            name-type: kRB5-NT-SRV-INST (2)
            sname-string: 2 items
                SNameString: krbtgt
                SNameString: noRealm

4. even when the krb5.conf file on hadoop-1 is properly configured:

[libdefaults]
  renew_lifetime = 7d
  forwardable = true
  default_realm = DOMAIN
  ticket_lifetime = 24h
  dns_lookup_realm = true
  dns_lookup_kdc = true
  default_ccache_name = /tmp/krb5cc_%{uid}
  #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
  #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[domain_realm]
  DOMAIN =  DOMAIN.LOCAL
[logging]
  default = FILE:/var/log/krb5kdc.log
  admin_server = FILE:/var/log/kadmind.log
  kdc = FILE:/var/log/krb5kdc.log
[realms]
  DOMAIN.TLD = {
    master_kdc = windc12.domain.tld
    admin_server = windc12.domain.tld
    kdc = windc12.domain.tld
   }

5. ....and DNS entries are right from the same hadoop-1 machine.

; <<>> DiG 9.9.4-RedHat-9.9.4-51.el7_4.2 <<>> SRV _ldap._tcp.domain.tld
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49372
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 6
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;_ldap._tcp.domain.tld. IN      SRV     
;; ANSWER SECTION:
_ldap._tcp.domain.tld. 600      IN      SRV     0 100 389 windc13.domain.tld.
_ldap._tcp.domain.tld. 600      IN      SRV     0 100 389 windc16.domain.tld.
_ldap._tcp.domain.tld. 600      IN      SRV     0 100 389 windc20.domain.tld.
_ldap._tcp.domain.tld. 600      IN      SRV     0 100 389 windc11.domain.tld.
_ldap._tcp.domain.tld. 600      IN      SRV     0 100 389 windc12.domain.tld.
;; ADDITIONAL SECTION:
windc13.domain.tld.     3600    IN      A       10.25.153.3
windc16.domain.tld.     3600    IN      A       10.41.40.4
windc20.domain.tld.     3600    IN      A       10.35.8.20
windc11.domain.tld.     3600    IN      A       10.25.8.23
windc12.domain.tld.     3600    IN      A       10.25.8.24
;; Query time: 136 msec
;; SERVER: 10.25.8.23#53(10.25.8.23)
;; WHEN: lun feb 19 12:23:36 CET 2018
;; MSG SIZE  rcvd: 344
<<< PRO4 >>> root@hadoop-1:/var/log/ambari-server# dig SRV _kerberos._tcp.domain.tld
; <<>> DiG 9.9.4-RedHat-9.9.4-51.el7_4.2 <<>> SRV _kerberos._tcp.domain.tld
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45966
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 6
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;_kerberos._tcp.domain.tld.     IN      SRV
;; ANSWER SECTION:
_kerberos._tcp.domain.tld. 600 IN       SRV     0 100 88 windc11.domain.tld.
_kerberos._tcp.domain.tld. 600 IN       SRV     0 100 88 windc16.domain.tld.
_kerberos._tcp.domain.tld. 600 IN       SRV     0 100 88 windc12.domain.tld.
_kerberos._tcp.domain.tld. 600 IN       SRV     0 100 88 windc20.domain.tld.
_kerberos._tcp.domain.tld. 600 IN       SRV     0 100 88 windc13.domain.tld.
;; ADDITIONAL SECTION:
windc11.domain.tld.     3600    IN      A       10.25.8.23
windc16.domain.tld.     3600    IN      A       10.41.40.4
windc12.domain.tld.     3600    IN      A       10.25.8.24
windc20.domain.tld.     3600    IN      A       10.35.8.20
windc13.domain.tld.     3600    IN      A       10.25.153.3
;; Query time: 130 msec
;; SERVER: 10.25.8.23#53(10.25.8.23)
;; WHEN: lun feb 19 12:23:43 CET 2018

############################################################################################################################################################################################################

What i did to setup the enviroment

  • windc was configured with LDAPS and enabled certification authority using this guide
  • openssl s_client -connect windc.domain.tld:636 -showcerts retrieve certificate

openssl s_client -connect windc12.DOMAIN.TLD:636 -showcertsCONNECTED(00000003)
depth=1 DC = TLD, DC = domain, CN = WINDC12-CA-1
verify return:1
depth=0 
verify return:1
---
Certificate chain
 0 s:
   i:/DC=tld/DC=domain/CN=WINDC12-CA-1
-----BEGIN CERTIFICATE-----
<SANITIZED>
-----END CERTIFICATE-----
---
Server certificate
subject=
issuer=/DC=tld/DC=domain/CN=WINDC12-CA-1
---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA512:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA512:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1
Peer signing digest: SHA1
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2034 bytes and written 495 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-SHA384
    Session-ID: 0014000092CC522ED518B2129168F67BF<SANITIZED>00B442B09077697C45B44
    Session-ID-ctx: 
    Master-Key: DCF5ED4CA2D89ADD1E84A1B6A89F82C38755669<SANITIZED>8FA32EC53A18F3D434EEDF45BC4977A34B704
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1519037714
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
  • kinit from the hadoo-ambari with the same credentials username@DOMAIN.TLD is successfull and i got a ticket.
Using default cache: /tmp/krb5cc_0
Using principal: ko-hadoop@DOMAIN.TLD
Password for ko-hadoop@DOMAIN.TLD: 
Authenticated to Kerberos v5
  • klist from the hadoop-ambari server shows the ticket against the DC/KDC
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: ko-hadoop@DOMAIN.TLD
Valid starting     Expires            Service principal
19/02/18 10:20:33  19/02/18 20:20:33  krbtgt/DOMAIN.TLD@DOMAIN.TLD
	renew until 26/02/18 10:20:27

############################################################################################################################################################################################################

Any ideas???

Thanks!

3 REPLIES 3

Re: KDC noRealm error on KDC-AD setup

It appears the culprit is

javax.naming.CommunicationException: simple bind failed: windc12.domain.tld:636 [Root exception is javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty]

My guess is that your JVM is not happy with the ciphers used in the AD's certificate. What version of Java are you using and is the unlimited key JCE policy installed? I suggest using a later version of JDK 1.8 as ion JDK 1.7, ECDHE-RSA-AES256-SHA384 may not be fully supported - see https://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html.

You can try to disable cert validation when enabling Kerberos by setting kerberos.operation.verify.kdc.trust to "false" in the ambari.properties file, but I think the issue is related to the certificate exchange with happens before the verification process.

Re: KDC noRealm error on KDC-AD setup

Guru

Just to add to @Robert Levas's answer:

Looks like the Ambari JVM is running without AD certificate in JVM's truststore. Please import AD certificate in <JAVA_HOME>/lib/security/cacerts store of the JVM used by Ambari and then restart Ambari server.

Hope this helps.

Highlighted

Re: KDC noRealm error on KDC-AD setup

New Contributor

Hi.

Yes, indeed was a problem from the certificate.

What i did to fix it:

  1. get the certificate from the AD using the java tool
  2. import all certificates from jssecacerts to cacerts
  3. import all certificates from cacerts to ambari trustore
  4. set ambari trustore
  5. check it with keytool

Then it worked.

Thanks!

Don't have an account?
Coming from Hortonworks? Activate your account here