Hi. This is my setup: 3 kafka hosts with 2 networks private internal network for data public for connection with everything else +--------------------------------------------------------------------------------------------------+ | | | 10.25.12.0/24 - PUBLIC | | | | --------------------------------------------------------------- | | | | | | | | | | | | | | | | | +------------+ +------------+ +------------+ | | | | | | | | | | | | | | | | | | | KFK-1 | | KFK-2 | | KFK-3 | | | | | | | | | | | | | | | | | | | +------------+ +------------+ +------------+ | | | | | | | - | | | | | | | | | | | | | | --------------------------------------------------------------- | | 192.168.160.0/24 - PRIVATE | | server.properties | | advertised.listeners=PLAINTEXT://10.25.12.[1,2,3]:6667 | | listeners=PLAINTEXT://0.0.0.0:6667 | | inter.broker.listener.name=PLAINTEXT://192.168.160.[1,2,3] | | zookeeper.connect=kfk-1.domain.tld:2181,kfk-2.domain.tld:2181,kfk-3.domain.tld:2181 | | | +--------------------------------------------------------------------------------------------------+ Every host, to use the data network is routed through the hosts file to the internal one: #INTERNAL NETWORK 192.168.160.1 kfk-1.domain.tld 192.168.160.2 kfk-2.domain.tld 192.168.160.3 kfk-3.domain.tld The problem: If i try to connect from the outside to a round-robin hostname which contains all the 3 public private addresses of the kafka-brokes using zookeeper (kafka.domain.tld:2181) > works If i try to connect to ANY of the brokers directly using bootstrat-server > doesnt even throw me and error. I tried placing on the advertised.listeners: plaintext://kfk-[1,2,3].domain.tld:6667 >> connect but can not find the leader plaintext://0.0.0.0:6667 >> dont connect plaintext://kafka.domain.tld:6667 >> dont connect Running HDP 2.6.1. I am ran out of ideas.....any clue? ... View more

Hi guys. I am facing a strange problems. I have a 3 node kafka 0.10 cluster and for one machine one specific paritition i got always this errors, producing the LAG on the partition between the source cluster and the destination cluster is increasing. [2018-03-25 17:56:57,398] WARN [ConsumerFetcherThread-KafkaMirror_hdp-dw-1-nn-1.domain.local-1521821432752-5248d2cb-0-1001], Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@74958add (kafka.consumer.ConsumerFetcherThread) java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:122) at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:114) at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:99) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:148) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:148) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:148) at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:147) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:147) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:147) at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:146) at kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:111) at kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:30) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) {metadata.broker.list=dcluster-3.pro2.domain.local:6667,dcluster-1.pro2.domain.local:6667,dcluster-2.pro2.domain.local:6667, request.timeout.ms=30000, client.id=KafkaMirror-0, security.protocol=PLAINTEXT} [2018-03-25 17:57:28,334] WARN [ConsumerFetcherThread-KafkaMirror_hdp-dw-1-nn-1.domain.local-1521821432752-5248d2cb-0-1001], Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@4acd8894 (kafka.consumer.ConsumerFetcherThread) java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:122) at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:114) at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:99) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:148) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:148) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:148) at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:147) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:147) at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:147) at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:146) at kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:111) at kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:30) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63) {metadata.broker.list=dcluster-3.pro2.domain.local:6667,dcluster-1.pro2.domain.local:6667,dcluster-2.pro2.domain.local:6667, request.timeout.ms=30000, client.id=KafkaMirror-0, security.protocol=PLAINTEXT} The other nodes on the cluster works perfect, and the faulty node on another topics, works perfect. Only partition 0 from topic ¨transactions¨ ... View more

Hi. We are running a hadoop 2.6.1 cluster and we are configured two networks: a) bond WAN> 10.25.0.0/16 b) bond DATA > 192.168.160.0/24 The hadoop nodes has one network interface with the same last octect in every network. The hostname is on the host files of the nodes. It is impossible to make the kafka work. I got always this message when i try to connect to the kafka producer broker from another machine. Firewall is ok. [2018-03-15 15:09:50,924] ERROR Error when sending message to topic kafka_mirror_test with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) it is suited Kafka to receive all the client request from a interface and do the inter broker network from another interface? It is a problem with zookeeper? HDP 2.6.1 running all the services (hdfs, hive, kafka, pig, zookeeper, etc) in every node. ... View more

Hi. Maybe this question has been managed but after struggling a week with kerberos i can not find the right way to do it: Enviroment: 3-node-PoC Hadoop cluster (hadoop-1. 2 and 3) Kerberized against windows AD 2012r2 What is working: Inside the nodes, it is possible to consume and produce messages to a topic created using > ./bin/kafka-console-consumer.sh/producer.sh --zookeeper hadoop-1.poc.domain.tld:2181 --topic test_topic --from-beginning --security-protocol PLAINTEXTSASL I know that if i provide the kafka/hadoop-X.poc.domain.tld to anynone and then he runs a kinit -kt keytab, should work but this is not a secure way to do so. I want to create new users or even better, grant to AD users the possibility to using is own credentials and kerberos, access the kafka brokers and being authorized by Ranger. Ranger is running but at the moment it is connected only to the unix account: has to be connected to AD to get this working? Sorry if this question is one like "how i can be rich" but i am a little lost. I managed to using this guide create a kafkapro4 user and i copied the keytab to the hadoop-X nodes on the right folder. But when i want to run the kinit -kt keytab ServiceAccout, it answer me: kinit -kt kafkapro.service.keytab kafkapro/hadoop-1.poc.domain.local@domain.LOCAL kinit: Client 'kafkapro4/hadoop-1.poc.domain.local@DOMAIN.LOCAL' not found in Kerberos database while getting initial credentials So i am quite a bit lost how to manage it. ... View more

Hi. I am triying to setup a kerberized cluster against a AD2012r2 and it fails with error due a simpel bind problem. Ambari-server.log 19 Feb 2018 11:05:48,109 WARN [ambari-client-thread-42] ADKerberosOperationHandler:470 - Failed to communicate with the Active Directory at ldaps://windc12.domain.tld:636: simple bind failed: windc12.domain.tld:636 javax.naming.CommunicationException: simple bind failed: windc12.domain.tld:636 [Root exception is javax.net.ssl.SSLException: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty] at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:219) at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2788) at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192) at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210) at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153) at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83) at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684) at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313) at javax.naming.InitialContext.init(InitialContext.java:244) at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154) at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createInitialLdapContext(ADKerberosOperationHandler.java:514) at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createLdapContext(ADKerberosOperationHandler.java:465) at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.open(ADKerberosOperationHandler.java:182) at org.apache.ambari.server.controller.KerberosHelperImpl.validateKDCCredentials(KerberosHelperImpl.java:1901) at org.apache.ambari.server.controller.KerberosHelperImpl.handle(KerberosHelperImpl.java:2027) at org.apache.ambari.server.controller.KerberosHelperImpl$$EnhancerByGuice$$1fa97c03.CGLIB$handle$0(<generated>) at org.apache.ambari.server.controller.KerberosHelperImpl$$EnhancerByGuice$$1fa97c03$$FastClassByGuice$$3f1a93b8.invoke(<generated>) at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:118) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72) at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52) at org.apache.ambari.server.controller.KerberosHelperImpl$$EnhancerByGuice$$1fa97c03.handle(<generated>) at org.apache.ambari.server.controller.KerberosHelperImpl.toggleKerberos(KerberosHelperImpl.java:228) at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateCluster(AmbariManagementControllerImpl.java:1949) at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateClusters(AmbariManagementControllerImpl.java:1521) at org.apache.ambari.server.controller.AmbariManagementControllerImpl$$EnhancerByGuice$$8e25cb18.CGLIB$updateClusters$47(<generated>) at org.apache.ambari.server.controller.AmbariManagementControllerImpl$$EnhancerByGuice$$8e25cb18$$FastClassByGuice$$16893f3a.invoke(<generated>) at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72) at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:128) at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72) at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52) at org.apache.ambari.server.controller.AmbariManagementControllerImpl$$EnhancerByGuice$$8e25cb18.updateClusters(<generated>) at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:313) at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:310) at org.apache.ambari.server.controller.internal.AbstractResourceProvider.invokeWithRetry(AbstractResourceProvider.java:455) at org.apache.ambari.server.controller.internal.AbstractResourceProvider.modifyResources(AbstractResourceProvider.java:336) at org.apache.ambari.server.controller.internal.ClusterResourceProvider.updateResourcesAuthorized(ClusterResourceProvider.java:310) at org.apache.ambari.server.controller.internal.AbstractAuthorizedResourceProvider.updateResources(AbstractAuthorizedResourceProvider.java:301) at org.apache.ambari.server.controller.internal.ClusterControllerImpl.updateResources(ClusterControllerImpl.java:319) at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.update(PersistenceManagerImpl.java:125) at org.apache.ambari.server.api.handlers.UpdateHandler.persist(UpdateHandler.java:45) at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:73) at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144) at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:126) at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:90) at org.apache.ambari.server.api.services.ClusterService.updateCluster(ClusterService.java:142) at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) TCPDUMP betwen windc12 (10.25.8.24) and hadoop-1 (10.43.30.35) 1. Exchange of certificates No. Time Source Destination Protocol Length Info 27 369.395533 10.43.30.35 10.25.8.24 TLSv1.2 341 Client Hello Frame 27: 341 bytes on wire (2728 bits), 341 bytes captured (2728 bits) on interface 0 Ethernet II, Src: Fortinet_09:00:06 (00:09:0f:09:00:06), Dst: Vmware_99:e7:af (00:50:56:99:e7:af) Internet Protocol Version 4, Src: 10.43.30.35, Dst: 10.25.8.24 Transmission Control Protocol, Src Port: 33482, Dst Port: 636, Seq: 1, Ack: 1, Len: 275 Source Port: 33482 Destination Port: 636 [Stream index: 2] [TCP Segment Len: 275] Sequence number: 1 (relative sequence number) [Next sequence number: 276 (relative sequence number)] Acknowledgment number: 1 (relative ack number) 1000 .... = Header Length: 32 bytes (8) Flags: 0x018 (PSH, ACK) Window size value: 229 [Calculated window size: 29312] [Window size scaling factor: 128] Checksum: 0x8e5c [unverified] [Checksum Status: Unverified] Urgent pointer: 0 Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps [SEQ/ACK analysis] TCP payload (275 bytes) Secure Sockets Layer TLSv1.2 Record Layer: Handshake Protocol: Client Hello Content Type: Handshake (22) Version: TLS 1.2 (0x0303) Length: 270 Handshake Protocol: Client Hello Handshake Type: Client Hello (1) Length: 266 Version: TLS 1.2 (0x0303) Random: 5a87095c50d1d156ccb93b473da7f0580f40edfc3a53d42f... GMT Unix Time: Feb 16, 2018 17:39:56.000000000 Romance Standard Time Random Bytes: 50d1d156ccb93b473da7f0580f40edfc3a53d42f5ac230d7... Session ID Length: 0 Cipher Suites Length: 100 Cipher Suites (50 suites) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028) Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d) Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 (0xc026) Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 (0xc02a) Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x006b) Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 (0x006a) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014) Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035) Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA (0xc005) Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA (0xc00f) Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039) Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027) Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c) Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 (0xc025) Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 (0xc029) Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x0067) Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 (0x0040) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013) Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f) Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA (0xc004) Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA (0xc00e) Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033) Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c) Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030) Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d) Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02e) Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 (0xc032) Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f) Cipher Suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 (0x00a3) Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f) Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c) Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02d) Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 (0xc031) Cipher Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e) Cipher Suite: TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 (0x00a2) Cipher Suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc008) Cipher Suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (0xc012) Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a) Cipher Suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc003) Cipher Suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA (0xc00d) Cipher Suite: TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016) Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013) Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff) Compression Methods Length: 1 Compression Methods (1 method) Extensions Length: 125 Extension: supported_groups (len=52) Type: supported_groups (10) Length: 52 Supported Groups List Length: 50 Supported Groups (25 groups) Extension: ec_point_formats (len=2) Type: ec_point_formats (11) Length: 2 EC point formats Length: 1 Elliptic curves point formats (1) Extension: signature_algorithms (len=28) Type: signature_algorithms (13) Length: 28 Signature Hash Algorithms Length: 26 Signature Hash Algorithms (13 algorithms) Extension: server_name (len=27) Type: server_name (0) Length: 27 Server Name Indication extension Server Name list length: 25 Server Name Type: host_name (0) Server Name length: 22 Server Name: windc12.knockout.local No. Time Source Destination Protocol Length Info 28 369.399674 10.25.8.24 10.43.30.35 TLSv1.2 1993 Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done Frame 28: 1993 bytes on wire (15944 bits), 1993 bytes captured (15944 bits) on interface 0 Ethernet II, Src: Vmware_99:e7:af (00:50:56:99:e7:af), Dst: Fortinet_09:00:06 (00:09:0f:09:00:06) Internet Protocol Version 4, Src: 10.25.8.24, Dst: 10.43.30.35 Transmission Control Protocol, Src Port: 636, Dst Port: 33482, Seq: 1, Ack: 276, Len: 1927 Source Port: 636 Destination Port: 33482 [Stream index: 2] [TCP Segment Len: 1927] Sequence number: 1 (relative sequence number) [Next sequence number: 1928 (relative sequence number)] Acknowledgment number: 276 (relative ack number) 1000 .... = Header Length: 32 bytes (8) Flags: 0x018 (PSH, ACK) Window size value: 513 [Calculated window size: 131328] [Window size scaling factor: 256] Checksum: 0x3a85 [unverified] [Checksum Status: Unverified] Urgent pointer: 0 Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps [SEQ/ACK analysis] TCP payload (1927 bytes) Secure Sockets Layer TLSv1.2 Record Layer: Handshake Protocol: Multiple Handshake Messages Content Type: Handshake (22) Version: TLS 1.2 (0x0303) Length: 1922 Handshake Protocol: Server Hello Handshake Type: Server Hello (2) Length: 77 Version: TLS 1.2 (0x0303) Random: 5a87095c08ab7ab8058b2d4454dfdd8fb292a021bb3c4cf7... GMT Unix Time: Feb 16, 2018 17:39:56.000000000 Romance Standard Time Random Bytes: 08ab7ab8058b2d4454dfdd8fb292a021bb3c4cf7aa0db8e2... Session ID Length: 32 Session ID: 0b4200002bacc095c1fa5c4c54eb43a8ba2e34064517b1ea... Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028) Compression Method: null (0) Extensions Length: 5 Extension: renegotiation_info (len=1) Type: renegotiation_info (65281) Length: 1 Renegotiation Info extension Handshake Protocol: Certificate Handshake Type: Certificate (11) Length: 1470 Certificates Length: 1467 Certificates (1467 bytes) Handshake Protocol: Server Key Exchange Handshake Type: Server Key Exchange (12) Length: 329 EC Diffie-Hellman Server Params Handshake Protocol: Certificate Request Handshake Type: Certificate Request (13) Length: 26 Certificate types count: 3 Certificate types (3 types) Signature Hash Algorithms Length: 18 Signature Hash Algorithms (9 algorithms) Distinguished Names Length: 0 Handshake Protocol: Server Hello Done Handshake Type: Server Hello Done (14) Length: 0 2. Then, weird fatal error No. Time Source Destination Protocol Length Info 31 369.530739 10.43.30.35 10.25.8.24 TLSv1.2 73 Alert (Level: Fatal, Description: Internal Error) Frame 31: 73 bytes on wire (584 bits), 73 bytes captured (584 bits) on interface 0 Ethernet II, Src: Fortinet_09:00:06 (00:09:0f:09:00:06), Dst: Vmware_99:e7:af (00:50:56:99:e7:af) Internet Protocol Version 4, Src: 10.43.30.35, Dst: 10.25.8.24 Transmission Control Protocol, Src Port: 33482, Dst Port: 636, Seq: 276, Ack: 1928, Len: 7 Source Port: 33482 Destination Port: 636 [Stream index: 2] [TCP Segment Len: 7] Sequence number: 276 (relative sequence number) [Next sequence number: 283 (relative sequence number)] Acknowledgment number: 1928 (relative ack number) 1000 .... = Header Length: 32 bytes (8) Flags: 0x018 (PSH, ACK) Window size value: 273 [Calculated window size: 34944] [Window size scaling factor: 128] Checksum: 0xe52a [unverified] [Checksum Status: Unverified] Urgent pointer: 0 Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps [SEQ/ACK analysis] TCP payload (7 bytes) Secure Sockets Layer TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Internal Error) Content Type: Alert (21) Version: TLS 1.2 (0x0303) Length: 2 Alert Message Level: Fatal (2) Description: Internal Error (80) 3. This seems to be the culprit a "couple of tcp packets later" > noRealm send No. Time Source Destination Protocol Length Info 38 413.210817 10.43.30.35 10.25.8.24 KRB5 225 AS-REQ Frame 38: 225 bytes on wire (1800 bits), 225 bytes captured (1800 bits) on interface 0 Ethernet II, Src: Fortinet_09:00:06 (00:09:0f:09:00:06), Dst: Vmware_99:e7:af (00:50:56:99:e7:af) Internet Protocol Version 4, Src: 10.43.30.35, Dst: 10.25.8.24 Transmission Control Protocol, Src Port: 37748, Dst Port: 88, Seq: 1, Ack: 1, Len: 159 Source Port: 37748 Destination Port: 88 [Stream index: 3] [TCP Segment Len: 159] Sequence number: 1 (relative sequence number) [Next sequence number: 160 (relative sequence number)] Acknowledgment number: 1 (relative ack number) 1000 .... = Header Length: 32 bytes (8) Flags: 0x018 (PSH, ACK) Window size value: 229 [Calculated window size: 29312] [Window size scaling factor: 128] Checksum: 0xcd8e [unverified] [Checksum Status: Unverified] Urgent pointer: 0 Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps [SEQ/ACK analysis] TCP payload (159 bytes) [PDU Size: 159] Kerberos Record Mark: 155 bytes 0... .... .... .... .... .... .... .... = Reserved: Not set .000 0000 0000 0000 0000 0000 1001 1011 = Record Length: 155 as-req pvno: 5 msg-type: krb-as-req (10) req-body Padding: 0 kdc-options: 00000020 (disable-transited-check) cname name-type: kRB5-NT-PRINCIPAL (1) cname-string: 1 item CNameString: noUser realm: noRealm sname name-type: kRB5-NT-SRV-INST (2) sname-string: 2 items SNameString: krbtgt SNameString: noRealm from: 2018-02-16 16:40:40 (UTC) till: 2018-02-17 00:40:40 (UTC) nonce: 3125110255 etype: 4 items ENCTYPE: eTYPE-AES256-CTS-HMAC-SHA1-96 (18) ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17) ENCTYPE: eTYPE-DES3-CBC-SHA1 (16) ENCTYPE: eTYPE-DES-CBC-MD5 (3) No. Time Source Destination Protocol Length Info 39 413.210966 10.25.8.24 10.43.30.35 KRB5 156 KRB Error: KDC_ERR_WRONG_REALM Frame 39: 156 bytes on wire (1248 bits), 156 bytes captured (1248 bits) on interface 0 Ethernet II, Src: Vmware_99:e7:af (00:50:56:99:e7:af), Dst: Fortinet_09:00:06 (00:09:0f:09:00:06) Internet Protocol Version 4, Src: 10.25.8.24, Dst: 10.43.30.35 Transmission Control Protocol, Src Port: 88, Dst Port: 37748, Seq: 1, Ack: 160, Len: 90 Source Port: 88 Destination Port: 37748 [Stream index: 3] [TCP Segment Len: 90] Sequence number: 1 (relative sequence number) [Next sequence number: 91 (relative sequence number)] Acknowledgment number: 160 (relative ack number) 1000 .... = Header Length: 32 bytes (8) Flags: 0x018 (PSH, ACK) Window size value: 513 [Calculated window size: 131328] [Window size scaling factor: 256] Checksum: 0x3aff [unverified] [Checksum Status: Unverified] Urgent pointer: 0 Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps [SEQ/ACK analysis] TCP payload (90 bytes) [PDU Size: 90] Kerberos Record Mark: 86 bytes 0... .... .... .... .... .... .... .... = Reserved: Not set .000 0000 0000 0000 0000 0000 0101 0110 = Record Length: 86 krb-error pvno: 5 msg-type: krb-error (30) stime: 2018-02-16 16:40:40 (UTC) susec: 71498 error-code: eRR-WRONG-REALM (68) realm: noRealm sname name-type: kRB5-NT-SRV-INST (2) sname-string: 2 items SNameString: krbtgt SNameString: noRealm 4. even when the krb5.conf file on hadoop-1 is properly configured: [libdefaults] renew_lifetime = 7d forwardable = true default_realm = DOMAIN ticket_lifetime = 24h dns_lookup_realm = true dns_lookup_kdc = true default_ccache_name = /tmp/krb5cc_%{uid} #default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 #default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5 [domain_realm] DOMAIN = DOMAIN.LOCAL [logging] default = FILE:/var/log/krb5kdc.log admin_server = FILE:/var/log/kadmind.log kdc = FILE:/var/log/krb5kdc.log [realms] DOMAIN.TLD = { master_kdc = windc12.domain.tld admin_server = windc12.domain.tld kdc = windc12.domain.tld } 5. ....and DNS entries are right from the same hadoop-1 machine. ; <<>> DiG 9.9.4-RedHat-9.9.4-51.el7_4.2 <<>> SRV _ldap._tcp.domain.tld ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49372 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 6 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4000 ;; QUESTION SECTION: ;_ldap._tcp.domain.tld. IN SRV ;; ANSWER SECTION: _ldap._tcp.domain.tld. 600 IN SRV 0 100 389 windc13.domain.tld. _ldap._tcp.domain.tld. 600 IN SRV 0 100 389 windc16.domain.tld. _ldap._tcp.domain.tld. 600 IN SRV 0 100 389 windc20.domain.tld. _ldap._tcp.domain.tld. 600 IN SRV 0 100 389 windc11.domain.tld. _ldap._tcp.domain.tld. 600 IN SRV 0 100 389 windc12.domain.tld. ;; ADDITIONAL SECTION: windc13.domain.tld. 3600 IN A 10.25.153.3 windc16.domain.tld. 3600 IN A 10.41.40.4 windc20.domain.tld. 3600 IN A 10.35.8.20 windc11.domain.tld. 3600 IN A 10.25.8.23 windc12.domain.tld. 3600 IN A 10.25.8.24 ;; Query time: 136 msec ;; SERVER: 10.25.8.23#53(10.25.8.23) ;; WHEN: lun feb 19 12:23:36 CET 2018 ;; MSG SIZE rcvd: 344 <<< PRO4 >>> root@hadoop-1:/var/log/ambari-server# dig SRV _kerberos._tcp.domain.tld ; <<>> DiG 9.9.4-RedHat-9.9.4-51.el7_4.2 <<>> SRV _kerberos._tcp.domain.tld ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45966 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 6 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4000 ;; QUESTION SECTION: ;_kerberos._tcp.domain.tld. IN SRV ;; ANSWER SECTION: _kerberos._tcp.domain.tld. 600 IN SRV 0 100 88 windc11.domain.tld. _kerberos._tcp.domain.tld. 600 IN SRV 0 100 88 windc16.domain.tld. _kerberos._tcp.domain.tld. 600 IN SRV 0 100 88 windc12.domain.tld. _kerberos._tcp.domain.tld. 600 IN SRV 0 100 88 windc20.domain.tld. _kerberos._tcp.domain.tld. 600 IN SRV 0 100 88 windc13.domain.tld. ;; ADDITIONAL SECTION: windc11.domain.tld. 3600 IN A 10.25.8.23 windc16.domain.tld. 3600 IN A 10.41.40.4 windc12.domain.tld. 3600 IN A 10.25.8.24 windc20.domain.tld. 3600 IN A 10.35.8.20 windc13.domain.tld. 3600 IN A 10.25.153.3 ;; Query time: 130 msec ;; SERVER: 10.25.8.23#53(10.25.8.23) ;; WHEN: lun feb 19 12:23:43 CET 2018 ############################################################################################################################################################################################################ What i did to setup the enviroment windc was configured with LDAPS and enabled certification authority using this guide openssl s_client -connect windc.domain.tld:636 -showcerts retrieve certificate openssl s_client -connect windc12.DOMAIN.TLD:636 -showcertsCONNECTED(00000003) depth=1 DC = TLD, DC = domain, CN = WINDC12-CA-1 verify return:1 depth=0 verify return:1 --- Certificate chain 0 s: i:/DC=tld/DC=domain/CN=WINDC12-CA-1 -----BEGIN CERTIFICATE----- <SANITIZED> -----END CERTIFICATE----- --- Server certificate subject= issuer=/DC=tld/DC=domain/CN=WINDC12-CA-1 --- No client certificate CA names sent Client Certificate Types: RSA sign, DSA sign, ECDSA sign Requested Signature Algorithms: RSA+SHA512:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1 Shared Requested Signature Algorithms: RSA+SHA512:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1 Peer signing digest: SHA1 Server Temp Key: ECDH, P-256, 256 bits --- SSL handshake has read 2034 bytes and written 495 bytes --- New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES256-SHA384 Session-ID: 0014000092CC522ED518B2129168F67BF<SANITIZED>00B442B09077697C45B44 Session-ID-ctx: Master-Key: DCF5ED4CA2D89ADD1E84A1B6A89F82C38755669<SANITIZED>8FA32EC53A18F3D434EEDF45BC4977A34B704 Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1519037714 Timeout : 300 (sec) Verify return code: 0 (ok) --- kinit from the hadoo-ambari with the same credentials username@DOMAIN.TLD is successfull and i got a ticket. Using default cache: /tmp/krb5cc_0 Using principal: ko-hadoop@DOMAIN.TLD Password for ko-hadoop@DOMAIN.TLD: Authenticated to Kerberos v5 klist from the hadoop-ambari server shows the ticket against the DC/KDC Ticket cache: FILE:/tmp/krb5cc_0 Default principal: ko-hadoop@DOMAIN.TLD Valid starting Expires Service principal 19/02/18 10:20:33 19/02/18 20:20:33 krbtgt/DOMAIN.TLD@DOMAIN.TLD renew until 26/02/18 10:20:27 ############################################################################################################################################################################################################ Any ideas??? Thanks! ... View more