Member since
02-16-2018
9
Posts
2
Kudos Received
0
Solutions
05-15-2018
07:54 AM
1 Kudo
I have the same problem. A little of help here would be nice
... View more
03-26-2018
09:54 AM
I answer myself. It seems because of the 2 network bond (one for access and another for data), if you configure on the hosts files some hosts to avoid dns resolution, it doesnt work on the long run and throw you that error, which is strange. I removed the source kafka cluster from the host files and now it works smooth.
... View more
03-25-2018
04:05 PM
Finally we go without Kerberos. it is really a pain. We will research go for Knox with SSL and LDAP......i think i will live better rather to have such pain of Kerberos 🙂
... View more
03-25-2018
04:03 PM
Hi guys. I am facing a strange problems. I have a 3 node kafka 0.10 cluster and for one machine one specific paritition i got always this errors, producing the LAG on the partition between the source cluster and the destination cluster is increasing. [2018-03-25 17:56:57,398] WARN [ConsumerFetcherThread-KafkaMirror_hdp-dw-1-nn-1.domain.local-1521821432752-5248d2cb-0-1001], Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@74958add (kafka.consumer.ConsumerFetcherThread)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:122)
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:114)
at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:99)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:148)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:148)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:148)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:147)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:147)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:147)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:146)
at kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:111)
at kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:30)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
{metadata.broker.list=dcluster-3.pro2.domain.local:6667,dcluster-1.pro2.domain.local:6667,dcluster-2.pro2.domain.local:6667, request.timeout.ms=30000, client.id=KafkaMirror-0, security.protocol=PLAINTEXT}
[2018-03-25 17:57:28,334] WARN [ConsumerFetcherThread-KafkaMirror_hdp-dw-1-nn-1.domain.local-1521821432752-5248d2cb-0-1001], Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@4acd8894 (kafka.consumer.ConsumerFetcherThread)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:122)
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:114)
at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:99)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SimpleConsumer.scala:148)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:148)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1$$anonfun$apply$mcV$sp$1.apply(SimpleConsumer.scala:148)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply$mcV$sp(SimpleConsumer.scala:147)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:147)
at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply(SimpleConsumer.scala:147)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:146)
at kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:111)
at kafka.consumer.ConsumerFetcherThread.fetch(ConsumerFetcherThread.scala:30)
at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:118)
at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:103)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
{metadata.broker.list=dcluster-3.pro2.domain.local:6667,dcluster-1.pro2.domain.local:6667,dcluster-2.pro2.domain.local:6667, request.timeout.ms=30000, client.id=KafkaMirror-0, security.protocol=PLAINTEXT}
The other nodes on the cluster works perfect, and the faulty node on another topics, works perfect. Only partition 0 from topic ¨transactions¨
... View more
Labels:
- Labels:
-
Apache Kafka
02-23-2018
01:32 PM
1 Kudo
Hi. Maybe this question has been managed but after struggling a week with kerberos i can not find the right way to do it: Enviroment: 3-node-PoC Hadoop cluster (hadoop-1. 2 and 3) Kerberized against windows AD 2012r2 What is working: Inside the nodes, it is possible to consume and produce messages to a topic created using > ./bin/kafka-console-consumer.sh/producer.sh --zookeeper hadoop-1.poc.domain.tld:2181 --topic test_topic --from-beginning --security-protocol PLAINTEXTSASL I know that if i provide the kafka/hadoop-X.poc.domain.tld to anynone and then he runs a kinit -kt keytab, should work but this is not a secure way to do so. I want to create new users or even better, grant to AD users the possibility to using is own credentials and kerberos, access the kafka brokers and being authorized by Ranger. Ranger is running but at the moment it is connected only to the unix account: has to be connected to AD to get this working? Sorry if this question is one like "how i can be rich" but i am a little lost. I managed to using this guide create a kafkapro4 user and i copied the keytab to the hadoop-X nodes on the right folder. But when i want to run the kinit -kt keytab ServiceAccout, it answer me: kinit -kt kafkapro.service.keytab kafkapro/hadoop-1.poc.domain.local@domain.LOCAL
kinit: Client 'kafkapro4/hadoop-1.poc.domain.local@DOMAIN.LOCAL' not found in Kerberos database while getting initial credentials So i am quite a bit lost how to manage it.
... View more
Labels:
- Labels:
-
Apache Hadoop
02-23-2018
01:07 PM
Hi. Yes, indeed was a problem from the certificate. What i did to fix it: get the certificate from the AD using the java tool import all certificates from jssecacerts to cacerts import all certificates from cacerts to ambari trustore set ambari trustore check it with keytool Then it worked. Thanks!
... View more
02-19-2018
11:27 AM
Hi.
I am triying to setup a kerberized cluster against a AD2012r2 and it fails with error due a simpel bind problem.
Ambari-server.log
19 Feb 2018 11:05:48,109 WARN [ambari-client-thread-42] ADKerberosOperationHandler:470 - Failed to communicate with the Active Directory at ldaps://windc12.domain.tld:636: simple bind failed: windc12.domain.tld:636
javax.naming.CommunicationException: simple bind failed: windc12.domain.tld:636 [Root exception is javax.net.ssl.SSLException: java.lang.RuntimeException:
Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty]
at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:219)
at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2788)
at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:319)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:192)
at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:210)
at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:153)
at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:83)
at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:684)
at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:313)
at javax.naming.InitialContext.init(InitialContext.java:244)
at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:154)
at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createInitialLdapContext(ADKerberosOperationHandler.java:514)
at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.createLdapContext(ADKerberosOperationHandler.java:465)
at org.apache.ambari.server.serveraction.kerberos.ADKerberosOperationHandler.open(ADKerberosOperationHandler.java:182)
at org.apache.ambari.server.controller.KerberosHelperImpl.validateKDCCredentials(KerberosHelperImpl.java:1901)
at org.apache.ambari.server.controller.KerberosHelperImpl.handle(KerberosHelperImpl.java:2027)
at org.apache.ambari.server.controller.KerberosHelperImpl$$EnhancerByGuice$$1fa97c03.CGLIB$handle$0(<generated>)
at org.apache.ambari.server.controller.KerberosHelperImpl$$EnhancerByGuice$$1fa97c03$$FastClassByGuice$$3f1a93b8.invoke(<generated>)
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:118)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at org.apache.ambari.server.controller.KerberosHelperImpl$$EnhancerByGuice$$1fa97c03.handle(<generated>)
at org.apache.ambari.server.controller.KerberosHelperImpl.toggleKerberos(KerberosHelperImpl.java:228)
at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateCluster(AmbariManagementControllerImpl.java:1949)
at org.apache.ambari.server.controller.AmbariManagementControllerImpl.updateClusters(AmbariManagementControllerImpl.java:1521)
at org.apache.ambari.server.controller.AmbariManagementControllerImpl$$EnhancerByGuice$$8e25cb18.CGLIB$updateClusters$47(<generated>)
at org.apache.ambari.server.controller.AmbariManagementControllerImpl$$EnhancerByGuice$$8e25cb18$$FastClassByGuice$$16893f3a.invoke(<generated>)
at com.google.inject.internal.cglib.proxy.$MethodProxy.invokeSuper(MethodProxy.java:228)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at org.apache.ambari.server.orm.AmbariJpaLocalTxnInterceptor.invoke(AmbariJpaLocalTxnInterceptor.java:128)
at com.google.inject.internal.InterceptorStackCallback$InterceptedMethodInvocation.proceed(InterceptorStackCallback.java:72)
at com.google.inject.internal.InterceptorStackCallback.intercept(InterceptorStackCallback.java:52)
at org.apache.ambari.server.controller.AmbariManagementControllerImpl$$EnhancerByGuice$$8e25cb18.updateClusters(<generated>)
at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:313)
at org.apache.ambari.server.controller.internal.ClusterResourceProvider$2.invoke(ClusterResourceProvider.java:310)
at org.apache.ambari.server.controller.internal.AbstractResourceProvider.invokeWithRetry(AbstractResourceProvider.java:455)
at org.apache.ambari.server.controller.internal.AbstractResourceProvider.modifyResources(AbstractResourceProvider.java:336)
at org.apache.ambari.server.controller.internal.ClusterResourceProvider.updateResourcesAuthorized(ClusterResourceProvider.java:310)
at org.apache.ambari.server.controller.internal.AbstractAuthorizedResourceProvider.updateResources(AbstractAuthorizedResourceProvider.java:301)
at org.apache.ambari.server.controller.internal.ClusterControllerImpl.updateResources(ClusterControllerImpl.java:319)
at org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.update(PersistenceManagerImpl.java:125)
at org.apache.ambari.server.api.handlers.UpdateHandler.persist(UpdateHandler.java:45)
at org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:73)
at org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144)
at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:126)
at org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:90)
at org.apache.ambari.server.api.services.ClusterService.updateCluster(ClusterService.java:142)
at sun.reflect.GeneratedMethodAccessor280.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
TCPDUMP betwen windc12 (10.25.8.24) and hadoop-1 (10.43.30.35)
1. Exchange of certificates
No. Time Source Destination Protocol Length Info
27 369.395533 10.43.30.35 10.25.8.24 TLSv1.2 341 Client Hello
Frame 27: 341 bytes on wire (2728 bits), 341 bytes captured (2728 bits) on interface 0
Ethernet II, Src: Fortinet_09:00:06 (00:09:0f:09:00:06), Dst: Vmware_99:e7:af (00:50:56:99:e7:af)
Internet Protocol Version 4, Src: 10.43.30.35, Dst: 10.25.8.24
Transmission Control Protocol, Src Port: 33482, Dst Port: 636, Seq: 1, Ack: 1, Len: 275
Source Port: 33482
Destination Port: 636
[Stream index: 2]
[TCP Segment Len: 275]
Sequence number: 1 (relative sequence number)
[Next sequence number: 276 (relative sequence number)]
Acknowledgment number: 1 (relative ack number)
1000 .... = Header Length: 32 bytes (8)
Flags: 0x018 (PSH, ACK)
Window size value: 229
[Calculated window size: 29312]
[Window size scaling factor: 128]
Checksum: 0x8e5c [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[SEQ/ACK analysis]
TCP payload (275 bytes)
Secure Sockets Layer
TLSv1.2 Record Layer: Handshake Protocol: Client Hello
Content Type: Handshake (22)
Version: TLS 1.2 (0x0303)
Length: 270
Handshake Protocol: Client Hello
Handshake Type: Client Hello (1)
Length: 266
Version: TLS 1.2 (0x0303)
Random: 5a87095c50d1d156ccb93b473da7f0580f40edfc3a53d42f...
GMT Unix Time: Feb 16, 2018 17:39:56.000000000 Romance Standard Time
Random Bytes: 50d1d156ccb93b473da7f0580f40edfc3a53d42f5ac230d7...
Session ID Length: 0
Cipher Suites Length: 100
Cipher Suites (50 suites)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 (0xc024)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA256 (0x003d)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 (0xc026)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 (0xc02a)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 (0x006b)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA256 (0x006a)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (0xc00a)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (0xc014)
Cipher Suite: TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA (0xc005)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_CBC_SHA (0xc00f)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_CBC_SHA (0x0038)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (0xc023)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (0xc027)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA256 (0x003c)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 (0xc025)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 (0xc029)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (0x0067)
Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA256 (0x0040)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (0xc009)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (0xc013)
Cipher Suite: TLS_RSA_WITH_AES_128_CBC_SHA (0x002f)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA (0xc004)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_CBC_SHA (0xc00e)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_CBC_SHA (0x0033)
Cipher Suite: TLS_DHE_DSS_WITH_AES_128_CBC_SHA (0x0032)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02c)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02b)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (0xc030)
Cipher Suite: TLS_RSA_WITH_AES_256_GCM_SHA384 (0x009d)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 (0xc02e)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 (0xc032)
Cipher Suite: TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (0x009f)
Cipher Suite: TLS_DHE_DSS_WITH_AES_256_GCM_SHA384 (0x00a3)
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (0xc02f)
Cipher Suite: TLS_RSA_WITH_AES_128_GCM_SHA256 (0x009c)
Cipher Suite: TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 (0xc02d)
Cipher Suite: TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 (0xc031)
Cipher Suite: TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (0x009e)
Cipher Suite: TLS_DHE_DSS_WITH_AES_128_GCM_SHA256 (0x00a2)
Cipher Suite: TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc008)
Cipher Suite: TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA (0xc012)
Cipher Suite: TLS_RSA_WITH_3DES_EDE_CBC_SHA (0x000a)
Cipher Suite: TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA (0xc003)
Cipher Suite: TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA (0xc00d)
Cipher Suite: TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA (0x0016)
Cipher Suite: TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA (0x0013)
Cipher Suite: TLS_EMPTY_RENEGOTIATION_INFO_SCSV (0x00ff)
Compression Methods Length: 1
Compression Methods (1 method)
Extensions Length: 125
Extension: supported_groups (len=52)
Type: supported_groups (10)
Length: 52
Supported Groups List Length: 50
Supported Groups (25 groups)
Extension: ec_point_formats (len=2)
Type: ec_point_formats (11)
Length: 2
EC point formats Length: 1
Elliptic curves point formats (1)
Extension: signature_algorithms (len=28)
Type: signature_algorithms (13)
Length: 28
Signature Hash Algorithms Length: 26
Signature Hash Algorithms (13 algorithms)
Extension: server_name (len=27)
Type: server_name (0)
Length: 27
Server Name Indication extension
Server Name list length: 25
Server Name Type: host_name (0)
Server Name length: 22
Server Name: windc12.knockout.local
No. Time Source Destination Protocol Length Info
28 369.399674 10.25.8.24 10.43.30.35 TLSv1.2 1993 Server Hello, Certificate, Server Key Exchange, Certificate Request, Server Hello Done
Frame 28: 1993 bytes on wire (15944 bits), 1993 bytes captured (15944 bits) on interface 0
Ethernet II, Src: Vmware_99:e7:af (00:50:56:99:e7:af), Dst: Fortinet_09:00:06 (00:09:0f:09:00:06)
Internet Protocol Version 4, Src: 10.25.8.24, Dst: 10.43.30.35
Transmission Control Protocol, Src Port: 636, Dst Port: 33482, Seq: 1, Ack: 276, Len: 1927
Source Port: 636
Destination Port: 33482
[Stream index: 2]
[TCP Segment Len: 1927]
Sequence number: 1 (relative sequence number)
[Next sequence number: 1928 (relative sequence number)]
Acknowledgment number: 276 (relative ack number)
1000 .... = Header Length: 32 bytes (8)
Flags: 0x018 (PSH, ACK)
Window size value: 513
[Calculated window size: 131328]
[Window size scaling factor: 256]
Checksum: 0x3a85 [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[SEQ/ACK analysis]
TCP payload (1927 bytes)
Secure Sockets Layer
TLSv1.2 Record Layer: Handshake Protocol: Multiple Handshake Messages
Content Type: Handshake (22)
Version: TLS 1.2 (0x0303)
Length: 1922
Handshake Protocol: Server Hello
Handshake Type: Server Hello (2)
Length: 77
Version: TLS 1.2 (0x0303)
Random: 5a87095c08ab7ab8058b2d4454dfdd8fb292a021bb3c4cf7...
GMT Unix Time: Feb 16, 2018 17:39:56.000000000 Romance Standard Time
Random Bytes: 08ab7ab8058b2d4454dfdd8fb292a021bb3c4cf7aa0db8e2...
Session ID Length: 32
Session ID: 0b4200002bacc095c1fa5c4c54eb43a8ba2e34064517b1ea...
Cipher Suite: TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (0xc028)
Compression Method: null (0)
Extensions Length: 5
Extension: renegotiation_info (len=1)
Type: renegotiation_info (65281)
Length: 1
Renegotiation Info extension
Handshake Protocol: Certificate
Handshake Type: Certificate (11)
Length: 1470
Certificates Length: 1467
Certificates (1467 bytes)
Handshake Protocol: Server Key Exchange
Handshake Type: Server Key Exchange (12)
Length: 329
EC Diffie-Hellman Server Params
Handshake Protocol: Certificate Request
Handshake Type: Certificate Request (13)
Length: 26
Certificate types count: 3
Certificate types (3 types)
Signature Hash Algorithms Length: 18
Signature Hash Algorithms (9 algorithms)
Distinguished Names Length: 0
Handshake Protocol: Server Hello Done
Handshake Type: Server Hello Done (14)
Length: 0
2. Then, weird fatal error
No. Time Source Destination Protocol Length Info
31 369.530739 10.43.30.35 10.25.8.24 TLSv1.2 73 Alert (Level: Fatal, Description: Internal Error)
Frame 31: 73 bytes on wire (584 bits), 73 bytes captured (584 bits) on interface 0
Ethernet II, Src: Fortinet_09:00:06 (00:09:0f:09:00:06), Dst: Vmware_99:e7:af (00:50:56:99:e7:af)
Internet Protocol Version 4, Src: 10.43.30.35, Dst: 10.25.8.24
Transmission Control Protocol, Src Port: 33482, Dst Port: 636, Seq: 276, Ack: 1928, Len: 7
Source Port: 33482
Destination Port: 636
[Stream index: 2]
[TCP Segment Len: 7]
Sequence number: 276 (relative sequence number)
[Next sequence number: 283 (relative sequence number)]
Acknowledgment number: 1928 (relative ack number)
1000 .... = Header Length: 32 bytes (8)
Flags: 0x018 (PSH, ACK)
Window size value: 273
[Calculated window size: 34944]
[Window size scaling factor: 128]
Checksum: 0xe52a [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[SEQ/ACK analysis]
TCP payload (7 bytes)
Secure Sockets Layer
TLSv1.2 Record Layer: Alert (Level: Fatal, Description: Internal Error)
Content Type: Alert (21)
Version: TLS 1.2 (0x0303)
Length: 2
Alert Message
Level: Fatal (2)
Description: Internal Error (80)
3. This seems to be the culprit a "couple of tcp packets later" > noRealm send
No. Time Source Destination Protocol Length Info
38 413.210817 10.43.30.35 10.25.8.24 KRB5 225 AS-REQ
Frame 38: 225 bytes on wire (1800 bits), 225 bytes captured (1800 bits) on interface 0
Ethernet II, Src: Fortinet_09:00:06 (00:09:0f:09:00:06), Dst: Vmware_99:e7:af (00:50:56:99:e7:af)
Internet Protocol Version 4, Src: 10.43.30.35, Dst: 10.25.8.24
Transmission Control Protocol, Src Port: 37748, Dst Port: 88, Seq: 1, Ack: 1, Len: 159
Source Port: 37748
Destination Port: 88
[Stream index: 3]
[TCP Segment Len: 159]
Sequence number: 1 (relative sequence number)
[Next sequence number: 160 (relative sequence number)]
Acknowledgment number: 1 (relative ack number)
1000 .... = Header Length: 32 bytes (8)
Flags: 0x018 (PSH, ACK)
Window size value: 229
[Calculated window size: 29312]
[Window size scaling factor: 128]
Checksum: 0xcd8e [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[SEQ/ACK analysis]
TCP payload (159 bytes)
[PDU Size: 159]
Kerberos
Record Mark: 155 bytes
0... .... .... .... .... .... .... .... = Reserved: Not set
.000 0000 0000 0000 0000 0000 1001 1011 = Record Length: 155
as-req
pvno: 5
msg-type: krb-as-req (10)
req-body
Padding: 0
kdc-options: 00000020 (disable-transited-check)
cname
name-type: kRB5-NT-PRINCIPAL (1)
cname-string: 1 item
CNameString: noUser
realm: noRealm
sname
name-type: kRB5-NT-SRV-INST (2)
sname-string: 2 items
SNameString: krbtgt
SNameString: noRealm
from: 2018-02-16 16:40:40 (UTC)
till: 2018-02-17 00:40:40 (UTC)
nonce: 3125110255
etype: 4 items
ENCTYPE: eTYPE-AES256-CTS-HMAC-SHA1-96 (18)
ENCTYPE: eTYPE-AES128-CTS-HMAC-SHA1-96 (17)
ENCTYPE: eTYPE-DES3-CBC-SHA1 (16)
ENCTYPE: eTYPE-DES-CBC-MD5 (3)
No. Time Source Destination Protocol Length Info
39 413.210966 10.25.8.24 10.43.30.35 KRB5 156 KRB Error: KDC_ERR_WRONG_REALM
Frame 39: 156 bytes on wire (1248 bits), 156 bytes captured (1248 bits) on interface 0
Ethernet II, Src: Vmware_99:e7:af (00:50:56:99:e7:af), Dst: Fortinet_09:00:06 (00:09:0f:09:00:06)
Internet Protocol Version 4, Src: 10.25.8.24, Dst: 10.43.30.35
Transmission Control Protocol, Src Port: 88, Dst Port: 37748, Seq: 1, Ack: 160, Len: 90
Source Port: 88
Destination Port: 37748
[Stream index: 3]
[TCP Segment Len: 90]
Sequence number: 1 (relative sequence number)
[Next sequence number: 91 (relative sequence number)]
Acknowledgment number: 160 (relative ack number)
1000 .... = Header Length: 32 bytes (8)
Flags: 0x018 (PSH, ACK)
Window size value: 513
[Calculated window size: 131328]
[Window size scaling factor: 256]
Checksum: 0x3aff [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
[SEQ/ACK analysis]
TCP payload (90 bytes)
[PDU Size: 90]
Kerberos
Record Mark: 86 bytes
0... .... .... .... .... .... .... .... = Reserved: Not set
.000 0000 0000 0000 0000 0000 0101 0110 = Record Length: 86
krb-error
pvno: 5
msg-type: krb-error (30)
stime: 2018-02-16 16:40:40 (UTC)
susec: 71498
error-code: eRR-WRONG-REALM (68)
realm: noRealm
sname
name-type: kRB5-NT-SRV-INST (2)
sname-string: 2 items
SNameString: krbtgt
SNameString: noRealm
4. even when the krb5.conf file on hadoop-1 is properly configured:
[libdefaults]
renew_lifetime = 7d
forwardable = true
default_realm = DOMAIN
ticket_lifetime = 24h
dns_lookup_realm = true
dns_lookup_kdc = true
default_ccache_name = /tmp/krb5cc_%{uid}
#default_tgs_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
#default_tkt_enctypes = aes des3-cbc-sha1 rc4 des-cbc-md5
[domain_realm]
DOMAIN = DOMAIN.LOCAL
[logging]
default = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
kdc = FILE:/var/log/krb5kdc.log
[realms]
DOMAIN.TLD = {
master_kdc = windc12.domain.tld
admin_server = windc12.domain.tld
kdc = windc12.domain.tld
}
5. ....and DNS entries are right from the same hadoop-1 machine.
; <<>> DiG 9.9.4-RedHat-9.9.4-51.el7_4.2 <<>> SRV _ldap._tcp.domain.tld
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49372
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 6
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;_ldap._tcp.domain.tld. IN SRV
;; ANSWER SECTION:
_ldap._tcp.domain.tld. 600 IN SRV 0 100 389 windc13.domain.tld.
_ldap._tcp.domain.tld. 600 IN SRV 0 100 389 windc16.domain.tld.
_ldap._tcp.domain.tld. 600 IN SRV 0 100 389 windc20.domain.tld.
_ldap._tcp.domain.tld. 600 IN SRV 0 100 389 windc11.domain.tld.
_ldap._tcp.domain.tld. 600 IN SRV 0 100 389 windc12.domain.tld.
;; ADDITIONAL SECTION:
windc13.domain.tld. 3600 IN A 10.25.153.3
windc16.domain.tld. 3600 IN A 10.41.40.4
windc20.domain.tld. 3600 IN A 10.35.8.20
windc11.domain.tld. 3600 IN A 10.25.8.23
windc12.domain.tld. 3600 IN A 10.25.8.24
;; Query time: 136 msec
;; SERVER: 10.25.8.23#53(10.25.8.23)
;; WHEN: lun feb 19 12:23:36 CET 2018
;; MSG SIZE rcvd: 344
<<< PRO4 >>> root@hadoop-1:/var/log/ambari-server# dig SRV _kerberos._tcp.domain.tld
; <<>> DiG 9.9.4-RedHat-9.9.4-51.el7_4.2 <<>> SRV _kerberos._tcp.domain.tld
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45966
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 6
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;_kerberos._tcp.domain.tld. IN SRV
;; ANSWER SECTION:
_kerberos._tcp.domain.tld. 600 IN SRV 0 100 88 windc11.domain.tld.
_kerberos._tcp.domain.tld. 600 IN SRV 0 100 88 windc16.domain.tld.
_kerberos._tcp.domain.tld. 600 IN SRV 0 100 88 windc12.domain.tld.
_kerberos._tcp.domain.tld. 600 IN SRV 0 100 88 windc20.domain.tld.
_kerberos._tcp.domain.tld. 600 IN SRV 0 100 88 windc13.domain.tld.
;; ADDITIONAL SECTION:
windc11.domain.tld. 3600 IN A 10.25.8.23
windc16.domain.tld. 3600 IN A 10.41.40.4
windc12.domain.tld. 3600 IN A 10.25.8.24
windc20.domain.tld. 3600 IN A 10.35.8.20
windc13.domain.tld. 3600 IN A 10.25.153.3
;; Query time: 130 msec
;; SERVER: 10.25.8.23#53(10.25.8.23)
;; WHEN: lun feb 19 12:23:43 CET 2018
############################################################################################################################################################################################################
What i did to setup the enviroment
windc was configured with LDAPS and enabled certification authority using this guide
openssl s_client -connect windc.domain.tld:636 -showcerts retrieve certificate
openssl s_client -connect windc12.DOMAIN.TLD:636 -showcertsCONNECTED(00000003)
depth=1 DC = TLD, DC = domain, CN = WINDC12-CA-1
verify return:1
depth=0
verify return:1
---
Certificate chain
0 s:
i:/DC=tld/DC=domain/CN=WINDC12-CA-1
-----BEGIN CERTIFICATE-----
<SANITIZED>
-----END CERTIFICATE-----
---
Server certificate
subject=
issuer=/DC=tld/DC=domain/CN=WINDC12-CA-1
---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA512:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA512:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA1:ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA1:DSA+SHA1
Peer signing digest: SHA1
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2034 bytes and written 495 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA384
Session-ID: 0014000092CC522ED518B2129168F67BF<SANITIZED>00B442B09077697C45B44
Session-ID-ctx:
Master-Key: DCF5ED4CA2D89ADD1E84A1B6A89F82C38755669<SANITIZED>8FA32EC53A18F3D434EEDF45BC4977A34B704
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1519037714
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
kinit from the hadoo-ambari with the same credentials username@DOMAIN.TLD is successfull and i got a ticket.
Using default cache: /tmp/krb5cc_0
Using principal: ko-hadoop@DOMAIN.TLD
Password for ko-hadoop@DOMAIN.TLD:
Authenticated to Kerberos v5
klist from the hadoop-ambari server shows the ticket against the DC/KDC
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: ko-hadoop@DOMAIN.TLD
Valid starting Expires Service principal
19/02/18 10:20:33 19/02/18 20:20:33 krbtgt/DOMAIN.TLD@DOMAIN.TLD
renew until 26/02/18 10:20:27
############################################################################################################################################################################################################
Any ideas???
Thanks!
... View more
Labels:
- Labels:
-
Apache Ambari
