Member since
04-16-2018
14
Posts
0
Kudos Received
0
Solutions
02-03-2019
01:17 AM
Figured it out... You have to specify a null SSL Endpoint Identification Algorithm when you call the console producer/consumer. /usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-consumer.sh --bootstrap-server myvm.mydomain.com:9093 --topic test_topic --consumer.config /usr/hdp/3.1.0.0-78/kafka/config/client-ssl.properties --consumer-property ssl.endpoint.identification.algorithm=
/usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list myvm.mydomain:9093 --topic test_topic --producer.config /usr/hdp/3.1.0.0-78/kafka/config/client-ssl.properties --producer-property ssl.endpoint.identification.algorithm=
... View more
02-03-2019
01:17 AM
I have an HDP cluster that I recently upgraded from 2.6.5 to 3.1.0. For Kafka, I only have the SSL listeners enabled but I've had issue with getting the certs right so in my calling apps (producer and consumer) I'm bypassing the SSL Endpoint Identification. With the change to Kafka 2.0.0 my calling apps seem to be fine, however when I try to spin up a console-consumer/producer I get the following error: ERROR Authentication failed: terminating consumer process (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1214)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1186)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at org.apache.kafka.common.network.SslTransportLayer.handshakeWrap(SslTransportLayer.java:439)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:304)
at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:258)
at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:125)
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:487)
at org.apache.kafka.common.network.Selector.poll(Selector.java:425)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:271)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:218)
at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:230)
at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:314)
at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1181)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1115)
at kafka.tools.ConsoleConsumer$ConsumerWrapper.<init>(ConsoleConsumer.scala:387)
at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:71)
at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:53)
at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:393)
at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:473)
at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:331)
... 17 more
Caused by: java.security.cert.CertificateException: No name matching {DNS NAME OF VM REMOVED} found
at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:221)
at sun.security.util.HostnameChecker.match(HostnameChecker.java:95)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)
... 26 more
In my server.properties both ssl.endpoint.identification.algorithm and listener.name.internal.ssl.endpoint.identification.algorithm are set to empty strings (I've tried null too). Here is how I'm calling the producer and consumer: /usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-consumer.sh --bootstrap-server myvm.mydomain.com:9093 --topic test_topic --consumer.config /usr/hdp/3.1.0.0-78/kafka/config/client-ssl.properties
/usr/hdp/3.1.0.0-78/kafka/bin/kafka-console-producer.sh --broker-list myvm.mydomain:9093 --topic test_topic --producer.config /usr/hdp/3.1.0.0-78/kafka/config/client-ssl.properties
Any tips on how I can get these working?
... View more
Labels:
11-06-2018
10:14 PM
I realize this is an old answer but it is wrong and I don't want people going down the wrong path. The auto.offset.reset config is on the consumer side, not the topic or broker side. You can add it as a custom property all day long in the Kafka configuration page but that won't help anything. This setting is driven from whatever code is consuming your topic. Also, some of the language has changed: "latest, earliest, or none" work for the new consumer API and "smallest, largest, or {any other value}" work for the old consumer API.
... View more
10-25-2018
01:54 PM
Thanks Jeff, this worked to help me upgrade from HDP 2.6.4.0 -> 2.6.5.0
... View more
09-15-2018
10:17 PM
@Jay Kumar SenSharma thank you for the answer, this fixed my issue. Adding force_https_protocol=PROTOCOL_TLSv1 to the Ambari Agents allowed them to connect to the server.
... View more
09-07-2018
01:06 AM
We are running RHEL 7, HDP 2.6.4, and Ambari 2.6.2.0 on a 6 VM cluster. This morning we patched the cluster and one of the items it upgraded was Python. It took Python from 2.7.5-68 to 2.7.5-69, effectively breaking communication between the Ambari Server and Agents. The following error was thrown every 10 seconds in the ambari-agent.log INFO 2018-09-06 13:09:36,133 NetUtil.py:70 - Connecting to https://{HOST_NAME}:8440/ca ERROR 2018-09-06 13:09:36,166 NetUtil.py:96 - EOF occurred in violation of protocol (_ssl.c:579) ERROR 2018-09-06 13:09:36,167 NetUtil.py:97 - SSLError: Failed to connect. Please check openssl library versions. Refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1022468 for more details. My question is, what version of HDP/Ambari do I need to be on in order to use the latest version of Python? Thanks, Andrew
... View more
Labels:
05-03-2018
03:32 PM
I was able to delete the offsets in Zookeeper and get this up and running, still wondering why it was happening though...
... View more
05-02-2018
07:54 PM
Sandeep, thanks for the response. As suggested, I have the following configurations established for the executors: spark.executor.logs.rolling.strategy time spark.executor.logs.rolling.maxRetainedFiles 72 spark.executor.logs.rolling.time.interval {various settings} I've tested both the hourly and minutely settings for the above time interval and both of those seem to work, as in they roll the executor logs as they should. I've also set yarn.nodemanager.log-aggregation.debug-enabled=true & yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds={various settings} and restarted YARN. Sadly, I'm not seeing the aggregate logs respect any of the settings, they just continue to grow and grow. Any other tips?
... View more
05-02-2018
07:12 PM
I am running HDP with YARN, Kafka 2.10-0.10.1.2.6.3, Spark2, and Zookeeper. I'm submitting these jobs to YARN and the Spark2 jobs store the offsets in Zookeeper and manually fetches and saves them. The jobs had been off for more than a week then I adjusted the retention of my Kafka data to 4 days, to reduce data size, and now I'm seeing this error: org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions: {global_transactions-2=133030} I have 6 partitions and each one fetches an offset from Zookeeper that used to be valid. Since I changed the retention policy to 4 days it deleted all of the old data, including where this offset starts, so now the jobs wont start. In my Spark2 job I set auto.offset.reset=earliest. Here is the string map I'm passing as the Kafka parameters as well as the code I'm using to kick off my direct stream. object Kafka {
def getParams(topicSet: Set[String], groupId: String, conf: AppConfig): Map[String, String] = {<br> return Map[String, String](<br> {"bootstrap.servers" -> conf.kafkaHosts},<br> {"group.id" -> groupId},<br> {"auto.offset.reset" -> "earliest"},<br> {"key.deserializer" -> "org.apache.kafka.common.serialization.StringDeserializer"},<br> {"value.deserializer" -> "org.apache.kafka.common.serialization.StringDeserializer"},<br> {"enable.auto.commit" -> "false"}<br> )<br> }<br>}
val kafkaData = KafkaUtils.createDirectStream[String, String](<br> streamingContext,<br> LocationStrategies.PreferConsistent,<br> ConsumerStrategies.Subscribe[String, String](topicSet, Kafka.getParams(topicSet = topicSet, groupId = appName, conf = config), fromOffsets)<br>) Can anyone provide any information as to why Kafka does not see the auto.offset.reset setting? If I change it to something invalid, like test I get an error back telling me that it's not a valid configuration so I know it's seeing it in that context.
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Spark
-
Apache YARN
04-20-2018
03:48 PM
Thanks Sandeep, I'm working on this now and will report back once I've got it all setup.
... View more
04-20-2018
01:37 PM
Thanks for the response Tarun. I've tried that setting with no luck. I currently have it set to 600 as a test, only want to see logs for the last 10 minutes, and I have logs in there from yesterday. Is there a minimum I might be missing? I know the setting has a disclaimer that says not to set it to low it it will spam the node but it does not indicate a minimum threshold. Does this work differently since it's a long-running (streaming) application and I'm technically using the same log file the entire time? The language in the description of this setting implies it deletes the file, in reality it needs to remove lines from a file that is being written to.
... View more
04-20-2018
01:21 PM
I'm running several Spark2 Streaming applications in a YARN cluster. I have yarn.log-aggregation-enable=true and the log files stored in HDFS grow unbounded. Before I was running this on YARN, using Spark Standalone, I used the spark.executor.logs.rolling.{interval, strategy, maxRetainedFiles} to mange log files and it worked great. I've tried all sorts of settings to keep the aggregate logs to a manageable size with no luck. Can someone direct me to the configuration setting(s) that can help define how these aggregate logs are purged? An ideal scenario would allow me to manage them by time and size. Thanks in advance.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Spark
-
Apache YARN
04-16-2018
01:51 PM
I know this is and old thread but can you also include instructions on how to change the log aggregation for Spark2?
... View more