Member since
11-19-2015
158
Posts
25
Kudos Received
21
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
11921 | 09-01-2018 01:27 AM | |
1144 | 09-01-2018 01:18 AM | |
3808 | 08-20-2018 09:39 PM | |
515 | 07-20-2018 04:51 PM | |
1526 | 07-16-2018 09:41 PM |
03-18-2019
07:08 PM
@Junfeng Chen, as mentioned, it depends on your use of it. It will run okay in most deployment patterns, and it can run fine in VMs, but of course having dedicated hardware is always preffered.
... View more
03-18-2019
07:05 PM
@Gaurang Shah If you see broker metrics, then that is where you exposed the port to. Kafka Connect is meant to be run on separate machines from the brokers, but if you are running on the same ones, then you must expose and monitor two differerent JMX ports.
... View more
03-11-2019
07:35 PM
"Can they" - Yes. "Should they" - I would say no. Kafka is very memory and disk sensitive. Depending on your use of it, it could even use more I/O than the combination of the DataNode and NodeManager on the same machine. Personally, I would recommend installing Kafka brokers on dedicated hardware, even separate from the Zookeeper servers it needs, if at all possible. The Spark executors do not need to be running on the Kafka brokers, they should work fine pulling remotely from the YARN NodeManagers.
... View more
03-11-2019
07:31 PM
JMX is not exposed in a property file. It is toggled from two environment variable. Confluent and Apache Kafka installations share the same variables for this. Source code here - https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L166-L174 Basically, if you export the JMX_PORT to a valid port number, it will open that port for JMX monitoring of any Kafka-related script that you run. The recommendation, however, is to use a cluster of machines running Connect Distributed, as it is a long-running process, and you can scale your metrics collection using tools like Prometheus JMX Exporter combined with a Grafana server for dashboarding.
... View more
10-09-2018
07:35 PM
btw, these questions are copied from https://data-flair.training/forums/topic/how-client-can-interact-with-hive
... View more
10-09-2018
07:30 PM
Can you please verify / show the Kafka Server properties? When you run Kafka within a container, you need to make sure that clients are getting the external addrress from Zookeeper. A simple port forward is not enough. https://rmoff.net/2018/08/02/kafka-listeners-explained/
... View more
10-06-2018
06:41 PM
It is a Java class for reading Hadoop SequenceFIles http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/lib/input/SequenceFileInputFormat.html
... View more
10-03-2018
10:36 PM
Hello @nicole wells Please find the question that you copied (and my answer) here https://stackoverflow.com/a/51687883/2308683
... View more
09-30-2018
01:31 AM
You only need to use a Schema Registry if you plain on using Confluent's AvroConverter Note: NiFI can also be used to do CDC from MySQL https://community.hortonworks.com/articles/113941/change-data-capture-cdc-with-apache-nifi-version-1-1.html
... View more
09-30-2018
01:27 AM
On brokers termination, they remove themselves from Zookeeper
... View more