@Junfeng Chen, as mentioned, it depends on your use of it. It will run okay in most deployment patterns, and it can run fine in VMs, but of course having dedicated hardware is always preffered. ... View more

@Gaurang Shah If you see broker metrics, then that is where you exposed the port to. Kafka Connect is meant to be run on separate machines from the brokers, but if you are running on the same ones, then you must expose and monitor two differerent JMX ports. ... View more

"Can they" - Yes. "Should they" - I would say no. Kafka is very memory and disk sensitive. Depending on your use of it, it could even use more I/O than the combination of the DataNode and NodeManager on the same machine. Personally, I would recommend installing Kafka brokers on dedicated hardware, even separate from the Zookeeper servers it needs, if at all possible. The Spark executors do not need to be running on the Kafka brokers, they should work fine pulling remotely from the YARN NodeManagers. ... View more

JMX is not exposed in a property file. It is toggled from two environment variable. Confluent and Apache Kafka installations share the same variables for this. Source code here - https://github.com/apache/kafka/blob/trunk/bin/kafka-run-class.sh#L166-L174 Basically, if you export the JMX_PORT to a valid port number, it will open that port for JMX monitoring of any Kafka-related script that you run. The recommendation, however, is to use a cluster of machines running Connect Distributed, as it is a long-running process, and you can scale your metrics collection using tools like Prometheus JMX Exporter combined with a Grafana server for dashboarding. ... View more

btw, these questions are copied from https://data-flair.training/forums/topic/how-client-can-interact-with-hive ... View more

Can you please verify / show the Kafka Server properties? When you run Kafka within a container, you need to make sure that clients are getting the external addrress from Zookeeper. A simple port forward is not enough. https://rmoff.net/2018/08/02/kafka-listeners-explained/ ... View more

It is a Java class for reading Hadoop SequenceFIles http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/lib/input/SequenceFileInputFormat.html ... View more

Hello @nicole wells Please find the question that you copied (and my answer) here https://stackoverflow.com/a/51687883/2308683 ... View more

You only need to use a Schema Registry if you plain on using Confluent's AvroConverter Note: NiFI can also be used to do CDC from MySQL https://community.hortonworks.com/articles/113941/change-data-capture-cdc-with-apache-nifi-version-1-1.html ... View more

On brokers termination, they remove themselves from Zookeeper ... View more