@dbains Hi, here is more info. kafka server.properties file content: listeners=SASL_PLAINTEXT://host:port security.inter.broker.protocol=SASL_PLAINTEXT sasl.mechanism.inter.broker.protocol=PLAIN sasl.enabled.mechanism=PLAIN flume.log file: .......... Date, 310 ERROR [lifecycleSupervisor-1-2] (org.apache.flume.lifecycle.Lifecycle.Supervisor$MonitorRunnable.run:251) - Unable to start PollableSourceRunner : {source:org.apache.flume.source.kafka.KafkaSource{name:source1, state:IDLE} counterGroup:{name:null counters:{}}} - Exception follows. java.lag.NoSuchMethodError: org.apache.flume.source.kafka.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/clients/consumer/ConsumerRebalanceListener;) .......... Regardless the error telling about "java.lag.NoSuchMethodError: org.apache.flume.source.kafka.KafkaConsumer.subscribe(...)" I looked at the flume's lib folder kafka-client jar file and made sure the method subscribe(...) is in there. So, the flume's agent still unable to function properly with kafka broker secured with SASL/PLAIN ... View more

@dbains Hi, I appreciate your quick response dbains. Let me make more clear with my use case: (1)Apache kafka_2.11-0.10.2.0 (which is not a component of HDP-2.6.5) is running on a Linux server. Flume apache-flume-1.8.0 is running on another Linux server. So, Kafka broker and Fflume agent are running on separate Linux servers/machines/boxes. Why not, if needed ports of Kafka/Zookeeper and Flume are opened for communications between servers? (2)Kafka security protocol is SASL_PLAINTEXT and sasl mechanism is PLAIN. There is NO Kerberos security setup/enabled on Kafka broker. Taking into account (1) and (2), why you are confused with "Flume and Kafka are running on different RHEL servers and Kafka secured with the security.protocol=SASL_PLAINTEXT" ? (3)Flume 1.8 User Guide https://flume.apache.org/FlumeUserGuide.html states in my case to use a1.sources.s1.kafka.consumer.security.protocol=SASL_PLAINTEXT and a1.sources.s1.kafka.consumer.sasl.mechanism=PLAIN Why should I use a1.sources.s1.kafka.consumer.security.protocol=PLAINTEXT ? (4)Now with the above mentioned Kafka and Flume setup I'm experiencing the following issues: (4-a)When I start Flume agent then I'm getting the ERROR ..... could Not find PlainLoginModule. Browsing different Flume discussions I concluded that kafka-clients-0.9.0.1.jar does Not contain the PlainLoginModule. (4-b)When I replaced original kafka-clients-0.9.0.1.jar with the kafka-clients-0.10.2.0.jar then Im getting the ERROR ... Unable to start PollableSourceRunner:...{name:s1, state:IDLE} counterGroup ... and this is my current issue I'm unable to resolve. Once again, I appreciate your help dbain and will try to run Flume agent with the a1.sources.s1.kafka.consumer.security.protocol=PLAINTEXT as you advised.i Also, I'll double check and let you know about your question "listeners property in your Kafka broker and let me know what security protocol is it listening to". I would appreciate any help to resolve this issue or point me out to any helpful information how to properly run Apache kafka_2.11-0.10.2.0 broker with Flume 1.8 and with Kafkasecurity protocol SASL_PLAINTEXT and sasl mechanism PLAIN (NO KERBEROS enabled). Thanks in advance. ... View more