Support Questions

Find answers, ask questions, and share your expertise
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

Does Flume-1.8 support SASL_PLAINTEXT when connecting to Kafka source ?


Environment: apache-flume-1.8.0, kafka 2_2.11-

Flume and Kafka are running on different RHEL servers and Kafka secured with the security.protocol=SASL_PLAINTEXT and sasl.mechanism=PLAIN contains:


flume_plain_jaas.conf contains:

KafkaClient { required



} contains:


Flume agent is not starting and produces the error:

ERROR ...Unexpected error performing start ... *common.KafkaException: Failed to construct kafka consumer

... Caused by org.apache.kafka.common.Kafka Exception: java.lang.IllegalArgumentException: No serviceName defined in either JAAS or Kafka config ...

It seems to me the serviceName is relevant to SASL_PLAINTEXT with Kerberos on Kafka, but this kafka broker/instance does Not use Kerberos.

Any help to resolve this issue will be appreciated.



Rising Star
@Vadim DzyubanHi,

Yes, the ServiceName is required in the Jaas when you are connecting to the secure Kafka (Kerberos enabled), for reference:

If you do not have Kerberos on Kafka then you need not to use :

Instead you can just use:

But I am little confused when you mentioned "Flume and Kafka are running on different RHEL servers and Kafka secured with the security.protocol=SASL_PLAINTEXT"

Can you verify the listeners property in your Kafka broker and let me know what security protocol is it listening to?

Thank you!



Hi, I appreciate your quick response dbains.

Let me make more clear with my use case:

(1)Apache kafka_2.11- (which is not a component of HDP-2.6.5) is running on a Linux server.

Flume apache-flume-1.8.0 is running on another Linux server.

So, Kafka broker and Fflume agent are running on separate Linux servers/machines/boxes. Why not, if needed ports of Kafka/Zookeeper and Flume are opened for communications between servers?

(2)Kafka security protocol is SASL_PLAINTEXT and sasl mechanism is PLAIN.

There is NO Kerberos security setup/enabled on Kafka broker.

Taking into account (1) and (2), why you are confused with "Flume and Kafka are running on different RHEL servers and Kafka secured with the security.protocol=SASL_PLAINTEXT" ?

(3)Flume 1.8 User Guide states in my case to use and a1.sources.s1.kafka.consumer.sasl.mechanism=PLAIN

Why should I use ?

(4)Now with the above mentioned Kafka and Flume setup I'm experiencing the following issues:

(4-a)When I start Flume agent then I'm getting the ERROR ..... could Not find PlainLoginModule.

Browsing different Flume discussions I concluded that kafka-clients- does Not contain the PlainLoginModule.

(4-b)When I replaced original kafka-clients- with the kafka-clients- then Im getting the ERROR ... Unable to start PollableSourceRunner:...{name:s1, state:IDLE} counterGroup ... and this is my current issue I'm unable to resolve.

Once again, I appreciate your help dbain and will try to run Flume agent with the as you advised.i

Also, I'll double check and let you know about your question "listeners property in your Kafka broker and let me know what security protocol is it listening to".

I would appreciate any help to resolve this issue or point me out to any helpful information how to properly run Apache kafka_2.11- broker with Flume 1.8 and with Kafkasecurity protocol SASL_PLAINTEXT and sasl mechanism PLAIN (NO KERBEROS enabled).

Thanks in advance.

Rising Star

Hi @Vadim Dzyuban

Thank you for detailed explanation. It was my mistake that I missed the part where you mentioned sasl mechanism as PLAIN and not GSSAPI. In that case you do not need to pass the ""

Regarding this error: "Unable to start PollableSourceRunner:...{name:s1, state:IDLE} counterGroup" Could you please provide complete error stack for more details on this error?

Thank you!



Hi, here is more info.

kafka file content:



flume.log file:


Date, 310 ERROR [lifecycleSupervisor-1-2] (org.apache.flume.lifecycle.Lifecycle.Supervisor$ - Unable to start PollableSourceRunner : {source:org.apache.flume.source.kafka.KafkaSource{name:source1, state:IDLE} counterGroup:{name:null counters:{}}} - Exception follows. java.lag.NoSuchMethodError: org.apache.flume.source.kafka.KafkaConsumer.subscribe(Ljava/util/List;Lorg/apache/clients/consumer/ConsumerRebalanceListener;) ..........

Regardless the error telling about "java.lag.NoSuchMethodError: org.apache.flume.source.kafka.KafkaConsumer.subscribe(...)" I looked at the flume's lib folder kafka-client jar file and made sure the method subscribe(...) is in there.

So, the flume's agent still unable to function properly with kafka broker secured with SASL/PLAIN

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.