Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error while getting events from Kafka

Error while getting events from Kafka

New Contributor

Hello Experts - Can someone please help in troubleshooting this issue. Below is the error I'm getting

 

WARN org.apache.flume.channel.kafka.KafkaChannel: Error while getting events from Kafka. This is usually caused by trying to read a non-flume event. Ensure the setting for parseAsFlumeEvent is correct

java.lang.IndexOutOfBoundsException

        at java.io.ByteArrayInputStream.read(ByteArrayInputStream.java:180)

        at org.apache.avro.io.DirectBinaryDecoder.doReadBytes(DirectBinaryDecoder.java:184)

        at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:263)

        at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201)

        at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:430)

        at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422)

        at org.apache.avro.generic.GenericDatumReader.readMapKey(GenericDatumReader.java:335)

        at org.apache.avro.generic.GenericDatumReader.readMap(GenericDatumReader.java:321)

        at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:177)

        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)

        at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:240)

        at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:230)

        at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:174)

        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:152)

        at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144)

        at org.apache.flume.channel.kafka.KafkaChannel$KafkaTransaction.deserializeValue(KafkaChannel.java:635)

        at org.apache.flume.channel.kafka.KafkaChannel$KafkaTransaction.doTake(KafkaChannel.java:497)

        at org.apache.flume.channel.BasicTransactionSemantics.take(BasicTransactionSemantics.java:113)

        at org.apache.flume.channel.BasicChannelSemantics.take(BasicChannelSemantics.java:95)

        at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:362)

        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)

        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)

        at java.lang.Thread.run(Thread.java:748)

1 REPLY 1
Highlighted

Re: Error while getting events from Kafka

Cloudera Employee

Hello,

 

As per the log traces itself, "This is usually caused by trying to read a non-flume event."  It means 

Flume is getting events that have not been written by Flume source. 

 

So I hope your Kafka producer is not Flume or you may have multiple sources that write into same topic.

 

So in this scenario, please set 'parseAsFlumeEvent' as 'false'.

 

Please refer Flume document[1]

 

[1] https://flume.apache.org/FlumeUserGuide.html#kafka-channel

Don't have an account?
Coming from Hortonworks? Activate your account here