ClassNotFoundException org.apache.flume.instrumentation.kafka.KafkaSinkCounter

[#1]

Hi all,

We are trying to write in a Kafka queue with Flume. We have HDP 2.2.4, with this Flume 1.5.2.2.2 is installed.

Below the kafka sink configuration:

occLogTcp.sinks.KAFKA.type = org.apache.flume.sink.kafka.KafkaSink
occLogTcp.sinks.KAFKA.topic = occTest
occLogTcp.sinks.KAFKA.brokerList = <broker_host_1>:9092,<broker_host_2>:9092
occLogTcp.sinks.KAFKA.requiredAcks = 1
occLogTcp.sinks.KAFKA.batchSize = 20
occLogTcp.sinks.KAFKA.channel = c1

Starting the flume agent throws the following error:

java.lang.NoClassDefFoundError:
org/apache/flume/instrumentation/kafka/KafkaSinkCounter 
at org.apache.flume.sink.kafka.KafkaSink.configure(KafkaSink.java:218) 
at org.apache.flume.conf.Configurables.configure(Configurables.java:41) 
at
org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:418) 
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103) 
at
org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) 
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) 
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) 
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) 
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) 
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
at java.lang.Thread.run(Thread.java:745)Caused by:
java.lang.ClassNotFoundException:
org.apache.flume.instrumentation.kafka.KafkaSinkCounter 
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) 
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
at java.security.AccessController.doPrivileged(Native Method) 
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:425) 
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:358) 
... 12 more

Thank you in advance, Christian

[#2]

It seems like the package org/apache/flume/instrumentation/kafka/ has been added in version 1.6. Therefore Flume 1.5.2 doesn't support the KafkaSink.

[#3 · Accepted solution]

we backported Kafka sink into 1.5.2, just make sure you are using HDP 2.3, I see you're on 2.2.4 which is quite old. Alternatively, try HDF powered by Apache nifi

https://community.hortonworks.com/questions/9425/is-flume-to-kafka-sink-supported-in-hdp-232.html

[#4]

Latest 2.3 release notes http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4.7/bk_HDP_RelNotes/content/errata_flume_kafka...

You do not need Flume 1.6 on HDP to have it working.