Posts: 15
Registered: ‎06-09-2015
Accepted Solution

ActiveMQ to Flume to HDFS

[ Edited ]

Hi All,


I'm trying to connect my Flume to ActiveMQ and then save the data in HDFS. When I start the flume service in cloudera manager, I got the following exception:


2016-04-18 17:00:39,935 ERROR org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: javax/jms/JMSException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(
at org.apache.flume.source.DefaultSourceFactory.getClass(
at org.apache.flume.source.DefaultSourceFactory.create(
at org.apache.flume.node.AbstractConfigurationProvider.loadSources(
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$
at java.util.concurrent.Executors$
at java.util.concurrent.FutureTask.runAndReset(
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(
at java.util.concurrent.ScheduledThreadPoolExecutor$
at java.util.concurrent.ThreadPoolExecutor.runWorker(
at java.util.concurrent.ThreadPoolExecutor$
Caused by: java.lang.ClassNotFoundException: javax.jms.JMSException
at Method)
at java.lang.ClassLoader.loadClass(
at sun.misc.Launcher$AppClassLoader.loadClass(
at java.lang.ClassLoader.loadClass(
... 14 more



My flume conf in cloudera manager, is this correct:


flume1.sources = activemq-source-1
flume1.channels = activemq-channel-1
flume1.sinks = hdfs-sink-1

flume1.sources.activemq-source-1.type = jms
flume1.sources.activemq-source-1.providerURL = tcp://
flume1.sources.activemq-source-1.destinationName =
flume1.sources.activemq-source-1.destinationType = TOPIC
flume1.sources.activemq-source-1.channels = activemq-channel-1 = activemq-channel-1
flume1.sinks.hdfs-sink-1.type = hdfs
flume1.sinks.hdfs-sink-1.hdfs.writeFormat = Text
flume1.sinks.hdfs-sink-1.hdfs.fileType = DataStream
flume1.sinks.hdfs-sink-1.hdfs.filePrefix = 00137a00000275e1
flume1.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true
flume1.sinks.hdfs-sink-1.hdfs.path = /user/activemq/%{topic}

# specify the capacity of the memory channel.
flume1.channels.activemq-channel-1.type = memory
flume1.channels.activemq-channel-1.capacity = 10000
flume1.channels.activemq-channel-1.transactionCapacity = 10000


May I know how can I resolve the NoClassDefFound Exception? Do I still need to add FLUME_CLASSPATH if I added/started service in Cloudera manager? Is there any third party jar that I need to download to fix the javax/jms/JMSException?


Any help is highly appreciated.


Thank You.


Cloudera Employee
Posts: 255
Registered: ‎01-09-2014

Re: ActiveMQ to Flume to HDFS

The javax/jms/JMSException class is found in the geronimo-jms jar file, which isn't included by default in the flume classpath, as it is a third party jar. To include that, you would need to use the flume plugin architecture [1] to add any of the jars needed by activemq


Posts: 15
Registered: ‎06-09-2015

Re: ActiveMQ to Flume to HDFS

Hi Pdvorak,


Thank you for your reply.


I tried adding the three jars needed (copied from activeMQ) to the plugins.d folder (path from the CM Flume Configuration page), restarted the flume service, but I am stil getting the same error.


  • geronimo-spec-jms.jar
  • geronimo-spec-jta.jar
  • geronimo-spec-j2ee-management.jar

Did I miss anything?


Thank You.

Posts: 15
Registered: ‎06-09-2015

Re: ActiveMQ to Flume to HDFS

Apparently I put in the wrong folder. It works! Thanks!
New Contributor
Posts: 3
Registered: ‎05-16-2016

Re: ActiveMQ to Flume to HDFS

Hi ,
What is the path you copied the jms jars. Please let me know
New Contributor
Posts: 1
Registered: ‎01-19-2017

Re: ActiveMQ to Flume to HDFS



By default, the Flume agent plugins reside in this directories:







New solutions