ActiveMQ to Flume to HDFS

[#1]

Hi All,

I'm trying to connect my Flume to ActiveMQ and then save the data in HDFS. When I start the flume service in cloudera manager, I got the following exception:

2016-04-18 17:00:39,935 ERROR org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: javax/jms/JMSException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:42)
at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:322)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: javax.jms.JMSException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 14 more

My flume conf in cloudera manager, is this correct:

flume1.sources = activemq-source-1
flume1.channels = activemq-channel-1
flume1.sinks = hdfs-sink-1

flume1.sources.activemq-source-1.type = jms
flume1.sources.activemq-source-1.providerURL = tcp://10.228.200.251:61616
flume1.sources.activemq-source-1.destinationName = .broker.pub.1.00137a00000275e1.01_ILLUMINANCEMEASURE_S.
flume1.sources.activemq-source-1.destinationType = TOPIC
flume1.sources.activemq-source-1.channels = activemq-channel-1

flume1.sinks.hdfs-sink-1.channel = activemq-channel-1
flume1.sinks.hdfs-sink-1.type = hdfs
flume1.sinks.hdfs-sink-1.hdfs.writeFormat = Text
flume1.sinks.hdfs-sink-1.hdfs.fileType = DataStream
flume1.sinks.hdfs-sink-1.hdfs.filePrefix = 00137a00000275e1
flume1.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true
flume1.sinks.hdfs-sink-1.hdfs.path = /user/activemq/%{topic}
flume1.sinks.hdfs-sink-1.hdfs.rollCount=100
flume1.sinks.hdfs-sink-1.hdfs.rollSize=0

# specify the capacity of the memory channel.
flume1.channels.activemq-channel-1.type = memory
flume1.channels.activemq-channel-1.capacity = 10000
flume1.channels.activemq-channel-1.transactionCapacity = 10000

May I know how can I resolve the NoClassDefFound Exception? Do I still need to add FLUME_CLASSPATH if I added/started service in Cloudera manager? Is there any third party jar that I need to download to fix the javax/jms/JMSException?

Any help is highly appreciated.

Thank You.

[#2 · Accepted solution]

The javax/jms/JMSException class is found in the geronimo-jms jar file, which isn't included by default in the flume classpath, as it is a third party jar. To include that, you would need to use the flume plugin architecture [1] to add any of the jars needed by activemq

-pd

[1] http://flume.apache.org/FlumeUserGuide.html#installing-third-party-plugins

[#3]

Hi Pdvorak,

Thank you for your reply.

I tried adding the three jars needed (copied from activeMQ) to the plugins.d folder (path from the CM Flume Configuration page), restarted the flume service, but I am stil getting the same error.

• geronimo-spec-jms.jar
• geronimo-spec-jta.jar
• geronimo-spec-j2ee-management.jar

Did I miss anything?

Thank You.

[#4]

Apparently I put in the wrong folder. It works! Thanks!

[#5]

Hi ,
What is the path you copied the jms jars. Please let me know

[#6]

teravaidya

By default, the Flume agent plugins reside in this directories:

/usr/lib/flume-ng/plugins.d

/var/lib/flume-ng/plugins.d