Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

ActiveMQ to Flume to HDFS

avatar
Contributor

Hi All,

 

I'm trying to connect my Flume to ActiveMQ and then save the data in HDFS. When I start the flume service in cloudera manager, I got the following exception:

 

2016-04-18 17:00:39,935 ERROR org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: javax/jms/JMSException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:42)
at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:322)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: javax.jms.JMSException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 14 more

 

 

My flume conf in cloudera manager, is this correct:

 

flume1.sources = activemq-source-1
flume1.channels = activemq-channel-1
flume1.sinks = hdfs-sink-1

flume1.sources.activemq-source-1.type = jms
flume1.sources.activemq-source-1.providerURL = tcp://10.228.200.251:61616
flume1.sources.activemq-source-1.destinationName = .broker.pub.1.00137a00000275e1.01_ILLUMINANCEMEASURE_S.
flume1.sources.activemq-source-1.destinationType = TOPIC
flume1.sources.activemq-source-1.channels = activemq-channel-1

flume1.sinks.hdfs-sink-1.channel = activemq-channel-1
flume1.sinks.hdfs-sink-1.type = hdfs
flume1.sinks.hdfs-sink-1.hdfs.writeFormat = Text
flume1.sinks.hdfs-sink-1.hdfs.fileType = DataStream
flume1.sinks.hdfs-sink-1.hdfs.filePrefix = 00137a00000275e1
flume1.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true
flume1.sinks.hdfs-sink-1.hdfs.path = /user/activemq/%{topic}
flume1.sinks.hdfs-sink-1.hdfs.rollCount=100
flume1.sinks.hdfs-sink-1.hdfs.rollSize=0

# specify the capacity of the memory channel.
flume1.channels.activemq-channel-1.type = memory
flume1.channels.activemq-channel-1.capacity = 10000
flume1.channels.activemq-channel-1.transactionCapacity = 10000

 

May I know how can I resolve the NoClassDefFound Exception? Do I still need to add FLUME_CLASSPATH if I added/started service in Cloudera manager? Is there any third party jar that I need to download to fix the javax/jms/JMSException?

 

Any help is highly appreciated.

 

Thank You.

 

1 ACCEPTED SOLUTION

avatar
The javax/jms/JMSException class is found in the geronimo-jms jar file, which isn't included by default in the flume classpath, as it is a third party jar. To include that, you would need to use the flume plugin architecture [1] to add any of the jars needed by activemq

-pd

[1] http://flume.apache.org/FlumeUserGuide.html#installing-third-party-plugins

View solution in original post

5 REPLIES 5

avatar
The javax/jms/JMSException class is found in the geronimo-jms jar file, which isn't included by default in the flume classpath, as it is a third party jar. To include that, you would need to use the flume plugin architecture [1] to add any of the jars needed by activemq

-pd

[1] http://flume.apache.org/FlumeUserGuide.html#installing-third-party-plugins

avatar
Contributor

Hi Pdvorak,

 

Thank you for your reply.

 

I tried adding the three jars needed (copied from activeMQ) to the plugins.d folder (path from the CM Flume Configuration page), restarted the flume service, but I am stil getting the same error.

 

  • geronimo-spec-jms.jar
  • geronimo-spec-jta.jar
  • geronimo-spec-j2ee-management.jar

Did I miss anything?

 

Thank You.

avatar
Contributor
Apparently I put in the wrong folder. It works! Thanks!

avatar
Explorer
Hi ,
What is the path you copied the jms jars. Please let me know

avatar
New Contributor

teravaidya

 

By default, the Flume agent plugins reside in this directories:

 

/usr/lib/flume-ng/plugins.d

 

/var/lib/flume-ng/plugins.d