Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ActiveMQ to Flume to HDFS

Solved Go to solution

ActiveMQ to Flume to HDFS

Explorer

Hi All,

 

I'm trying to connect my Flume to ActiveMQ and then save the data in HDFS. When I start the flume service in cloudera manager, I got the following exception:

 

2016-04-18 17:00:39,935 ERROR org.apache.flume.node.PollingPropertiesFileConfigurationProvider: Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: javax/jms/JMSException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:42)
at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:322)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: javax.jms.JMSException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 14 more

 

 

My flume conf in cloudera manager, is this correct:

 

flume1.sources = activemq-source-1
flume1.channels = activemq-channel-1
flume1.sinks = hdfs-sink-1

flume1.sources.activemq-source-1.type = jms
flume1.sources.activemq-source-1.providerURL = tcp://10.228.200.251:61616
flume1.sources.activemq-source-1.destinationName = .broker.pub.1.00137a00000275e1.01_ILLUMINANCEMEASURE_S.
flume1.sources.activemq-source-1.destinationType = TOPIC
flume1.sources.activemq-source-1.channels = activemq-channel-1

flume1.sinks.hdfs-sink-1.channel = activemq-channel-1
flume1.sinks.hdfs-sink-1.type = hdfs
flume1.sinks.hdfs-sink-1.hdfs.writeFormat = Text
flume1.sinks.hdfs-sink-1.hdfs.fileType = DataStream
flume1.sinks.hdfs-sink-1.hdfs.filePrefix = 00137a00000275e1
flume1.sinks.hdfs-sink-1.hdfs.useLocalTimeStamp = true
flume1.sinks.hdfs-sink-1.hdfs.path = /user/activemq/%{topic}
flume1.sinks.hdfs-sink-1.hdfs.rollCount=100
flume1.sinks.hdfs-sink-1.hdfs.rollSize=0

# specify the capacity of the memory channel.
flume1.channels.activemq-channel-1.type = memory
flume1.channels.activemq-channel-1.capacity = 10000
flume1.channels.activemq-channel-1.transactionCapacity = 10000

 

May I know how can I resolve the NoClassDefFound Exception? Do I still need to add FLUME_CLASSPATH if I added/started service in Cloudera manager? Is there any third party jar that I need to download to fix the javax/jms/JMSException?

 

Any help is highly appreciated.

 

Thank You.

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: ActiveMQ to Flume to HDFS

Super Collaborator
The javax/jms/JMSException class is found in the geronimo-jms jar file, which isn't included by default in the flume classpath, as it is a third party jar. To include that, you would need to use the flume plugin architecture [1] to add any of the jars needed by activemq

-pd

[1] http://flume.apache.org/FlumeUserGuide.html#installing-third-party-plugins
5 REPLIES 5

Re: ActiveMQ to Flume to HDFS

Super Collaborator
The javax/jms/JMSException class is found in the geronimo-jms jar file, which isn't included by default in the flume classpath, as it is a third party jar. To include that, you would need to use the flume plugin architecture [1] to add any of the jars needed by activemq

-pd

[1] http://flume.apache.org/FlumeUserGuide.html#installing-third-party-plugins

Re: ActiveMQ to Flume to HDFS

Explorer

Hi Pdvorak,

 

Thank you for your reply.

 

I tried adding the three jars needed (copied from activeMQ) to the plugins.d folder (path from the CM Flume Configuration page), restarted the flume service, but I am stil getting the same error.

 

  • geronimo-spec-jms.jar
  • geronimo-spec-jta.jar
  • geronimo-spec-j2ee-management.jar

Did I miss anything?

 

Thank You.

Re: ActiveMQ to Flume to HDFS

Explorer
Apparently I put in the wrong folder. It works! Thanks!

Re: ActiveMQ to Flume to HDFS

Explorer
Hi ,
What is the path you copied the jms jars. Please let me know

Re: ActiveMQ to Flume to HDFS

New Contributor

teravaidya

 

By default, the Flume agent plugins reside in this directories:

 

/usr/lib/flume-ng/plugins.d

 

/var/lib/flume-ng/plugins.d

 

 

Don't have an account?
Coming from Hortonworks? Activate your account here