Reply
New Contributor
Posts: 3
Registered: ‎01-07-2014
Accepted Solution

Custom jar not loaded with flume service - cloudera manager

Hello,

tried  to configure flume as a service so that can be managed inside cloudera manager however I think that the custom jar from cloudera guide is not picked up. Exception from the flume log is following:

org.apache.flume.node.PollingPropertiesFileConfigurationProvider     

Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
    at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
    at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:40)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource

 

I have set $FLUME_HOME, set addional java parameters in cloudera manager pointing to my jar, changed permission on the jar but none resulted in any change. Do you have any other suggestions?

 

I have assumed that flume service in the cloudera manager is running by "flume" user. Is that correct?  
 

Cloudera Employee
Posts: 16
Registered: ‎01-07-2014

Re: Custom jar not loaded with flume service - cloudera manager

What version of Cloudera Manager and CDH are you using?

 

The easiest way is to:
 - Copy third-party jars to a directory and specify this directory in Cloudera Manager's Flume Plugin directories configuration (in Flume service -> Configuration -> Agent -> Plugin directories)
 - Restart Flume service

 

You don't need to set FLUME_HOME explicitly when using Cloudera Manager.

New Contributor
Posts: 3
Registered: ‎01-07-2014

Re: Custom jar not loaded with flume service - cloudera manager

[ Edited ]

It does not work.  I have changed the permission from root:root to flume:flume but without success. The error still persist.

 

Cloudera version: Cloudera Standard 4.7.2

Cloudera Employee
Posts: 16
Registered: ‎01-07-2014

Re: Custom jar not loaded with flume service - cloudera manager

I think the plugin directory layout in the README.md is incorrect. Could you try using the plugin directory layout documented in the Flume User Guide instead ?: http://archive.cloudera.com/cdh4/cdh/4/flume-ng/FlumeUserGuide.html#installing-third-party-plugins

 

your_plugins_dir/some_custom_name/lib/foo.jar

 

For example: If you specified /var/lib/flume-ng/plugins.d as the plugin directory (in CM's Flume Plugin directories configuration), then your jar location should be:

/var/lib/flume-ng/plugins.d/twitter-streaming/lib/flume-sources-1.0-SNAPSHOT.jar

 

thanks,
Mohit

New Contributor
Posts: 3
Registered: ‎01-07-2014

Re: Custom jar not loaded with flume service - cloudera manager

Your suggestion with a different path worked.. I had another issue with conflicting jars witch was solved with help of this post.

Explorer
Posts: 18
Registered: ‎04-10-2014

Re: Custom jar not loaded with flume service - cloudera manager

I changed the path first, and didn't work, then I changed the permissions (even for directories and subdirectories) and then it worked, thanks.

--
Lefevre Kevin
Cloudera Support
Posts: 3
Registered: ‎07-07-2015

Re: Custom jar not loaded with flume service - cloudera manager

Hi All,

 

This is an issue with fully qualified name of the Class.

 

In the older versions, class name is:  com.cloudera.flume.source.TwitterSource

 

In the latest version of the flume TwitterSource is already shipped and no need to download separately. 

 

The class name is changed to org.apache.flume.source.twitter.TwitterSource

 

Please carefully change the class name, defitely It will work for you

 

Thanks

Naga

New Contributor
Posts: 1
Registered: ‎10-11-2015

Re: Custom jar not loaded with flume service - cloudera manager

Hi Naga

 

I changed the class name to org.apache.flume.source.twitter.TwitterSource in flume.conf file. I am still getting the same error.

Do you have any other suggestions?

 

Thanks

Yogesh

New Contributor
Posts: 1
Registered: ‎11-10-2015

Re: Custom jar not loaded with flume service - cloudera manager

I did exactly what you said moving /usr/lib/flume-ng/plugins.d/tweeterStream/flume-sources-1.0-SNAPSHOT.jar and still have the same message.

 

15/11/10 21:24:04 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:69)
	at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:42)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:322)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
	... 11 more

If I change to the "new" class org.apache.flume.source.twitter.TwitterSource as mentioned early, my keywords filters just does not work.

 

Thank you in advance.

Explorer
Posts: 14
Registered: ‎05-22-2016

Re: Custom jar not loaded with flume service - cloudera manager

I am also having the same problem after I created directory and move the jar to the plugin folder:

Here is the details:

Screenshot - 2016_6_28 , 11_03_10.jpg

 

You can see I have the .jar file in the right lib folder:

 

 Screenshot - 2016_6_28 , 11_06_42.jpg

 

My environment variables are:

 

export FLUME_HOME=/opt/flume
export FLUME_CONF_DIR=$FLUME_HOME/conf
export FLUME_CLASSPATH=$FLUME_HOME_DIR
export PATH=$FLUME_HOME/bin:$PATH

and I am doing this under root user, below is the error message:

 

16/06/28 10:48:21 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
	at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:40)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:65)

 

Can anyone shed any light on this with me? Thank you very much.

Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.