Custom jar not loaded with flume service - cloudera manager

[#1]

Hello,

tried  to configure flume as a service so that can be managed inside cloudera manager however I think that the custom jar from cloudera guide is not picked up. Exception from the flume log is following:

org.apache.flume.node.PollingPropertiesFileConfigurationProvider     

Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
    at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
    at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:40)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource

I have set $FLUME_HOME, set addional java parameters in cloudera manager pointing to my jar, changed permission on the jar but none resulted in any change. Do you have any other suggestions?

I have assumed that flume service in the cloudera manager is running by "flume" user. Is that correct?  
 

[#2]

What version of Cloudera Manager and CDH are you using?

The easiest way is to:
 - Copy third-party jars to a directory and specify this directory in Cloudera Manager's Flume Plugin directories configuration (in Flume service -> Configuration -> Agent -> Plugin directories)
 - Restart Flume service

You don't need to set FLUME_HOME explicitly when using Cloudera Manager.

[#3]

It does not work.  I have changed the permission from root:root to flume:flume but without success. The error still persist.

Cloudera version: Cloudera Standard 4.7.2

[#4 · Accepted solution]

I think the plugin directory layout in the README.md is incorrect. Could you try using the plugin directory layout documented in the Flume User Guide instead ?: http://archive.cloudera.com/cdh4/cdh/4/flume-ng/FlumeUserGuide.html#installing-third-party-plugins

your_plugins_dir/some_custom_name/lib/foo.jar

For example: If you specified /var/lib/flume-ng/plugins.d as the plugin directory (in CM's Flume Plugin directories configuration), then your jar location should be:

/var/lib/flume-ng/plugins.d/twitter-streaming/lib/flume-sources-1.0-SNAPSHOT.jar

thanks,
Mohit

[#5]

Your suggestion with a different path worked.. I had another issue with conflicting jars witch was solved with help of this post.

[#6]

I changed the path first, and didn't work, then I changed the permissions (even for directories and subdirectories) and then it worked, thanks.

[#7]

Hi All,

This is an issue with fully qualified name of the Class.

In the older versions, class name is:  com.cloudera.flume.source.TwitterSource

In the latest version of the flume TwitterSource is already shipped and no need to download separately. 

The class name is changed to org.apache.flume.source.twitter.TwitterSource

Please carefully change the class name, defitely It will work for you

Thanks

Naga

[#8]

Hi Naga

I changed the class name to org.apache.flume.source.twitter.TwitterSource in flume.conf file. I am still getting the same error.

Do you have any other suggestions?

Thanks

Yogesh

[#9]

I did exactly what you said moving /usr/lib/flume-ng/plugins.d/tweeterStream/flume-sources-1.0-SNAPSHOT.jar and still have the same message.

15/11/10 21:24:04 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:69)
	at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:42)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:322)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
	... 11 more

If I change to the "new" class org.apache.flume.source.twitter.TwitterSource as mentioned early, my keywords filters just does not work.

Thank you in advance.

[#10]

I am also having the same problem after I created directory and move the jar to the plugin folder:

Here is the details:

You can see I have the .jar file in the right lib folder:

 

My environment variables are:

export FLUME_HOME=/opt/flume
export FLUME_CONF_DIR=$FLUME_HOME/conf
export FLUME_CLASSPATH=$FLUME_HOME_DIR
export PATH=$FLUME_HOME/bin:$PATH

and I am doing this under root user, below is the error message:

16/06/28 10:48:21 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
	at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:40)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:65)

Can anyone shed any light on this with me? Thank you very much.