Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Custom jar not loaded with flume service - cloudera manager

SOLVED Go to solution

Custom jar not loaded with flume service - cloudera manager

New Contributor

Hello,

tried  to configure flume as a service so that can be managed inside cloudera manager however I think that the custom jar from cloudera guide is not picked up. Exception from the flume log is following:

org.apache.flume.node.PollingPropertiesFileConfigurationProvider     

Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
    at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
    at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:40)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource

 

I have set $FLUME_HOME, set addional java parameters in cloudera manager pointing to my jar, changed permission on the jar but none resulted in any change. Do you have any other suggestions?

 

I have assumed that flume service in the cloudera manager is running by "flume" user. Is that correct?  
 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Custom jar not loaded with flume service - cloudera manager

Cloudera Employee

I think the plugin directory layout in the README.md is incorrect. Could you try using the plugin directory layout documented in the Flume User Guide instead ?: http://archive.cloudera.com/cdh4/cdh/4/flume-ng/FlumeUserGuide.html#installing-third-party-plugins

 

your_plugins_dir/some_custom_name/lib/foo.jar

 

For example: If you specified /var/lib/flume-ng/plugins.d as the plugin directory (in CM's Flume Plugin directories configuration), then your jar location should be:

/var/lib/flume-ng/plugins.d/twitter-streaming/lib/flume-sources-1.0-SNAPSHOT.jar

 

thanks,
Mohit

9 REPLIES 9

Re: Custom jar not loaded with flume service - cloudera manager

Cloudera Employee

What version of Cloudera Manager and CDH are you using?

 

The easiest way is to:
 - Copy third-party jars to a directory and specify this directory in Cloudera Manager's Flume Plugin directories configuration (in Flume service -> Configuration -> Agent -> Plugin directories)
 - Restart Flume service

 

You don't need to set FLUME_HOME explicitly when using Cloudera Manager.

Re: Custom jar not loaded with flume service - cloudera manager

New Contributor

It does not work.  I have changed the permission from root:root to flume:flume but without success. The error still persist.

 

Cloudera version: Cloudera Standard 4.7.2

Re: Custom jar not loaded with flume service - cloudera manager

Cloudera Employee

I think the plugin directory layout in the README.md is incorrect. Could you try using the plugin directory layout documented in the Flume User Guide instead ?: http://archive.cloudera.com/cdh4/cdh/4/flume-ng/FlumeUserGuide.html#installing-third-party-plugins

 

your_plugins_dir/some_custom_name/lib/foo.jar

 

For example: If you specified /var/lib/flume-ng/plugins.d as the plugin directory (in CM's Flume Plugin directories configuration), then your jar location should be:

/var/lib/flume-ng/plugins.d/twitter-streaming/lib/flume-sources-1.0-SNAPSHOT.jar

 

thanks,
Mohit

Re: Custom jar not loaded with flume service - cloudera manager

New Contributor

Your suggestion with a different path worked.. I had another issue with conflicting jars witch was solved with help of this post.

Highlighted

Re: Custom jar not loaded with flume service - cloudera manager

Explorer

I changed the path first, and didn't work, then I changed the permissions (even for directories and subdirectories) and then it worked, thanks.

--
Lefevre Kevin

Re: Custom jar not loaded with flume service - cloudera manager

New Contributor

Hi All,

 

This is an issue with fully qualified name of the Class.

 

In the older versions, class name is:  com.cloudera.flume.source.TwitterSource

 

In the latest version of the flume TwitterSource is already shipped and no need to download separately. 

 

The class name is changed to org.apache.flume.source.twitter.TwitterSource

 

Please carefully change the class name, defitely It will work for you

 

Thanks

Naga

Re: Custom jar not loaded with flume service - cloudera manager

New Contributor

Hi Naga

 

I changed the class name to org.apache.flume.source.twitter.TwitterSource in flume.conf file. I am still getting the same error.

Do you have any other suggestions?

 

Thanks

Yogesh

Re: Custom jar not loaded with flume service - cloudera manager

New Contributor

I did exactly what you said moving /usr/lib/flume-ng/plugins.d/tweeterStream/flume-sources-1.0-SNAPSHOT.jar and still have the same message.

 

15/11/10 21:24:04 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:69)
	at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:42)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:322)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
	... 11 more

If I change to the "new" class org.apache.flume.source.twitter.TwitterSource as mentioned early, my keywords filters just does not work.

 

Thank you in advance.

Re: Custom jar not loaded with flume service - cloudera manager

Explorer

I am also having the same problem after I created directory and move the jar to the plugin folder:

Here is the details:

Screenshot - 2016_6_28 , 11_03_10.jpg

 

You can see I have the .jar file in the right lib folder:

 

 Screenshot - 2016_6_28 , 11_06_42.jpg

 

My environment variables are:

 

export FLUME_HOME=/opt/flume
export FLUME_CONF_DIR=$FLUME_HOME/conf
export FLUME_CLASSPATH=$FLUME_HOME_DIR
export PATH=$FLUME_HOME/bin:$PATH

and I am doing this under root user, below is the error message:

 

16/06/28 10:48:21 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
	at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:40)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:65)

 

Can anyone shed any light on this with me? Thank you very much.