question Re: Custom jar not loaded with flume service - cloudera manager in Archives of Support Questions (Read Only)

Custom jar not loaded with flume service - cloudera manager

mertez — Fri, 16 Sep 2022 08:52:07 GMT

Hello,

tried to configure flume as a service so that can be managed inside cloudera manager however I think that the custom jar from cloudera guide is not picked up. Exception from the flume log is following:

org.apache.flume.node.PollingPropertiesFileConfigurationProvider     

Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
    at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
    at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:40)
    at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
    at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
    at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
    at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource

I have set $FLUME_HOME, set addional java parameters in cloudera manager pointing to my jar, changed permission on the jar but none resulted in any change. Do you have any other suggestions?

I have assumed that flume service in the cloudera manager is running by "flume" user. Is that correct?

Re: Custom jar not loaded with flume service - cloudera manager

mohit — Tue, 07 Jan 2014 23:42:02 GMT

What version of Cloudera Manager and CDH are you using?

The easiest way is to:
- Copy third-party jars to a directory and specify this directory in Cloudera Manager's Flume Plugin directories configuration (in Flume service -> Configuration -> Agent -> Plugin directories)
- Restart Flume service

You don't need to set FLUME_HOME explicitly when using Cloudera Manager.

Re: Custom jar not loaded with flume service - cloudera manager

mertez — Wed, 08 Jan 2014 20:13:28 GMT

It does not work. I have changed the permission from root:root to flume:flume but without success. The error still persist.

Cloudera version: Cloudera Standard 4.7.2

Re: Custom jar not loaded with flume service - cloudera manager

mohit — Wed, 08 Jan 2014 22:29:44 GMT

I think the plugin directory layout in the README.md is incorrect. Could you try using the plugin directory layout documented in the Flume User Guide instead ?: http://archive.cloudera.com/cdh4/cdh/4/flume-ng/FlumeUserGuide.html#installing-third-party-plugins

your_plugins_dir/some_custom_name/lib/foo.jar

For example: If you specified /var/lib/flume-ng/plugins.d as the plugin directory (in CM's Flume Plugin directories configuration), then your jar location should be:

/var/lib/flume-ng/plugins.d/twitter-streaming/lib/flume-sources-1.0-SNAPSHOT.jar

thanks,
Mohit

Re: Custom jar not loaded with flume service - cloudera manager

mertez — Tue, 14 Jan 2014 10:06:18 GMT

Your suggestion with a different path worked.. I had another issue with conflicting jars witch was solved with help of this post.

Re: Custom jar not loaded with flume service - cloudera manager

Kulssaka — Thu, 26 Jun 2014 15:00:53 GMT

I changed the path first, and didn't work, then I changed the permissions (even for directories and subdirectories) and then it worked, thanks.

Re: Custom jar not loaded with flume service - cloudera manager

inelu — Thu, 03 Sep 2015 18:54:26 GMT

Hi All,

This is an issue with fully qualified name of the Class.

In the older versions, class name is: com.cloudera.flume.source.TwitterSource

In the latest version of the flume TwitterSource is already shipped and no need to download separately.

The class name is changed to org.apache.flume.source.twitter.TwitterSource

Please carefully change the class name, defitely It will work for you

Thanks

Naga

Re: Custom jar not loaded with flume service - cloudera manager

yogcdh — Mon, 12 Oct 2015 05:19:19 GMT

Hi Naga

I changed the class name to org.apache.flume.source.twitter.TwitterSource in flume.conf file. I am still getting the same error.

Do you have any other suggestions?

Thanks

Yogesh

Re: Custom jar not loaded with flume service - cloudera manager

brtt — Wed, 11 Nov 2015 02:30:23 GMT

I did exactly what you said moving /usr/lib/flume-ng/plugins.d/tweeterStream/flume-sources-1.0-SNAPSHOT.jar and still have the same message.

15/11/10 21:24:04 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:69)
	at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:42)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:322)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:97)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
	... 11 more

If I change to the "new" class org.apache.flume.source.twitter.TwitterSource as mentioned early, my keywords filters just does not work.

Thank you in advance.

Re: Custom jar not loaded with flume service - cloudera manager

PasLeChoix — Tue, 28 Jun 2016 15:12:16 GMT

I am also having the same problem after I created directory and move the jar to the plugin folder:

Here is the details:

You can see I have the .jar file in the right lib folder:

My environment variables are:

export FLUME_HOME=/opt/flume
export FLUME_CONF_DIR=$FLUME_HOME/conf
export FLUME_CLASSPATH=$FLUME_HOME_DIR
export PATH=$FLUME_HOME/bin:$PATH

and I am doing this under root user, below is the error message:

16/06/28 10:48:21 ERROR node.PollingPropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows.
org.apache.flume.FlumeException: Unable to load source type: com.cloudera.flume.source.TwitterSource, class: com.cloudera.flume.source.TwitterSource
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:67)
	at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:40)
	at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:327)
	at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
	at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: com.cloudera.flume.source.TwitterSource
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:264)
	at org.apache.flume.source.DefaultSourceFactory.getClass(DefaultSourceFactory.java:65)

Can anyone shed any light on this with me? Thank you very much.