Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Flume agent failed because dependencies were not found in classpath.

avatar
Contributor

2686-flume-envsh.png

2685-classnotfoundexception.png

org.apache.flume.sink.DefaultSinkFactory.create:42) - Creating instance of sink: hdfs-sink, type: hdfs 09 Mar 2016 02:07:33,594

ERROR

[conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:145)  (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:145)  - Failed to start agent because dependencies were not found in classpath. Error follows.
java.lang.NoClassDefFoundError: org/apache/hadoop/io/SequenceFile$CompressionType
        at org.apache.flume.sink.hdfs.HDFSEventSink.configure(HDFSEventSink.java:239)
        at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
        at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:413)
        at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:98)
        at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744) 

Caused by:

java.lang.ClassNotFoundException: org.apache.hadoop.io.SequenceFile$CompressionType
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

hdfsconf.png
1 ACCEPTED SOLUTION

avatar
Contributor

FLUME_CLASSPATH=/root/flume/lib/

copied comon jar files from hadoop folder to the flume folder.

cp /root/hadoop/share/hadoop/common/*.jar /root/flume/lib

cp /root/hadoop/share/hadoop/common/lib/*.jar /root/flume/lib

Now the above error is rectified.

View solution in original post

3 REPLIES 3

avatar
Contributor

FLUME_CLASSPATH=/root/flume/lib/

copied comon jar files from hadoop folder to the flume folder.

cp /root/hadoop/share/hadoop/common/*.jar /root/flume/lib

cp /root/hadoop/share/hadoop/common/lib/*.jar /root/flume/lib

Now the above error is rectified.

avatar
Contributor

Copied the below path in the flume-env.sh

FLUME_CLASSPATH="/home/hadoop/hadoop/share/hadoop/hdfs/"

Hdfs sink is working fine.

avatar
Rising Star

This comment fixed my problem..You are Awsome vikas