Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Flume plugin directories are not in classpath

Solved Go to solution

Flume plugin directories are not in classpath

Contributor

Hi,

 

A jar with custom flume components under default plugin folders /usr/lib/flume-ng/plugins.d or /var/lib/flume-ng/plugins.d is not picked up by Flume.

I don't event see these folders included in the classpath.

 

All works just fine if I add my jar to one of the default CP folders, e.g. 

/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/flume-ng/lib.

 

Did anyone have a similar issue?

 

Cheers,

Gin

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Flume plugin directories are not in classpath

Super Collaborator
Are you following this plugin directory architecture:

http://flume.apache.org/FlumeUserGuide.html#the-plugins-d-directory

If you look in the flume stderr.log, you should see it on the cmd line:
stderr.log:+ exec /opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.19/lib/flume-ng/bin/flume-ng agent --conf /var/run/cloudera-scm-agent/process/434-flume-AGENT --classpath /var/run/cloudera-scm-agent/process/434-flume-AGENT/hbase-conf:/var/run/cloudera-scm-agent/process/434-flume-AGENT/hadoop-conf: --conf-file /var/run/cloudera-scm-agent/process/434-flume-AGENT/flume.conf --name tier2 -Djava.net.preferIPv4Stack=true -Duser.home=/var/lib/flume-ng -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/FLUME-1_FLUME-AGENT-1_pid30399.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dflume.monitoring.type=HTTP -Dflume.monitoring.port=24001 --plugins-path /usr/lib/flume-ng/plugins.d:/var/lib/flume-ng/plugins.d

2 REPLIES 2

Re: Flume plugin directories are not in classpath

Super Collaborator
Are you following this plugin directory architecture:

http://flume.apache.org/FlumeUserGuide.html#the-plugins-d-directory

If you look in the flume stderr.log, you should see it on the cmd line:
stderr.log:+ exec /opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.19/lib/flume-ng/bin/flume-ng agent --conf /var/run/cloudera-scm-agent/process/434-flume-AGENT --classpath /var/run/cloudera-scm-agent/process/434-flume-AGENT/hbase-conf:/var/run/cloudera-scm-agent/process/434-flume-AGENT/hadoop-conf: --conf-file /var/run/cloudera-scm-agent/process/434-flume-AGENT/flume.conf --name tier2 -Djava.net.preferIPv4Stack=true -Duser.home=/var/lib/flume-ng -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/FLUME-1_FLUME-AGENT-1_pid30399.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dflume.monitoring.type=HTTP -Dflume.monitoring.port=24001 --plugins-path /usr/lib/flume-ng/plugins.d:/var/lib/flume-ng/plugins.d

Highlighted

Re: Flume plugin directories are not in classpath

Contributor

Spot on, I missed the directory structure and just dumped my jars under plugins.d.

 

The cmd line did not give me clues since it did not have anything related to plugin directories, e.g. "--plugins-path /usr/lib/flume-ng/plugins.d:/var/lib/flume-ng/plugins.d".

 

Thanks!