Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Flume plugin directories are not in classpath

avatar
Rising Star

Hi,

 

A jar with custom flume components under default plugin folders /usr/lib/flume-ng/plugins.d or /var/lib/flume-ng/plugins.d is not picked up by Flume.

I don't event see these folders included in the classpath.

 

All works just fine if I add my jar to one of the default CP folders, e.g. 

/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/flume-ng/lib.

 

Did anyone have a similar issue?

 

Cheers,

Gin

1 ACCEPTED SOLUTION

avatar
Are you following this plugin directory architecture:

http://flume.apache.org/FlumeUserGuide.html#the-plugins-d-directory

If you look in the flume stderr.log, you should see it on the cmd line:
stderr.log:+ exec /opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.19/lib/flume-ng/bin/flume-ng agent --conf /var/run/cloudera-scm-agent/process/434-flume-AGENT --classpath /var/run/cloudera-scm-agent/process/434-flume-AGENT/hbase-conf:/var/run/cloudera-scm-agent/process/434-flume-AGENT/hadoop-conf: --conf-file /var/run/cloudera-scm-agent/process/434-flume-AGENT/flume.conf --name tier2 -Djava.net.preferIPv4Stack=true -Duser.home=/var/lib/flume-ng -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/FLUME-1_FLUME-AGENT-1_pid30399.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dflume.monitoring.type=HTTP -Dflume.monitoring.port=24001 --plugins-path /usr/lib/flume-ng/plugins.d:/var/lib/flume-ng/plugins.d

View solution in original post

2 REPLIES 2

avatar
Are you following this plugin directory architecture:

http://flume.apache.org/FlumeUserGuide.html#the-plugins-d-directory

If you look in the flume stderr.log, you should see it on the cmd line:
stderr.log:+ exec /opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.19/lib/flume-ng/bin/flume-ng agent --conf /var/run/cloudera-scm-agent/process/434-flume-AGENT --classpath /var/run/cloudera-scm-agent/process/434-flume-AGENT/hbase-conf:/var/run/cloudera-scm-agent/process/434-flume-AGENT/hadoop-conf: --conf-file /var/run/cloudera-scm-agent/process/434-flume-AGENT/flume.conf --name tier2 -Djava.net.preferIPv4Stack=true -Duser.home=/var/lib/flume-ng -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/FLUME-1_FLUME-AGENT-1_pid30399.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dflume.monitoring.type=HTTP -Dflume.monitoring.port=24001 --plugins-path /usr/lib/flume-ng/plugins.d:/var/lib/flume-ng/plugins.d

avatar
Rising Star

Spot on, I missed the directory structure and just dumped my jars under plugins.d.

 

The cmd line did not give me clues since it did not have anything related to plugin directories, e.g. "--plugins-path /usr/lib/flume-ng/plugins.d:/var/lib/flume-ng/plugins.d".

 

Thanks!