question Re: Flume plugin directories are not in classpath in Archives of Support Questions (Read Only)

Flume plugin directories are not in classpath

ginsul — Fri, 16 Sep 2022 11:14:52 GMT

Hi,

A jar with custom flume components under default plugin folders /usr/lib/flume-ng/plugins.d or /var/lib/flume-ng/plugins.d is not picked up by Flume.

I don't event see these folders included in the classpath.

All works just fine if I add my jar to one of the default CP folders, e.g.

/opt/cloudera/parcels/CDH-5.9.0-1.cdh5.9.0.p0.23/lib/flume-ng/lib.

Did anyone have a similar issue?

Cheers,

Gin

Re: Flume plugin directories are not in classpath

pdvorak — Mon, 13 Mar 2017 16:10:01 GMT

Are you following this plugin directory architecture:

http://flume.apache.org/FlumeUserGuide.html#the-plugins-d-directory

If you look in the flume stderr.log, you should see it on the cmd line:
stderr.log:+ exec /opt/cloudera/parcels/CDH-5.10.2-1.cdh5.10.2.p0.19/lib/flume-ng/bin/flume-ng agent --conf /var/run/cloudera-scm-agent/process/434-flume-AGENT --classpath /var/run/cloudera-scm-agent/process/434-flume-AGENT/hbase-conf:/var/run/cloudera-scm-agent/process/434-flume-AGENT/hadoop-conf: --conf-file /var/run/cloudera-scm-agent/process/434-flume-AGENT/flume.conf --name tier2 -Djava.net.preferIPv4Stack=true -Duser.home=/var/lib/flume-ng -Xms1073741824 -Xmx1073741824 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/FLUME-1_FLUME-AGENT-1_pid30399.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh -Dflume.monitoring.type=HTTP -Dflume.monitoring.port=24001 --plugins-path /usr/lib/flume-ng/plugins.d:/var/lib/flume-ng/plugins.d

Re: Flume plugin directories are not in classpath

ginsul — Tue, 14 Mar 2017 11:49:15 GMT

Spot on, I missed the directory structure and just dumped my jars under plugins.d.

The cmd line did not give me clues since it did not have anything related to plugin directories, e.g. "--plugins-path /usr/lib/flume-ng/plugins.d:/var/lib/flume-ng/plugins.d".

Thanks!