question Re: how to start flume agent with the monitor port on? in Archives of Support Questions (Read Only)

how to start flume agent with the monitor port on?

tanglei_0312 — Tue, 16 Jan 2018 16:14:02 GMT

The open source flume can start with a monitor port on, for searching flume state. In ambari, the start command is:

{flume_bin} agent --name {{0}} --conf {{1}} --conf-file {{2}}{{3}} > {flume_log_dir}/{{4}}.out 2>&1

Is it possible to add a properties:

-Dflume.monitoring.type=http -D flume.monitoring.port=xxx

Re: how to start flume agent with the monitor port on?

asirna — Tue, 16 Jan 2018 16:37:19 GMT

@Lei Tang,

Yes. You can add these properties. Check this link for more info

Thanks,

Aditya

Re: how to start flume agent with the monitor port on?

tanglei_0312 — Tue, 16 Jan 2018 17:22:33 GMT

Thanks.

I wander if there are any configurations to run flume agent using ambari to enable monitoring. Or this is not application scenario of ambari.

Re: how to start flume agent with the monitor port on?

asirna — Tue, 16 Jan 2018 18:43:41 GMT

@Lei Tang,

Currently Ambari doesn't allow passing monitoring type and port from GUI. You have to pass it through command line.

Below is the snippet from (/var/lib/ambari-server/resources/common-services/FLUME/1.4.0.2.0/package/scripts/flume.py). You can see that monitoring type is added only when AMS is enabled.

if params.has_metric_collector:
  extra_args = '-Dflume.monitoring.type=org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink ' \
               '-Dflume.monitoring.node={0}:{1}'
  # TODO check if this is used.
  extra_args = extra_args.format(params.metric_collector_host, params.metric_collector_port)

If you want to hack it, you can put the values here in the file and restart ambari server .

Thanks,

Aditya

Re: how to start flume agent with the monitor port on?

tanglei_0312 — Tue, 16 Jan 2018 18:54:08 GMT

Thank you for your help.

Re: how to start flume agent with the monitor port on?

asirna — Tue, 16 Jan 2018 19:00:12 GMT

@Lei Tang,

Glad that it helped. Can you please accept the answer by clicking "Accept" button. This will be really useful for other community users to find the answer quickly