Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (2)
avatar
Master Guru

It's very easy to integrate Apache Flume with Apache NiFi, either as a source or destination (sink).

An example of using ExecuteFlumeSource and ExecuteFlumeSink in Apache NiFi. We use a Flume source to read Netcat and send to Slack. So you can send messages from Linux to a Slack chat. For the Flume sink to HDFS, we read data from a web feed. We can also connect our Flume Source to Flume Sink.

6183-flumenifi.png

Configuring a Flume Source

If you have configured Apache Flume configuration files, this is the same thing.

6192-flumenifi11.png

6194-flumenifi12.png

Data Received in Slack (Join and See Messages Live)

6193-nifislackflumenetcat.png

Checking Provenance Event

6186-flumenifi5.png

Configure The ExecuteFlumeSink to Store to HDFS

6185-flumenifi3.png

-rw-r--r-- 3 flume hdfs 28 2016-07-25 16:14 /flume/events/16-07-25/1610/00/events-.1469463242392

To monitor what's going on in the Flume side of the process: tail -f /var/log/flume/flume-a1.log

25 Jul 2016 16:14:38,854 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.batch.empty == 302
25 Jul 2016 16:14:38,854 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.batch.underflow == 1
25 Jul 2016 16:14:38,855 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.connection.closed.count == 1
25 Jul 2016 16:14:38,855 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.connection.creation.count == 1
25 Jul 2016 16:14:38,855 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.connection.failed.count == 0
25 Jul 2016 16:14:38,855 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.event.drain.attempt == 1
25 Jul 2016 16:14:38,855 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.event.drain.sucess == 1
25 Jul 2016 16:14:38,856 INFO  [agent-shutdown-hook] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.stop:83)  - Configuration provider stopping
25 Jul 2016 16:14:38,856 INFO  [agent-shutdown-hook] (org.apache.flume.source.NetcatSource.stop:190)  - Source stopping
25 Jul 2016 16:14:39,357 INFO  [agent-shutdown-hook] (org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink.stop:74)  - Stopping Flume Metrics Sink

Make sure you are not running regular Flume Agent of same name! They will clash!

Using the Netcat

telnet localhost 44444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Slack, Apache Nifi,
Apache Flume, Netcat
OK^]
telnet> quit
Connection closed.

Resources:

5,704 Views
0 Kudos
Comments
avatar
Contributor

I also have this requirement, can you give me a complete configuration? Thank you

avatar
Master Guru

You should not use Flume.   Flume and it's connectors are deprecated.   This and any flow can easily move to NiFi.

 

https://dev.to/tspannhw/migrating-apache-flume-flows-to-apache-nifi-jms-to-x-and-x-to-jms-1g02