Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (2)
Super Guru

It's very easy to integrate Apache Flume with Apache NiFi, either as a source or destination (sink).

An example of using ExecuteFlumeSource and ExecuteFlumeSink in Apache NiFi. We use a Flume source to read Netcat and send to Slack. So you can send messages from Linux to a Slack chat. For the Flume sink to HDFS, we read data from a web feed. We can also connect our Flume Source to Flume Sink.

6183-flumenifi.png

Configuring a Flume Source

If you have configured Apache Flume configuration files, this is the same thing.

6192-flumenifi11.png

6194-flumenifi12.png

Data Received in Slack (Join and See Messages Live)

6193-nifislackflumenetcat.png

Checking Provenance Event

6186-flumenifi5.png

Configure The ExecuteFlumeSink to Store to HDFS

6185-flumenifi3.png

-rw-r--r-- 3 flume hdfs 28 2016-07-25 16:14 /flume/events/16-07-25/1610/00/events-.1469463242392

To monitor what's going on in the Flume side of the process: tail -f /var/log/flume/flume-a1.log

25 Jul 2016 16:14:38,854 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.batch.empty == 302
25 Jul 2016 16:14:38,854 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.batch.underflow == 1
25 Jul 2016 16:14:38,855 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.connection.closed.count == 1
25 Jul 2016 16:14:38,855 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.connection.creation.count == 1
25 Jul 2016 16:14:38,855 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.connection.failed.count == 0
25 Jul 2016 16:14:38,855 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.event.drain.attempt == 1
25 Jul 2016 16:14:38,855 INFO  [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177)  - Shutdown Metric for type: SINK, name: k1. sink.event.drain.sucess == 1
25 Jul 2016 16:14:38,856 INFO  [agent-shutdown-hook] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.stop:83)  - Configuration provider stopping
25 Jul 2016 16:14:38,856 INFO  [agent-shutdown-hook] (org.apache.flume.source.NetcatSource.stop:190)  - Source stopping
25 Jul 2016 16:14:39,357 INFO  [agent-shutdown-hook] (org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink.stop:74)  - Stopping Flume Metrics Sink

Make sure you are not running regular Flume Agent of same name! They will clash!

Using the Netcat

telnet localhost 44444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Slack, Apache Nifi,
Apache Flume, Netcat
OK^]
telnet> quit
Connection closed.

Resources:

2,361 Views
0 Kudos
Comments
New Contributor

I also have this requirement, can you give me a complete configuration? Thank you

Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
2 of 2
Last update:
‎08-17-2019 11:06 AM
Updated by:
 
Contributors
Top Kudoed Authors