Using Apache Flume Sources and Sinks with Apache NiFi 0.70
It's very easy to integrate Apache Flume with Apache NiFi, either as a source or destination (sink).
An example of using ExecuteFlumeSource and ExecuteFlumeSink in Apache NiFi. We use a Flume source to read Netcat and send to Slack. So you can send messages from Linux to a Slack chat. For the Flume sink to HDFS, we read data from a web feed. We can also connect our Flume Source to Flume Sink.
Configuring a Flume Source
If you have configured Apache Flume configuration files, this is the same thing.
Data Received in Slack (Join and See Messages Live)
Checking Provenance Event
Configure The ExecuteFlumeSink to Store to HDFS
-rw-r--r-- 3 flume hdfs 28 2016-07-25 16:14 /flume/events/16-07-25/1610/00/events-.1469463242392
To monitor what's going on in the Flume side of the process: tail -f /var/log/flume/flume-a1.log
25 Jul 2016 16:14:38,854 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177) - Shutdown Metric for type: SINK, name: k1. sink.batch.empty == 302
25 Jul 2016 16:14:38,854 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177) - Shutdown Metric for type: SINK, name: k1. sink.batch.underflow == 1
25 Jul 2016 16:14:38,855 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177) - Shutdown Metric for type: SINK, name: k1. sink.connection.closed.count == 1
25 Jul 2016 16:14:38,855 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177) - Shutdown Metric for type: SINK, name: k1. sink.connection.creation.count == 1
25 Jul 2016 16:14:38,855 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177) - Shutdown Metric for type: SINK, name: k1. sink.connection.failed.count == 0
25 Jul 2016 16:14:38,855 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177) - Shutdown Metric for type: SINK, name: k1. sink.event.drain.attempt == 1
25 Jul 2016 16:14:38,855 INFO [agent-shutdown-hook] (org.apache.flume.instrumentation.MonitoredCounterGroup.stop:177) - Shutdown Metric for type: SINK, name: k1. sink.event.drain.sucess == 1
25 Jul 2016 16:14:38,856 INFO [agent-shutdown-hook] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.stop:83) - Configuration provider stopping
25 Jul 2016 16:14:38,856 INFO [agent-shutdown-hook] (org.apache.flume.source.NetcatSource.stop:190) - Source stopping
25 Jul 2016 16:14:39,357 INFO [agent-shutdown-hook] (org.apache.hadoop.metrics2.sink.flume.FlumeTimelineMetricsSink.stop:74) - Stopping Flume Metrics Sink
Make sure you are not running regular Flume Agent of same name! They will clash!
Using the Netcat
telnet localhost 44444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Slack, Apache Nifi,
Apache Flume, Netcat
OK^]
telnet> quit
Connection closed.
Resources:
- https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.flume.ExecuteFlumeSource/additionalDetails.html
- https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.flume.ExecuteFlumeSink/additionalDetails.html
- https://community.hortonworks.com/questions/14713/flume-tutorials.html
