Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

NIFI - Flume HDFS sink not working

NIFI - Flume HDFS sink not working

Hi All,

I am using HDF 3.4.0, I have designed NIFI flow to read events using NETCAT and dump into HDFS .

1. I'm using FlumeSource and FlumeSink processors in NIFI. When I execute the flow, events are being read by source but the sink is not able to write event into HDFS. Below is error in nifi-app.log


109744-flumesink-error2.png


Here is source and sink configurations

Source

109774-netcat-source.jpg


Sink

109775-hdfs-sink.jpg

2. Also nifi-app.log file shows multiple start and stop for NETCAT source can someone suggest on cause ? Ideally it should be once.


2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Source stopping
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Source starting
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/host:44444]
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Source stopping
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Source starting
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/host:44444]
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Source stopping
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Source starting
2019-07-05 17:11:11,341 INFO [Timer-Driven Process Thread-4] org.apache.flume.source.NetcatSource Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/host:44444]




2 REPLIES 2

Re: NIFI - Flume HDFS sink not working

Community Manager

The above question was originally posted in the Community Help track. On Sat Jul 6 17:30 UTC 2019, a member of the HCC moderation staff moved it to the Data Ingestion & Streaming track. The Community Help Track is intended for questions about using the HCC site itself, not technical questions about HDF, HFDS, Nifi or using the FlumeSource or FlumeSink processors.

Bill Brooks, Community Manager
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Highlighted

Re: NIFI - Flume HDFS sink not working

Can someone please advice on mentioned issue, I just can't see data in HDFS

FileSystem's output stream doesn't support getNumCurrentReplicas; --HDFS-826 not available; fsOut=org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer; err=java.lang.NoSuchMethodException: org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.getNumCurrentReplicas()
Don't have an account?
Coming from Hortonworks? Activate your account here