PUTHDFS- Files ingested into HDFS but I don't see Files in HDFS

[#1]

IN NiFi I have a flow which fetches files from remote server and post data into HDFS. PUTHDFS  relationship was set to both success and Failure. So, the files which are loaded to fail exists in same processor. Could you please let me know how to view list of files failed other than from data provenance?

[#2 · Accepted solution]

Hello,

I'm not sure to understand your problem, but i'll try to answer to "Files ingested into HDFS but I don't see Files in HDFS".

When you process a flowfile with a processor, the processor will route the flowfile to relationship according with what it managed to do with the flowfile.

The PutHDFS processor have two relationship : success and failure. If the processor managed to put the flowfile to HDFS, the flowfile is routed to the relationship success and you can continue to process it. This unless you check automatically terminate the relationship success, then NiFi won't do anything more with the flowfile. Same for the failure relationship.

Here you automatically terminate both success and failure relationship.
For success relationship this is ok, because your file was put to HDFS.

But for failure relationship, the file is not put to HDFS and you don't handle the failure.

Just route the failure relationship to any other processor, you don't even need to use it at this point, you just want to see if flowfile are route to the failure relationship, and so if the queue corresponding to this relationship is filled.

You can also see the failure if a red error box appear in the top right corner of the processor, but previous method if to understand the concept of relationship routes.

If so, the flowfile are not put to HDFS, probably because of a bad configuration of the PutHDFS processor. 

In this case, i'll be happy to help you with this configuration (properties tab).