Because I don't think it works the way my supervisor think it works. We're taking in a series of about 8 csv files from an FTP and these files are rather small (under 1MB). He's (rightfully) concerned that cluster size on HDFS is going to be wasted. So he wants to use the Merge Content processor to resolve this. He seems to believe that the Merge Content processor will 'collate' files with the same name, making a bigger single file. To clarify: The way he wants it to work is if today's "sales_report.csv" comes in and there's already a "sales_report.csv" existing in the directory, he wants the new data from today's "sales_report.csv" to be added as new rows to the existing file. I hope that makes sense. Instead, I'm getting very different results. I have the flow set up so that it picks the files up from the FTP, creates a directory on HDFS based on the folder, and then a subfolder based on the year. When I leave the MC processor out, this all works perfectly. When I put the MC processor in, I get three files - one of them has its original name and two of them have a long string of random characters. We're using the default settings for the Merge Content processor. Sorry, I've written a bit of an essay. But based on what I've described above it does it sound like the MC processor is what he's looking for? ... View more

In Nifi I have a processor that needs to place some parquet files into an HDFS directory. The idea is that if it fails, it should wait one second, then try again. If that fails, it will wait another second then try again. Finally if there is one more failure, it will email an admin. I configured it exactly as suggested in this article: https://kisstechdocs.wordpress.com/2015/01/15/creating-a-limited-failure-loop-in-nifi/ However, this will only retry the process - it won't wait before doing the next step. Is there a way to configure this so that steps 2 and 3 will wait one second before proceeding? ... View more

I have Nifi taking a number of json files from Kafka. I set up a "ConsumeKafka" to start the process. Now I need to read the json file's "creationDate" field, which is a normal UNIX timestamp (i.e., 2019-01-19T04:34:28.527722+00:00). It then needs to take part of that date and format it so it matches a corresponding HDFS directory, so the files can go into a directory that matches the date - i.e., the directories are named "01-19-2019", "01-20-2019", "01-21-2019", etc. I was thinking of setting up an "EvaluateJsonPath" processor with a property of "creationDate" and a value of "$.creationDate". Then there would be an "UpdateAttribute" processor with a property of "creationDate" and a value of "${creationDate('yyyy/MM/dd HH:mm:ss:SSS'Z'):format('MM-dd-yyyy')}" Finally there would be a "PutHDFS" processor with a directory of "/${creationDate}". I'm not sure about the expression (or if this is going to work the way I think it is). Especially because the directory format doesn't quite match the timestamp format. Maybe the directory names can change, but I'm going to assume that they can't. Any thoughts on how I can make this work? ... View more