How to merge multiple HDFS files using Nifi Processor

s198 — Fri, 29 Mar 2024 05:51:59 GMT

We have a requirement to merge files in a specific HDFS path. The merged file should also be created in the same HDFS path but with a different name. Could you please suggest how we can achieve this using the Nifi Processor?

Re: How to merge multiple HDFS files using Nifi Processor

MattWho — Fri, 29 Mar 2024 13:51:35 GMT

@s198 NiFi has no ability to merge files remotely. NiFi would need to consume all the files (ListHDFS --> FetchHDFS), then merge the content of those FlowFiles (MergeContent or MergeRecord), then use UpdateAttribute to set desired filename on merged file, and finally write the merged file back to HDFS using PutHDFS processor.

If you are using a NiFi Cluster, you would need to do all this merging on one node of the cluster, NiFi nodes can only execute against the FlowFiles present on that one specific node.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: How to merge multiple HDFS files using Nifi Processor

s198 — Mon, 01 Apr 2024 09:44:00 GMT

Thanks a lot @MattWho

question Re: How to merge multiple HDFS files using Nifi Processor in Support Questions

How to merge multiple HDFS files using Nifi Processor

Re: How to merge multiple HDFS files using Nifi Processor

Re: How to merge multiple HDFS files using Nifi Processor