NIFI Large Dataset

[#1]

@Matt Burgess

I have 1M+ records in a CSV file, and I have many of these files; where Data and Time are separate columns.

What i need to do is combine them into TimeStamp.

Is it efficient and / or possible to combine columns into a timestamp in the flowfiles via a record processor of some sort?

[#2]

Using UpdateRecord should be efficient.

[#3]

As mentioned by Bryan, you can use UpdateRecord processor, plus you can use distributed processing using RPG or multiple concurrent tasks to distribute the workload into multiple instances and process them parallelly for a better performance.

[#4]

That should not be a problem. I would make sure you have enough RAM and CPUs on your machine. SSD drives for NiFi repositories and enough nodes in your cluster.

Are these wide records? How big is each file? How many do you want to process at once? Where are you landing them?