I have established a NiFi flow that receives TCP messages at an IOT device (using ListenTCP) and forwards them to a RPG on cloud which eventually pushes it into Cassandra. I have been monitoring the 5 minute statistics to obtain the throughput from the IOT device as well as the rate at which it gets inserted into Cassandra. I have observed that for larger amounts of data, the stats steadily rise till a point and then start fluctuating even though the entire data has not been transferred yet.
1> Can someone please explain this behaviour to me? Is there a something I need to correct/modify?
2> Is there a better alternative to monitor the flow to get the final throughput? I'm interested in obtaining the total time it took for data to reach and leave the RPG and the PutCassandra processors respectively.
Thanks in advance,