question Re: MiNifi/State/Local Huge Disk Size Consumption in Support Questions

MiNifi/State/Local Huge Disk Size Consumption

Arash — Tue, 07 Sep 2021 11:16:08 GMT

I have a MiNiFi agent that tails from a file and sends it to a NiFi Cluster. After a while, The state directory takes 27 Gigabytes of the disk and the content_repository only takes 34MB of the disk. Any idea about this problem?

Re: MiNifi/State/Local Huge Disk Size Consumption

Arash — Sat, 11 Sep 2021 12:11:18 GMT

I found the below problem in the minifi-app.log:

2021-09-08 15:00:04,696 WARN [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.TailFile TailFile[id=1697d3af-4b0c-3a38-0000-000000000000] Unable to open file /log/MyLog/myLog_120_20210908_125952.log; will attempt to access file again after the configured Yield Duration has elapsed: java.nio.file.NoSuchFileException: /log/MyLog/myLog_120_20210908_125952.log:

In addition, I found that the line below is repeated regularly in the journal file of this directory: ./minifi-0.5.0/state/local/partition-0/

3877966761 file.205.filename \/log/MyLog/myLog_120_20210908_125952.log.tar.gz file.331.checksum 967031169 file.420.checksum

Re: MiNifi/State/Local Huge Disk Size Consumption

Wynner — Wed, 29 Dec 2021 17:31:44 GMT

@Arash

What exactly does the agent flow do? TailFile --> RemoteProcessGroup to NiFi?