Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi: How to check if large file has completely been written to directory without using Minimum File Age setting?

Solved Go to solution
Highlighted

Nifi: How to check if large file has completely been written to directory without using Minimum File Age setting?

New Contributor

Is the Minimum File Age setting on a GetFile processor the only way to check if large files have been completely written to its directory?

Instead of using a timer, is there a way to ask for the current filesize attribute after I use GetFile? I was trying to use UpdateAttribtue/RouteOnStrategy processors to repeatedly check the filesize attribute until it's been completed.

Thanks

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Nifi: How to check if large file has completely been written to directory without using Minimum File Age setting?

Super Guru

@K Hajjar

Sharing some thoughts you could try. You may have option to use execute process which would fetch size from file and store size to local disk. Run the loop again and check new size against size stored to local disk. If size has not changed you can routeonstragegy thereafter.

6 REPLIES 6

Re: Nifi: How to check if large file has completely been written to directory without using Minimum File Age setting?

Super Guru

@K Hajjar

Sharing some thoughts you could try. You may have option to use execute process which would fetch size from file and store size to local disk. Run the loop again and check new size against size stored to local disk. If size has not changed you can routeonstragegy thereafter.

Re: Nifi: How to check if large file has completely been written to directory without using Minimum File Age setting?

New Contributor

Awesome, exactly what i'm looking for thank you -- i'll try that

Re: Nifi: How to check if large file has completely been written to directory without using Minimum File Age setting?

New Contributor
@Sunile Manjee

So the "command" property I would configure to fetch the current filesize of the file using unix commands?

then it would be written and stored to local disk?

Reading the documentation, it seems like it's used to provide path of an executable, but I don't have one to specify

Re: Nifi: How to check if large file has completely been written to directory without using Minimum File Age setting?

Super Guru

I would use the executeprocess processor. You would execute for example a python or shell script to do the logic I shared above. make sense?

Re: Nifi: How to check if large file has completely been written to directory without using Minimum File Age setting?

New Contributor

got it! thanks

Re: Nifi: How to check if large file has completely been written to directory without using Minimum File Age setting?

A common way to do this is to have the file written to ".filename" first and renamed to "filename" when done. This is why the GetFile processor File Filter property defaults to:

 [^\\.]\.*  

That regular expression says any filename that doesn't start with a period.

I realize you may not have control over how the files are being written to the directory though, so this may not be an option if you can't control that.

Don't have an account?
Coming from Hortonworks? Activate your account here