Ranger logs are in gz format - is there any way of extracting in the Nifi flow?

[#1]

I have identified that some of my Ranger logs are coming in in gz format (but without the extension). Is there a way of extracting the log data within the Nifi flow?

[#2 · Accepted solution]

@robnew 

The "CompressContent" [1] processor can be used to decompress gz files.

My suggestion here since only some log files are compressed is to set up a flow that passes these FlowFiles through an "IdentifyMimeType" [2] processor.  This will write out a new mime.type attribute on the FlowFiles.  Then use a "RouteOnAttribute" [3] processor to route FlowFiles with mime.type = application/gzip (each new dynamic property becomes a new outbound relationship) to that "CompressContent" processor and the "unmatched" relationship (which will have every other non gz file) on through your flow without passing through "CompressContent" processor.   

[1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...

[2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...

[3] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...

If you found this helpful with your question, please take a moment to login and click accept on this solution.

Thanks,

Matt

[#3]

Thanks, I tried something just like this yesterday, and it would work, but these logs don't have a file extension like what would be needed, (.gz) but I think I have an issue somewhere else, so will carry on trying to fix the overall issue.