Ranger logs are in gz format - is there any way of extracting in the Nifi flow?

robnew — Thu, 06 May 2021 11:20:26 GMT

I have identified that some of my Ranger logs are coming in in gz format (but without the extension). Is there a way of extracting the log data within the Nifi flow?

Re: Ranger logs are in gz format - is there any way of extracting in the Nifi flow?

MattWho — Thu, 06 May 2021 20:37:04 GMT

@robnew

The "CompressContent" [1] processor can be used to decompress gz files.

My suggestion here since only some log files are compressed is to set up a flow that passes these FlowFiles through an "IdentifyMimeType" [2] processor. This will write out a new mime.type attribute on the FlowFiles. Then use a "RouteOnAttribute" [3] processor to route FlowFiles with mime.type = application/gzip (each new dynamic property becomes a new outbound relationship) to that "CompressContent" processor and the "unmatched" relationship (which will have every other non gz file) on through your flow without passing through "CompressContent" processor.

[1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.CompressContent/index.html

[2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.IdentifyMimeType/index.html

[3] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache.nifi.processors.standard.RouteOnAttribute/index.html

If you found this helpful with your question, please take a moment to login and click accept on this solution.

Thanks,

Matt

Re: Ranger logs are in gz format - is there any way of extracting in the Nifi flow?

robnew — Fri, 07 May 2021 13:28:12 GMT

Thanks, I tried something just like this yesterday, and it would work, but these logs don't have a file extension like what would be needed, (.gz) but I think I have an issue somewhere else, so will carry on trying to fix the overall issue.

question Re: Ranger logs are in gz format - is there any way of extracting in the Nifi flow? in Support Questions

Ranger logs are in gz format - is there any way of extracting in the Nifi flow?

Re: Ranger logs are in gz format - is there any way of extracting in the Nifi flow?

Re: Ranger logs are in gz format - is there any way of extracting in the Nifi flow?