Support Questions

Find answers, ask questions, and share your expertise

Ranger logs are in gz format - is there any way of extracting in the Nifi flow?

avatar
Contributor

I have identified that some of my Ranger logs are coming in in gz format (but without the extension). Is there a way of extracting the log data within the Nifi flow?

1 ACCEPTED SOLUTION

avatar
Master Mentor

@robnew 

 

The "CompressContent" [1] processor can be used to decompress gz files.

My suggestion here since only some log files are compressed is to set up a flow that passes these FlowFiles through an "IdentifyMimeType" [2] processor.  This will write out a new mime.type attribute on the FlowFiles.  Then use a "RouteOnAttribute" [3] processor to route FlowFiles with mime.type = application/gzip (each new dynamic property becomes a new outbound relationship) to that "CompressContent" processor and the "unmatched" relationship (which will have every other non gz file) on through your flow without passing through "CompressContent" processor.   

[1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...

[2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...

[3] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...


If you found this helpful with your question, please take a moment to login and click accept on this solution.

Thanks,

Matt

View solution in original post

2 REPLIES 2

avatar
Master Mentor

@robnew 

 

The "CompressContent" [1] processor can be used to decompress gz files.

My suggestion here since only some log files are compressed is to set up a flow that passes these FlowFiles through an "IdentifyMimeType" [2] processor.  This will write out a new mime.type attribute on the FlowFiles.  Then use a "RouteOnAttribute" [3] processor to route FlowFiles with mime.type = application/gzip (each new dynamic property becomes a new outbound relationship) to that "CompressContent" processor and the "unmatched" relationship (which will have every other non gz file) on through your flow without passing through "CompressContent" processor.   

[1] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...

[2] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...

[3] http://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.13.2/org.apache...


If you found this helpful with your question, please take a moment to login and click accept on this solution.

Thanks,

Matt

avatar
Contributor

Thanks, I tried something just like this yesterday, and it would work, but these logs don't have a file extension like what would be needed, (.gz) but I think I have an issue somewhere else, so will carry on trying to fix the overall issue.