question Decompress .json.gz inside folder in NIFI in Support Questions

Decompress .json.gz inside folder in NIFI

nada — Thu, 09 Jun 2022 02:39:06 GMT

I receive a response from InvokeHTTP as a folder->another_folder->file.json.gz and I want to do some operations on the json file using SplitJson processor, so, I need to decompress the file.json.gz

I tried to UpdateAttribute to rename the parent folder to be folder.zip then UnpackContent but it's not supporting .gz format

Then, I tried instead of UnpackContent, the ExecuteStreamCommand with unzip command with a plan to use PutFile then ListFile to get the file.json.gz and pass to SplitJson but I got this error in ExecuteStreamCommand step

Failed to write flow file to stdin due to Broken pipe: java.io.IOException: Broken pipe

What's the solution for this? I tried to set Connection Timeout propriety to 120s but still the same error, What is the best way to get file.json.gz, decompress it and pass it to SplitJson processor?

Re: Decompress .json.gz inside folder in NIFI

araujo — Thu, 09 Jun 2022 03:16:06 GMT

@nada ,

Could you share a sample of the InvokeHTTP response and also the flow that you currently have?

Cheers,

André

Re: Decompress .json.gz inside folder in NIFI

nada — Thu, 09 Jun 2022 10:24:31 GMT

the InvokeHTTP response:

from the queue list:

after I download:

Now I have a flow file in the nonzero status queue from ExecuteStreamCommand as follows

the whole cycle:

Re: Decompress .json.gz inside folder in NIFI

MattWho — Thu, 09 Jun 2022 17:33:33 GMT

@nada

You can use the CompressContent processor to decompress gzip files.
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.16.2/org.apache.nifi.processors.standard.CompressContent/index.html

Set "Mode" to "Decompress", "compression format" to "gzip", and "Update Filename" to "True".

If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post.

Thank you,

Matt

Re: Decompress .json.gz inside folder in NIFI

nada — Mon, 13 Jun 2022 01:43:08 GMT

@MattWho

Hi Matt,

my incoming flowfile is basically a folder that contains another folder that contains a .json.jz file

(folder->another_folder->file.json.gz), CompressContent processor is meant for .gz incoming file..

How to recursively decompress so it decompresses the .json.jz that is part of the subfolder? or how to unpack the first folder without touching subfolders/files?

Re: Decompress .json.gz inside folder in NIFI

araujo — Wed, 22 Jun 2022 23:13:29 GMT

@nada ,

Please check this solution: https://community.cloudera.com/t5/Community-Articles/Decompressing-nested-ZIP-files-in-NiFi/ta-p/346169

Cheers,

André