question Re: nifi UnpackContent source file missing in Archives of Support Questions (Read Only)

nifi UnpackContent source file missing

camatulli — Fri, 31 Mar 2017 23:44:36 GMT

I am trying to utilize the UnpackContent processor being fed from a "FetchHDFS", but I am having an issue where the source path of the zip file, prior to being unpacked, is being dropped after being unpacked. Is there a way to either retain the path through the processor or another method I can ensure that path gets added as an attribute to the unpacked flow file?

Re: nifi UnpackContent source file missing

bbende — Tue, 04 Apr 2017 21:39:10 GMT

Can you elaborate more on where you see the source path and where it is getting dropped?

Going into FetchHDFS there should be a flow file with the content being a path to fetch like /data/foo.zip, after FetchHDFS it wrote the content of foo.zip to the flow file content and the filename attribute of the flow file should be foo.zip, then it goes to UnpackContent which produced multiple child flow files that were unpacked and each one should have segment.original.filename with foo.zip.

Are you asking to retain the original HDFS path that went into FetchHDFS?

Re: nifi UnpackContent source file missing

camatulli — Tue, 04 Apr 2017 21:49:12 GMT

From the UnpackContent processor on, the original file and path seem to be lost.

I see the path in queue going into the UnpackContent Processor, but after, when the new flow files are generated by the unpackcontent processor, the source file is not in the attributes anymore.

Re: nifi UnpackContent source file missing

bbende — Tue, 04 Apr 2017 22:05:14 GMT

Ok what are the exact attribute names that you see in the queue going into UnpackContent that are being lost?

Re: nifi UnpackContent source file missing

camatulli — Tue, 04 Apr 2017 22:29:20 GMT

segment.original.filename is gone and filename and path are with the new values after extract. i am looking for the file name and path that came from the fetch

Re: nifi UnpackContent source file missing

camatulli — Sun, 18 Aug 2019 08:41:34 GMT

Before unpack, i have the path of the file in HDFS, and the actual file name in HDFS

After unpack, the "segment.original.filename" contains the filename without extension, and no reference to the source path anymore.

My main issue is i need that path when feeding to spark to create the relationships.

Re: nifi UnpackContent source file missing

bbende — Thu, 06 Apr 2017 23:34:10 GMT

Thanks for uploading the screenshots. I can see in the code that segment.original.filename is specifically removing the extension and this appears to have been like this since the initial code for NiFi was open-sourced, so I'm not sure if this is considered a bug or really a preference. The path attribute is being updated to reflect the path within the archive, although I believe there could be a bug here, but I believe it makes sense since the path of the children is not necessarily the path of the original flow file.

In the short-term, I think the easiest thing to do is stick an UpdateAttribute processor right before UnpackContent and add two properties that copy the filename and path to new attributes like this:

archive.filename = ${filename}

archive.path = ${path}

The flow files for the unpacked files should retain these attributes.

Re: nifi UnpackContent source file missing

camatulli — Fri, 07 Apr 2017 00:27:20 GMT

Thanks. This ended up being a good workaround to the attributes dropping, and gave me a couple ideas on how to extend the information i need.