About elloyd

elloyd · ‎03-07-2017

Well, I have altered the text in the filename attribute so that there isn't any errors but still getting the same error in PutHDFS. It does look slightly different now (see screenshot) it is producing the correct filename when I look at the list queue and the output of the filename in the error (with the odd exception there is a dot before the first letter of the filename... not sure if thats the issue) A consideration was possibly that the I was creating the directory structure dynamically in PutHDFS (see original screenshot) but I changed it to a non-dynamic directory specification and it still produced the same error. Its interesting to me that the error is produced on the PutHDFS processor yet its complaining about the filename that is dynamically created in UpdateAttribute. I have attached more screenshots. I appreciate any help.

elloyd · ‎03-07-2017

Great catch. I fixed the incorrect quote character but still getting the same error on PutHDFS

elloyd · ‎03-06-2017

I am attempting to use the UpdateAttribute processor to update the filename of my incoming files dynamically according to date. I am getting this error on the PutHDFS processor: Relative path in absolute URI: (then it has the code Im using to create the relative path). I have uploaded screenshots of both my UpdateAttribute and PutHDFS processors config page as well as a screenshot of the error

elloyd · ‎03-06-2017

Once again, thanks Matt! I think I'm finally starting to wrap my brain around how to adjust parameters to alter file flow.

elloyd · ‎03-06-2017

I have been hunting online for awhile trying to find how one would tell Nifi to format the HDFS output with one event per line instead of one event immediately after the next. I am trying to set up a virtual index through HUNK and its registering my entire output file as one event. Thanks in advance.

elloyd · ‎03-06-2017

Also, after viewing your answer, Im wondering if processing 24 hours worth of data and having it stored on the JVM heap memory would be too much. Probably. This is unfortunate. When we were using Flume, it would create a .tmp file that would be constantly gathering the data into it rather than storing it in memory so you could make them for as large as your wanted. This is not an appealling part of Nifi.

elloyd · ‎03-06-2017

So is it correct to say that a bin and a bundle are the same thing?

elloyd · ‎03-06-2017

Hello, I am trying to use the MergeContent processor in Nifi to deliver a single hdfs file for a specified length of time (no maximum number of entries) rather than the options presented in the MergeContent processor: Number of Entries, Group Size. I see there is something called Max Bin Age, but I am under the impression that Bins are different than Bundles and it doesn't work as I'd hoped when I use Max Bin Size to do what I am trying to do. To be clear, I am trying to deliver data into separate directories that are divided according to /year/month/day with one file in each day directory. I still have yet to attempt to create the directory structure. Thanks for any help.

elloyd · ‎03-06-2017

We have a setup of GetKafka processor to an endpoint of PutHDFS. I am still working on it to get it to function correctly. I notice when I run it, it will pull data from when it was last run instead of pulling the immediate data from the moment I started it (in the topic). Is this "as designed"? Its possible this is and I'm just unaware ... I expected when I ran it to open the HDFS file and see data that is being pulled from the current time. Is there a way to set it so it only grabs data in the topic that is from the present moment and not all the previous data? Any insight into this would be helpful.

elloyd · ‎03-02-2017

Perfect. Thank you. Still learning how to rethink data ingestion from Flume to Nifi.

Online	Offline
Last Visited	‎03-14-2018 05:14 PM

Member Since	‎01-05-2017 02:25 PM
Last Visited	‎03-14-2018 05:14 PM
Posts	153
Kudos received	10

Cloudera Community

Re: TailFile cannot find directory/file which exis...

Re: Unusual data placement on file rollover in Nif...

Re: Relative path in absolute URI error

Re: Relative path in absolute URI error

Relative path in absolute URI error

Re: Output to HDFS separate events one per line

Output to HDFS separate events one per line

Re: Merge Fileflow files based on time rather than...

Re: Merge Fileflow files based on time rather than...

Merge Fileflow files based on time rather than siz...

Understanding how Nifi retrieves from Kakfa

Re: Block size in PutHDFS in Nifi preventing HDFS ...