About maxwell_evers

maxwell_evers · ‎11-15-2017

I'm trying to come up with a means for dynamic attribute creation. Say I've got a flat Json structured string of name value pairs, in a cassandra db as a text field. {"A":"1", "B":"2"} Is there a way to parse this out to a result of an A attribute with value 1 and B attribute with value 2 without explicitly creating a A attribute then polling the jsonpath? I haven't had any luck in finding a way to dynamically set the lefthand side of an 'update attribute' and wondering if I'm just overlooking something.

maxwell_evers · ‎10-16-2017

I had the same question a few months later after forgetting what I did last time and lo and behold my google search results in my own question on hwx, so thanks again.

maxwell_evers · ‎07-13-2017

We had an issue with our nifi instance where someone left something running that they shouldn't have in the morning the server is a bit of a mess. While we're in the process of sorting through the errors and logs, I'm thinking I'd like to start the nifi instance with all the processors in a stopped state. Is there a way to do this? I can't find anything with google searches or skimming the administrators docs.

maxwell_evers · ‎06-14-2017

Thank you, this was very helpful. For my specific purposes and my specific scenario it would seem using the difference in time should be adequate. I will keep in mind your warnings regarding time in queue, which was also the reason I didn't want to use lineage duration either, as we have events sitting in front of a control rate.

maxwell_evers · ‎06-14-2017

When looking in the lineage of a flowfile in provenance, there is a Time field for each node in the lineage Looks of the format... Time 06/13/2017 14:18:47.678 EDT When is this field set with regards to the processors task? I ask because I am trying to track the time each processor is taking, and the event duration is not always set, so I am simply taking the timestamps of each processor and taking the difference between them. To date I've assumed the Time is taken as the flowfile 'enters' the processor so diffing time between node 2 and node 1 gives the time taken for node 1 to handle the flowfile, but I'd was wondering if someone can confirm this.

maxwell_evers · ‎05-25-2017

As mentioned above on Matt's comment, yes, the one left behind always has the latest timestamp.

maxwell_evers · ‎05-25-2017

Yes, the one that is left behind is the latest generated file. The last file gets picked up on the second run. My use case was looking for a listing of all the files in an hdfs directory at a given moment. GetHDFS provides that functionality with the inefficient overhead of bringing the actual files into nifi. I was hoping to just get the list of files with listHDFS. I'm thinking I might look into ExecuteStreamCommand to generate the list with a hdfs dfs -ls and parse that list.

maxwell_evers · ‎05-25-2017

I am running a ListHDFS processor pointing to a directory on hdfs on a timer driven schedule set to execute once per hour. After making sure the state is clear on the processor, I run it and see that it creates a flow file for all but 1 file in the directory. There are 5 files in the directory, and only 4 flowfiles are created. If I add more files and clear the state and attempt to run again, the pattern repeats, always one less flowfile is create, so one file is missed. It is not the same file that is missed with each run. Why is the processor missing 1 file each time? Is this by design? This is in HDF 2.1.0.1 and Apache NiFi - Version 1.1.0.2.1.0.1-1

Online	Offline
Last Visited	‎11-15-2017 09:56 PM

Member Since	‎12-21-2016 07:49 PM
Last Visited	‎11-15-2017 09:56 PM
Posts	16

Cloudera Community

Nifi: Dynamic Attribute Creation

Re: Start NIFI in a stopped state

Start NIFI in a stopped state

Re: Nifi Provenance Timestamp

Nifi Provenance Timestamp

Re: Nifi ListHDFS missing 1 file per poll.

Re: Nifi ListHDFS missing 1 file per poll.

Nifi ListHDFS missing 1 file per poll.