Member since
01-05-2017
153
Posts
10
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3235 | 02-20-2018 07:40 PM | |
2162 | 05-04-2017 06:46 PM |
03-14-2018
05:14 PM
ok changed to 2 secs. Will keep an eye on it to monitor for future data duplication. Going to accept this as answer for now. Thank you.
... View more
03-14-2018
04:20 PM
Hi there @Matt Clarke do you have a suggestion for a value to switch run time to other than 0 secs or is that more of a choice dependent on the flow/system itself?
... View more
03-07-2018
06:40 PM
Okay yes this all makes sense. Do you think I would reduce the amount of data duplication by switching the property in the TailFile processor "Initial Start Position" from Beginning of File to Current Time would reduce the amount of duplication? Or would I possibly be facing data loss then?
... View more
03-07-2018
06:31 PM
The file is visible to both nodes. The state location is set to remote on the TailFile processors. My use case: 2 nodes in a cluster. Both have TailFiles processors on them with the setting for the Primary Node to be the one tailing. When started, one of the nodes is the Primary node (lets say m1) So now only m1's Tailfile processor is tailing the log file that is visible to both nodes. Suddenly, without warning, the Primary Node switches to m3. So now the Tailfile processor on m1 stops tailing the log file and the m3 Tailfile processor starts tailing it. But from where does the m3 Tailfile processor start tailing it? Seeing from the duplicate data it seems to indicate it starts at the Beginning of File rather than were the m1 Tialfile processor left off.
... View more
03-07-2018
06:15 PM
We are experiencing duplicate data when our cluster's primary node switches over. Two nodes, m1 and m3, both TailFile processors in them tailing a log but only set to tail the Primary Node. When the Primary Node switches (say from m1 to m3), one TailFile processor stops tailing and the other picks it up (I am assuming it is picking it up at the beginning of the file according to the settings in the TailFile prcoessor) which causes data duplication. I saw there was an option to have it start at the current time instead of the Beginning of File. Would this be a reasonable fix to ensure when the Primary Node switches we dont get duplicate data? We've always had this set at Beginning of File and didn't seem to have this problem before so I am hesitant to change it without a response as to if I undestand this functionality in this situation correctly.
... View more
Labels:
- Labels:
-
Apache NiFi
02-20-2018
08:11 PM
1 Kudo
I stand corrected. It seems if you are doing Tailfile in a cluster, the directories and files must exist on all hosts.
... View more
02-20-2018
07:40 PM
Nevermind, I figured it out. We have two hosts m1 and m3. m3 is currently our primary controller. The dummy directory I created on m1 instead of m3.
... View more
02-20-2018
07:35 PM
Also here is the error from the interface
... View more
02-20-2018
07:24 PM
I actually don't see the stack trace for this error in nifi-app.log. I do have debugging on the TailFile processor. I searched by ID. I suspect its because the processor never actually runs. This error appears before I hit play on it and does not appear even if I press play on it. It has the yellow exclamation point before it is run so I assume nifi has functionality that goes and searches for a folder once you hit ACCEPT on the processors configuration. I did not see any whitespace in my processor config. Here are some screenshots.
... View more
02-20-2018
06:46 PM
I am getting an error with my TailFile processor saying that it cannot ingest cause the directory doesn't exist (and it does, believe me.) Interestingly enough, I have in /testData/versions called dzl.log which it finds and doesnt throw an error. So its going into /testData/versions and finding dzl.log and saying "Ok, that file exists" but then its telling me that the directory /testData/versions does not exist. Anybody have any idea what Im missing here? Permissions for /testData = 777, root root Permissions for /testData/versions = 777, root root I have tried changing the group and owner to nifi/nifi and still get the same problem (besides I have another production directory that is 755, root/root that is ingesting fine) For some reason Nifi refuses to acknowledge this folder. Any ideas?
... View more
Labels:
- Labels:
-
Apache NiFi