Member since
01-05-2017
153
Posts
10
Kudos Received
2
Solutions
03-14-2018
05:14 PM
ok changed to 2 secs. Will keep an eye on it to monitor for future data duplication. Going to accept this as answer for now. Thank you.
... View more
03-14-2018
04:20 PM
Hi there @Matt Clarke do you have a suggestion for a value to switch run time to other than 0 secs or is that more of a choice dependent on the flow/system itself?
... View more
03-07-2018
06:40 PM
Okay yes this all makes sense. Do you think I would reduce the amount of data duplication by switching the property in the TailFile processor "Initial Start Position" from Beginning of File to Current Time would reduce the amount of duplication? Or would I possibly be facing data loss then?
... View more
03-07-2018
06:31 PM
The file is visible to both nodes. The state location is set to remote on the TailFile processors. My use case: 2 nodes in a cluster. Both have TailFiles processors on them with the setting for the Primary Node to be the one tailing. When started, one of the nodes is the Primary node (lets say m1) So now only m1's Tailfile processor is tailing the log file that is visible to both nodes. Suddenly, without warning, the Primary Node switches to m3. So now the Tailfile processor on m1 stops tailing the log file and the m3 Tailfile processor starts tailing it. But from where does the m3 Tailfile processor start tailing it? Seeing from the duplicate data it seems to indicate it starts at the Beginning of File rather than were the m1 Tialfile processor left off.
... View more
03-07-2018
06:15 PM
We are experiencing duplicate data when our cluster's primary node switches over.
Two nodes, m1 and m3, both TailFile processors in them tailing a log but only set to tail the Primary Node. When the Primary Node switches (say from m1 to m3), one TailFile processor stops tailing and the other picks it up (I am assuming it is picking it up at the beginning of the file according to the settings in the TailFile prcoessor) which causes data duplication. I saw there was an option to have it start at the current time instead of the Beginning of File. Would this be a reasonable fix to ensure when the Primary Node switches we dont get duplicate data? We've always had this set at Beginning of File and didn't seem to have this problem before so I am hesitant to change it without a response as to if I undestand this functionality in this situation correctly.
... View more
Labels:
- Labels:
-
Apache NiFi