We are experiencing duplicate data when our cluster's primary node switches over. Two nodes, m1 and m3, both TailFile processors in them tailing a log but only set to tail the Primary Node. When the Primary Node switches (say from m1 to m3), one TailFile processor stops tailing and the other picks it up (I am assuming it is picking it up at the beginning of the file according to the settings in the TailFile prcoessor) which causes data duplication. I saw there was an option to have it start at the current time instead of the Beginning of File. Would this be a reasonable fix to ensure when the Primary Node switches we dont get duplicate data? We've always had this set at Beginning of File and didn't seem to have this problem before so I am hesitant to change it without a response as to if I undestand this functionality in this situation correctly. ... View more

I am getting an error with my TailFile processor saying that it cannot ingest cause the directory doesn't exist (and it does, believe me.) Interestingly enough, I have in /testData/versions called dzl.log which it finds and doesnt throw an error. So its going into /testData/versions and finding dzl.log and saying "Ok, that file exists" but then its telling me that the directory /testData/versions does not exist. Anybody have any idea what Im missing here? Permissions for /testData = 777, root root Permissions for /testData/versions = 777, root root I have tried changing the group and owner to nifi/nifi and still get the same problem (besides I have another production directory that is 755, root/root that is ingesting fine) For some reason Nifi refuses to acknowledge this folder. Any ideas? ... View more