About thierry_vernhet

thierry_vernhet · ‎06-22-2016

Hi, I've got more than one hundred files to tail and I would like to use patterns in the File to Tail property. For example : /donapp/*/p00/wi*/log/*.?.*log.2016-06-22 Do you think it's possible to improve this processor ? Or is there another way to do it ? Thanks

thierry_vernhet · ‎06-16-2016

Hi We'd like to get/ingest WMI windows events ... with NIFI. Is it possible ? Has anyone tried to do it ? Thanks

thierry_vernhet · ‎06-06-2016

Hi @Pierre Villard Which value in your test for property "Initial Start Position" ? Is the Nifi release important ? We use "0.5.1". Thanks

thierry_vernhet · ‎06-03-2016

@Pierre Villard Results are the same even with the new pattern "wls.log.*" and also with all the value of "initial start position". I think there's a problem. All I want is to receive (tail) the new records that are written in the wls.log file. And also when the log file rotates and keeps the same name "wls.log". This case seems very simple yet. But I can't achieve it with Nifi. So I think there's a little bug. Don't you think the same ? Please let me know Thanks

thierry_vernhet · ‎06-03-2016

No it's not what I did. I'm going to test again with Rolling Filename Pattern : wls.log.*.

thierry_vernhet · ‎06-03-2016

Hi @Pierre Villard and @Bryan Bende I also tested with rolling file pattern property = wls.log and initial start = beggining of file Test1 the same result Test2 the same resut Test3 .... the same result than without file pattern property (see my previous comment to Bryan) Nifi Flow Data Provenance shows : 06/03/2016 08:44:24.759 CEST RECEIVE 389968cd-a1b6-497d-9e0a-c0e93ffe90f6 4 bytes . But I want to complete my comment. The 4 bytes (of the 32) "789" are the differnece beetween the size of wls.log file (before its init with "cp empty.txt wls.log") minus the size of the new record. So if the size before init of wls.log is 10 MB (for example), the next new records won't be ingested as the new records size will not reach 10 MB. . I think there's a bug... So do you think it's possible to contact Hortonworks support to improve Nifi product in this case ? . Thanks for your feedback.

thierry_vernhet · ‎06-03-2016

Hi @Bryan Bende So with the second echo "record0123456789" >> ... (I've made a mistake in my original question, it's an echo instead of a cat command)... the results are Test1 the same Test2 the same Test3 .... not the same result cp empty.txt wls.log -rw-r--r-- 1 itve6530 userdsi 0 Jun 3 08:44 wls.log echo "record33333333333xxxx0123456789" >> wls.log Nifi Flow Data Provenance shows : 06/03/2016 08:44:24.759 CEST RECEIVE 389968cd-a1b6-497d-9e0a-c0e93ffe90f6 4 bytes And the 4 bytes are "789" the last of the echo command which contains 32 bytes ... funny, isn't it ? So I wrote another different record echo "record33333333333xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx0123456789" >> wls.log Nifi Flow Data Provenance shows :06/03/2016 08:45:32.218 CEST RECEIVE fb6d2013-03ab-4576-989e-02e2bd9981f4 64 bytes with all the 64 bytes of the echo command... in progress ! So I wrote another different record echo "record44" >> wls.log OK in Nifi Flow Data Provenance : 06/03/2016 08:57:58.926 CEST RECEIVE e253f961-3a8a-43a2-97b1-25d54526a894 9 bytes echo "record55" >> wls.log KO in Nifi Flow Data Provenance (same size than previous record) echo "record5566" >> wls.log OK in Nifi Flow Data Provenance (differnent siez). Now it shows the last and the one before the last. 06/03/2016 08:58:28.013 CEST RECEIVE 3c646f8c-84c8-4353-af66-e7e06fc76560 11 bytes 06/03/2016 08:58:10.944 CEST RECEIVE e158b0ce-a856-44f9-b7cf-4769ce8b0f69 9 bytes Very interesting ...

thierry_vernhet · ‎06-03-2016

Hi @Pierre Villard I've tried this in a similar situation (see the question posted Nifi : How avoid ingesting an old rolling file in TailFileProcessor ?) And I've got a problem with the odl rolled file that is always ingested when a new record is written into the new file. But there was a little difference. So I'll test in this situation and give you my feedback.

thierry_vernhet · ‎06-03-2016

Hi @Bryan Bende I' gonna test this and give you a feedback. Please could you also have a look to Pierre's answer ?

thierry_vernhet · ‎06-02-2016

Hi, I've got another problem with the logfiles linux weblogic. I've tested several cases (see below) . And I don't know how ingest new data without stop the processor, cancel the state and restart Nifi agent. First question : Is there another way to do ? Second question : Is it possible to script the stop processor and the clear state ? Thanks The TailFile processor properties : File to Tail /home/wls.log Rolling Filename Pattern State File No value set Initial Start Position Beginning of File File Location Local Test 1 : cat ‘record01’ >> wls.log The NiFi Flow Data Provenance is OK. The line ‘record01’ appears. mv wls.log wls.log.01 cat ‘record02’ >> wls.log The NiFi Flow Data Provenance is KO. The line ‘record02’ doesn’t appear. The line ‘record02’ recovers only when I stop the processor TailFile, clear state and restart Nifi. Test 2 : cat ‘record01’ >> wls.log The NiFi Flow Data Provenance is OK. The line ‘record01’ appears. cp wls.log wls.log.01 rm wls.log cat ‘record02’ >> wls.log The NiFi Flow Data Provenance is KO. The line ‘record02’ doesn’t appear. The line ‘record02’ appears only when I stop the processor TailFile, clear state and restart Nifi. Test 3 : cat ‘record01’ >> wls.log The NiFi Flow Data Provenance is OK. The line ‘record01’ appears. touch empty.txt cp empty.txt wls.log (inode ws.log doesn’t change) cat ‘record02’ >> wls.log The NiFi Flow Data Provenance is KO. The line ‘record02’ doesn’t appear. The line ‘record02’ appears only when I stop the processor TailFile, clear state and restart Nifi.

Online	Offline
Last Visited	‎05-23-2019 01:10 PM

Member Since	‎05-24-2016 01:03 PM
Last Visited	‎05-23-2019 01:10 PM
Posts	45
Kudos received	4

Cloudera Community

Improve Tailfile Processor for use patterns in the...

Is there an API for WMI Windows ?

Re: Tailfile with rotate log files

Re: Tailfile with rotate log files

Re: Tailfile with rotate log files

Re: Tailfile with rotate log files

Re: Tailfile with rotate log files

Re: Tailfile with rotate log files

Re: Tailfile with rotate log files

Tailfile with rotate log files