Member since
04-11-2016
471
Posts
325
Kudos Received
118
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2077 | 03-09-2018 05:31 PM | |
2649 | 03-07-2018 09:45 AM | |
2538 | 03-07-2018 09:31 AM | |
4406 | 03-03-2018 01:37 PM | |
2476 | 10-17-2017 02:15 PM |
06-07-2016
11:19 AM
Hi @Roberto Sancho. Any relevant information in ressources manager logs?
... View more
06-07-2016
11:12 AM
- I don't understand what is the problem with Windows. Do your files contain timestamps in the filename? If yes, you can use regular expression to filter on the filename based on the date.
- If you want all files having an extension but with the extension not being equal to .xls or .xlsx, you can use the following regular expression: ^.*\.(?!xls$|xlsx$)[^.]+$ I would recommend you testing your regular expressions on this website : http://regexr.com/ This site provides a "explain tab" giving information regarding the regular expression and how it works.
... View more
06-07-2016
09:20 AM
1 Kudo
Hi @Iyappan Gopalakrishnan, If your files are located on local file system you may use ListFile processor. This processor has properties accepting the use of expression language. Using date functions of expression language [1], you can filter the files you list based on the timestamp (if the filenames contain timestamp). Regarding your second question, you can use the File filter property in ListFile processor which accepts a regular expression. It allows you to exclude some extensions if you don't want to list .xls and .xlsx files. [1] https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#dates
... View more
06-07-2016
07:43 AM
2 Kudos
Hi @Rajib Mandal, There is no issue at loading large volumes of data using Apache NiFi. Of course, depending of the characteristics of your setup and of your input files, the performance may change. From what you described, a simple flow with List/FetchFile processors and PutHDFS processor should perform as expected. Hope this helps.
... View more
06-06-2016
09:55 PM
1 Kudo
Hi @Sean Byrne, If you want to update the filename before updating the file in HDFS, I would suggest using UpdateAttribute to update attribute 'filename' using expression language. If you want to insert the date inside the flow file content, you could use ReplaceText processor which allows you to leverage the expression language and insert the date where you want and with the format you want. Hope this helps.
... View more
06-06-2016
01:20 PM
I'm using the same as you: Beginning of file. I think the version should not matter on this processor: it didn't change since 5 months. However if you are able to try with the PR mentioned above, it may be worth it.
... View more
06-06-2016
09:27 AM
Hi @Thierry Vernhet, I've tried to reproduce the issue but when I set the rolling filename pattern, it works as expected. Here are the steps I did: echo "test" >> /tmp/test.log echo "test" >> /tmp/test.log mv /tmp/test.log /tmp/test.log.01 echo "test" >> /tmp/test.log With the rolling filename pattern set to test.log.* The JIRA I raised is regarding the case the property is not set and I submitted a PR for that. But in case the property is not set, at this point I am not able to reproduce the error. Are there specific properties regarding your environment? I could suggest you to turn logging to DEBUG level and check if you have some interesting messages...
... View more
06-06-2016
09:11 AM
Hi @Dave South, I'm not familiar with Stomp but, as you suggest, I'd try to leverage existing AMQP processors. If it requires specific developments, your best option is probably to fork AMQP processors and develop your own processors. If you choose this path, you are more than welcome to contribute your code into NiFi by raising a JIRA (https://issues.apache.org/jira/browse/NIFI) and submitting your PR.
... View more
06-06-2016
09:07 AM
1 Kudo
Hi @Suresh K, I'd recommend you using List/FetchSFTP processors to get data from your SFTP server and then use PutHDFS to send your data to your HDFS cluster.
... View more
06-03-2016
09:17 AM
Not sure to understand. If you do the following test: cat ‘record01’ >> /home/wls.log The NiFi Flow Data Provenance is OK. The line ‘record01’ appears. mv /home/wls.log /home/wls.log.01 cat ‘record02’ >> /home/wls.log I believe the parameters should be the following: File to Tail /home/wls.log Rolling Filename Pattern : wls.log.01 Is that what you did? The observation you made are valid because the processor keeps in its state the position of the last read bytes in the tailed file. This is why if the roll over is not detected by the processor, it won't pick up new data or it will but partially (from the last known position).
... View more