Hi, Im throwing multiple csv files on my hdfs every minute using logstash. I need to get the files from the past minute from the current time. Im using nifi in this process. For example right now is 11:30 AM, I need to get ONLY all the files that are saved 1 minute ago or 11:29AM. What is the best approach here using nifi? Thank you. ... View more

Hi, How can i aggregate my csv data in nifi? Can you please give me options. I have csv coming every hour. For example my csv data is: Name, Subject stud1, math stud2, english stud3, math stud4, math stud5, english stud6, science My needed output is: math, 3 english, 2 science, 1 Can i make this without using any execute scripts? Thank you. ... View more

Hi Guys, I have a log simulator, which is sending 10k events per second on my port 5043. I used the ListenTCP Processor to fetch the events but im having an error. 1. 2. So, i tried to lowered the Socket Buffer Size. The first image error was gone, but the second error is still showing. Am i correct that i can already listen to 5043 port and i'm still getting this error(second pic), because my nifi cannot handle the amount of events? I'm sorry, newbie here. Thank you. Regie ... View more

Hi guys, i would like to ask, if this is possible in nifi alone without using any script execution. test.txt <--- so this file of mine need to be extracted. Please see the attached file. I just need to get the VALUE of line1 to line4 and save it to hbase. The problem here is it has multiple request in 1 file. I need to get all the line1 to line4 per request. PS. the count of lines per requests are different. My file is just an example of what the file looks like. Thank you. ... View more

Hi, Im using NiFi here and the data flow is running every minute. Im fetching csvs from a web service and put it in hdfs. Now every file inserted in HDFS, i need to save it to HBASE. But I need to delete rows that is over 2 hours ago. For example the time now is 11:03AM and i have a records/rows that is inserted at 9:03 AM, when the time became 11:04AM i need to delete the records that was insert at 9:03AM. This process of deleting records should also run every minute. And this deleted records also need to be deleted in SOLR+Banana UI. Is this possible? Thanks. ... View more

Can i use SolR + Banana Ui as Big Data reporting? Im looking for realtime update reporting for our project and i saw this Solr. Now im trying it and its fine. But the problem is, when im running my nifi job going to Solr, it will fucntion well but after 20 secs the solr and banana started to crashed and and the service stops automatically. I saw that the JVM Memory is full. By the way im using sandbox. Any tips how can i fixed this? Thank you! Regie ... View more

Is there any NiFi processor i can use in executing SQOOP? If none, any data flow i can use in getting my table then save it to HDFS? Thanks. ... View more

My table in hive have more than 80 million records. Now we have to make a back-up of it. Do you have a best practice to do this? i tried to use my data flow in nifi but it is hanging. Now im trying PUTTY (hive -e 'select * from your_Table' | sed 's/[\t]/,/g' > /home/yourfile.csv), nothing happened when i run it. Im using sandbox hdp 2.5. Thanks! ... View more

Hi guys, Im using the GetSoap processor in NiFi. I need to get my data in web service. But it always have an error, GetSOAP[id=63baafa9-0159-1000-e53d-5195bbfe3819] GetSOAP[id=63baafa9-0159-1000-e53d-5195bbfe3819] failed to process session due to java.lang.NoSuchMethodError: org.apache.nifi.processors.soap.GetSOAP.getLogger()Lorg/apache/nifi/logging/ProcessorLog;: java.lang.NoSuchMethodError: org.apache.nifi.processors.soap.GetSOAP.getLogger()Lorg/apache/nifi/logging/ProcessorLog; Im using https, is this the one that causes the error? Thank you. ... View more

Hi guys, Is there any process where i can make my XSL File to CSV File? Thanks. ... View more