Member since
06-26-2017
191
Posts
10
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1448 | 09-22-2017 07:13 PM |
10-23-2017
05:25 PM
Hi All, Thanks a lot this community. I have a nifi cluster, in one the nodes on a nifi cluster, there is a PHP cron script which create files in a directory. For ingesting these files I research on this community, to my understanding I should use "ListFile processor" on the node on which the file is generated and then use remote process group to input to fetch file processor. I read on this community that the Listfile processor should be used on Primary nifi node? Is it necessary, if yes, then primary keeps changing if the primary goes down, then the elected primary will not be abe to access the directory. Or I can use it on one of the nodes regardless of which is node is primary. I read this link https://community.hortonworks.com/questions/109247/how-to-distribute-files-on-nifi-cluster-and-proces.html Thanks Dheeru
... View more
Labels:
- Labels:
-
Apache NiFi
10-23-2017
04:57 PM
Hi All, Thanks to this awesome community I am trying to emulate my data sources (pfSense firewall), which sends events at 10000 events per second. So I configured a Generate flow file processor with 2kb event coming from firewall as custom text. Is there any way I control/set the speed of generate flow file processor? Thanks Dheeru
... View more
Labels:
- Labels:
-
Apache NiFi
10-20-2017
07:07 PM
@Wynner you were correct the content and flowfile were refering to root, changed it and fixed it by referring content repo to different disk and flow file to different raid/disk this link also helped me https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html
... View more
10-20-2017
05:56 PM
Hi All, Thanks a lot for help on this community Since the number of events is huge 15000 events per second, in order to make it efficient and quick I am using distributeload processor and 2 merge content in parallel. However my confusion is, will it change the order of events coming ?2017-10-20-photo-00001717.jpg
... View more
Labels:
- Labels:
-
Apache NiFi
10-19-2017
06:40 PM
@Matt Clarke Thanks a lot for your response. I clicked on a process group which was already running fine and good, while my intention was to just see its configuration I accidentally clicked on "Stop". So the whole process group stopped and then we did not receive the data from the pfsense firewall for that little duration. what I was looking for exactly what you suggested, like when we know the data flow is ready and does not need any more changes, we can change it to read mode only or Stop, options freezes. Yes I agree completely asking twice for confirmation will be annoying, but the dataflows which are ready and we mark them to ask us twice, that can be a good feature. Thank you Dheeru
... View more
10-19-2017
06:03 PM
Hi All, Thanks a lot everyone on this awesome community. I have data flows in PROD now, and I am adding new data flows and testing them, however accidentally I stopped one the flows which was good and running in PROD, reuslting in data loss. Is there any way I can freeze the data flow or dataflow confirms twice before I stop a data flow? Thanks Dheeru
... View more
Labels:
- Labels:
-
Apache NiFi
10-19-2017
03:17 PM
@Wynner Appreciate your help. Yes it is indicating that I have run out of the disk space. which was proved by this screenshot, however the acutal size of disk 4.5TB. Thanks contentstorage.pngflowstorage1.png
... View more
10-19-2017
03:12 PM
Hi @Wynner Appreciate your response.contentrepo.png And the nifi.content.repository.directory.default=/data02/content_repository
... View more
10-19-2017
02:52 PM
@Matt Clarke Thanks a lot this helps. Appreciate your help. Is there anyway I can know the size of flow file coming out from ListenUDP processors PS How can accept your answer? Thanks
... View more
10-19-2017
02:03 PM
Hi All, Thanks The NiFi UI is showing the content and flow file repository to be used 100% (and set to only 1.91 GB) but in the nifi.properties file it has been configured for 3 TB each. flowstorage1.pngcontentstorage.png
... View more
Labels:
- Labels:
-
Apache NiFi