Member since
06-26-2017
191
Posts
10
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1448 | 09-22-2017 07:13 PM |
10-30-2017
04:24 PM
Hi All, Thanks for help to this awesome community nifi.cluster.node.protocol.max.threads= The field is empty by default in nifi.properties We have a data source feeds producing 15000 events per second. What do you guys recommend. As per recommendaions from this community I changed these properties for the above mentioned event per second nifi.cluster.node.connection.timeout=15 sec nifi.cluster.node.read.timeout=15 sec nifi.cluster.node.protocol.threads=50 Need suggestions recomendations. I know I have to tune it by changing these properties to reach optimal performance, however a startin value for this nifi.cluster.node.protocol.max.threads= be awesome Thanks Dheeru
... View more
Labels:
- Labels:
-
Apache NiFi
10-26-2017
02:25 PM
@Bryan Bende Thanks a lot. Could you please help my clear my understanding on this. Appreciate it
... View more
10-26-2017
01:12 PM
@Abdelkrim Hadjidj @Matt Burgess The use case here is getting the files from Mainframe, once a each day, however I got to know that there is no concept directory structure in Mainframes ( no idea how mainframes work), so it is not able to list the files (GetFTP as well as ListFTP and FetchFTP) Is there any other way to get around? I read some blogs and answers which suggested to use syncsort or Informatica powercenter. We tried using our current way of running shell script which goes and fetch the files. We can run the script using executeProcess and save it on one of the nodes (primary nodes), however primary node keeps changing. yesterday it was different primary today it is different one. In addition, if we mount it to share the directory across the node, it will be against the policies (too much admin work) Any help, thoughts
... View more
10-25-2017
11:46 PM
Hi All, Thanks a lot to this awesome community. I was wondering about the significance of Max Request Size field in PublishKafka processor attached in the image. To imporve performance I have set "Message Demaractor" as new line (shift+enter) Also as per my understanding the custom property added in Publish kafka processor (see image attached) does the same thing. My single event size is not more than 4 kb, when it pushed from source to listenTCP processor, in the listenTCP porcessor we batch them and them merge them to 128 MB , the block size of hdfs cluster. So my Max request size should be 128 MB or 4 Kb or not dependent on this. Thanks Dheeru
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
10-25-2017
08:29 PM
@Matt Burgess it worked my bad, it was basically creating flow file for each of the nodes Appreciate it Thanks
... View more
10-25-2017
08:25 PM
@Matt Burgess I tried the approach of using generating flow file and schedule it using Cron, and set a custom property as filename, attached is the image, the problem is as you mentioned runs at the rate of generateflow file, how can control the speed, I scheduled it run only once still it runs very fast
... View more
10-25-2017
07:27 PM
Hi All, Thanks a lot to this awesome community I want to fetch some files from ftp and I do not intend to use on a clsuter, using primary only for this, because it runs only once a day. Right now fetchftp says it needs an upstream connection, basically looking for a corresponding listFTP Can we just fetch the file using ftp, since the use case is top fetch from a mainframe and it does not have a directory structure to list the files
... View more
Labels:
- Labels:
-
Apache NiFi
10-25-2017
06:47 PM
1 Kudo
@Shu Thanks I just did that before I saw this answer, it worked. Appreciate it. Using the website you refered yesterday http://www.cronmaker.com/ Thanks again
... View more
10-25-2017
06:24 PM
Hi All, Thanks a lot to this awesome community I followed this article https://community.hortonworks.com/articles/110097/use-excecuteprocess-processor-to-execute-shell-scr.html to run a shell script, the behavior as of now is that is running the command multiple times. Can we control how many time the script should run? Thanks Dheeru
... View more
Labels:
- Labels:
-
Apache NiFi
10-25-2017
04:03 PM
@Bryan Bende Aweosme Thanks and appreciate it
... View more