@F_Amini @Green_ Is absolutely correct here. You should be careful when increasing concurrent tasks as just blindly increasing it everywhere can have the opposite effect on throughput. I recommend stopping setting the concurrent tasks back to 1 or maybe 2 on all the processor where you have adjusted away from the default of 1 concurrent task. Then take a look at the processor further downstream in your dataflow where it has a red input connection but black (no backpressure) outbound connections. This processor s @Green_ mentioned is the processor causing all your upstream backlog. Your'll want to monitor your CPU usage as you make small incremental adjustments to this processors concurrent tasks until you see the upstream backlog start to come down. If while monitoring CPU, you see it spike pretty consistently at 100% usage across all your cores, then your dataflow has pretty much reached the max throughput it can handle for yoru specific dataflow design. At this point you need to look at other options like setting up a NiFi cluster where this work load can be spread across multiple servers or designing your datafow differently with different processors to accomplish same use case that may have a lesser impact on CPU (not always a possibility). Thanks, Matt
... View more
@Sinchan Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks
... View more
@grb Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks
... View more
@dreaminz Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks
... View more
@Ghilani NiFi stores Templates in the flow.xml.gz file. the flow.xml.gz is just a compressed copy of dataflow(s) which reside inside NiFi's heap memory while NiFi is running. It is not recommended to keep templates in your NiFi. NiFi templates are also deprecated and will go away in next major release. It is recommended to use NiFi-registry to store version controlled flows. If not using NiFi-Registry, Flow definitions should be downloaded instead of creating templates and stored safely somewhere outside of NiFi itself. A flow definition can be downloaded by right clicking on a process group in NiFi and selecting "Download flow definition". This json file will be generated of that flow and downloaded. Flow definitions can be uploaded to NiFi by dragging the create Process Group icon to the canvas and selecting option to upload flow definition. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
@Mohamed_Shaaban I recommend starting a new community question with the details specific to your setup. This allows the community to address/assist with your specific setup versus comparing your issue to what was shared in this post. Thanks, Matt
... View more
id, name, balance, join_date, notes 1, John, 48.23, 04/03/2007 "Our very first customer!" 2, Jane, 1245.89, 08/22/2009, 3, Frank Franklin, "48481.29", 04/04/2016, the logic may fail in the above case, if column contains multiple rows.
... View more
@Kushisabishii could you please advise what list of host needs to be provided on the NIFI_WEB_PROXY_HOST I have added the docker host and port that I access the NiFi UI. For example my NiFi is accessible via https://docker-host.comp.com/nifi/ Hence I have added the NIFI_WEB_PROXY_HOST = docker-host.comp.com:443 into my docker compose file. Do I need to consider any other hosts to be added ?
... View more
@drewski7 New processors are created within the community all the time and the documentation for processors should include resource considerations for CPU usage and Memory usage. Just because a processor list memory as a resource consideration, that impact is often a byproduct of how that processor has been configured. you can refer to the imbedded documentation in your installed NiFi instance or you can right click on a processor added to the canvas and select "view usage" from the displayed context menu to go directly to that components embedded documentation page. But processors like ReplaceText, SplitText, SplitContent, SplitJson,... would be examples. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
