Skip to main content
F_Amini
F_Amini
Silver ⭐️
November 25, 2022
Question

Nifi queues are full and create backlogs

Hi guys, I am facing a problem with Nifi I have about 80G data in queues that make backlogs and I need the data to be ingested faster and but I can't clear the queues in case of missing files, Please help me solve this issue, thanks a lot. 

NIFI full queues:((((NIFI full queues:((((

5 replies

Green_
Green_
Gold ⭐️⭐️
November 25, 2022

Hey @F_Amini ,

 

If the data has already been loaded into nifi, I don't believe you can do much other than to increase your resources. I see you're already familar with increasing concurrent tasks (it appears your HashContent processor has an increased number) - perhaps if your entire flow is getting bottlenecked by a single processor, you could temporarily increase its tasks even further (at the cost of other processors getting less).

Translate
F_Amini
F_AminiAuthor
Silver ⭐️
November 25, 2022

Hi dear @Green_ Thanks for your answer,
Yes I raised the current task but it doesn't change anything, so the ultimate solution you suggest is increasing the CPU, Is there anything we can do for tuning to speed up the ingestion in Nifi?
It's crucial to my job so I appreciate any suggestion you have, Thanks a lot.

Translate
    Green_
    Green_
    Gold ⭐️⭐️
    November 25, 2022

    So long as backpressure is building up behind a processor, the processor that is previous in line will not be scheduled. This will fall back all the way throughout your flow until it reaches the processor that fetches the data and cause it to stop. As such, all it takes is one weak-link processor to limit your entire flow's throughput. I see the relationship coming out of your HashContent is also red, which implies that there is a different processor further down-flow that is causing backpressure. Can you describe which processor causes this? perhaps fixing its slow performance will improve your entire flow.

    Translate
      Green_
      Green_
      Gold ⭐️⭐️
      December 9, 2022

      Hey @F_Amini,

       

      Has your issue been resolved? I'd be glad to help if needed. 

      Translate
      Terms & ConditionsAccessibility statement