Support Questions
Find answers, ask questions, and share your expertise

How control count livy-session LivySessionController in YARN from NIFI

Contributor

Hi all!

we are running pyspark code with NIFI ExecuteSparkInteractive, but in YARN see a lot of generated livy-session about 103024

Why this happen ?

91691-livy-count-sessions.png

4 REPLIES 4

Re: How control count livy-session LivySessionController in YARN from NIFI

Super Collaborator

103024 sessions could translate to a session every 90 seconds for the last few months. How many sessions did you expect to have (and what is the context)?


- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'. Also check out my techincal portfolio at https://portfolio.jaheruddin.nl

Re: How control count livy-session LivySessionController in YARN from NIFI

New Contributor

Did you get a solution on this issue ? We are facing a similar problem with the sessions created by the nifi-livy integration not cleaning up/closing up.

Re: How control count livy-session LivySessionController in YARN from NIFI

Explorer

Even I am facing the same issue. As soon as I enable the controller service its creating livy sessions. I havent started any processors.

Re: How control count livy-session LivySessionController in YARN from NIFI

Explorer

Hi,

Did you get any resolution on this issue? Can you please help with this?

 

Thanks,

Gazal