Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How control count livy-session LivySessionController in YARN from NIFI

Highlighted

How control count livy-session LivySessionController in YARN from NIFI

Contributor

Hi all!

we are running pyspark code with NIFI ExecuteSparkInteractive, but in YARN see a lot of generated livy-session about 103024

Why this happen ?

91691-livy-count-sessions.png

4 REPLIES 4
Highlighted

Re: How control count livy-session LivySessionController in YARN from NIFI

Expert Contributor

103024 sessions could translate to a session every 90 seconds for the last few months. How many sessions did you expect to have (and what is the context)?


- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'. Also check out my techincal portfolio at https://portfolio.jaheruddin.nl

Re: How control count livy-session LivySessionController in YARN from NIFI

New Contributor

Did you get a solution on this issue ? We are facing a similar problem with the sessions created by the nifi-livy integration not cleaning up/closing up.

Highlighted

Re: How control count livy-session LivySessionController in YARN from NIFI

Explorer

Even I am facing the same issue. As soon as I enable the controller service its creating livy sessions. I havent started any processors.

Highlighted

Re: How control count livy-session LivySessionController in YARN from NIFI

Explorer

Hi,

Did you get any resolution on this issue? Can you please help with this?

 

Thanks,

Gazal

Don't have an account?
Coming from Hortonworks? Activate your account here