Support Questions
Find answers, ask questions, and share your expertise

Spark livy execution from nifi: livy-session are keep instantiating.

Spark livy execution from nifi: livy-session are keep instantiating.

Contributor

Hi All,

 

I am running spark (livy) from nifi. Currently using HDP 3.0/HDF 3.1 the CDA version.

 

I was running spark livy through nifi. I could see that nifi flow was completed successfully. But there were several livy-session keep executing in Yarn. I've tried to kill some of them but they were re-appearing in Yarn. 

This is causing issue that these livy sessions are consuming CPUs even though I am not running any spark application through livy-nifi.

 

I am not sure how to resolve this issue. Needed your help here.

Screenshot from 2020-11-06 16-54-46.png

2 REPLIES 2

Re: Spark livy execution from nifi: livy-session are keep instantiating.

Contributor

Hi @TimothySpann 

 

Could you please help here.

Re: Spark livy execution from nifi: livy-session are keep instantiating.

Super Guru

I don't recommend using the spark livy connector anymore.

 

Also by default it's not setup for use for anything other than zeppelin.

 

i would connect via kafka.

 

in nifi you can check the livy controller settings and make sure it's not respawning new ones.