Created 11-06-2020 07:00 AM
Hi All,
I am running spark (livy) from nifi. Currently using HDP 3.0/HDF 3.1 the CDA version.
I was running spark livy through nifi. I could see that nifi flow was completed successfully. But there were several livy-session keep executing in Yarn. I've tried to kill some of them but they were re-appearing in Yarn.
This is causing issue that these livy sessions are consuming CPUs even though I am not running any spark application through livy-nifi.
I am not sure how to resolve this issue. Needed your help here.
Created 11-06-2020 08:55 AM
Created 11-06-2020 09:22 AM
I don't recommend using the spark livy connector anymore.
Also by default it's not setup for use for anything other than zeppelin.
i would connect via kafka.
in nifi you can check the livy controller settings and make sure it's not respawning new ones.