I've started a streaming job based on a tutorial in our data lake from PySpark shell using a YARN as the master. As usual, it has started a YARN application and the application ran fine and moved to the FINISHED state in the YARN web UI. But Spark history server still lists the job in the "Show Incomplete Applications" section.
Even though nothing seems to appear on the Spark Web UI, no updates as such.
I've even tried to kill the YARN application but because it's already in the FINISHED state, that didn't work. I don't even see the "kill" button on the Spark Web UI.
Could you please suggest me what else am I missing.
Spark : 2.3.2
Please let me know if any other information is needed.