Do I need Spark running to see Data Lineage in Atlas? (HDP 3.0 Sandbox)

[#1]

I have done the Getting started with HDP sandbox tutorial so I have created hive tables with DAS and done some Spark using Zeppelin.

However, as I try to use Atlas, it is suggested on the tutorial that you turn off both Spark and Zeppelin so that the Sandbox works properly (I tried to enable Atlas with Spark and Zeppelin running and it crashed and I had to restart the VM).

My question is, do I need Spark running to see Data Lineage in Atlas? I am able to see hive tables, hive processes... and everything I did on DAS but there is nothing showing in relation to Spark.

Thank you.

[#2 · Accepted solution]

You do not need Spark running to see Hive table lineage in Atlas.

However, if you're using the Spark hook, you must have Spark turned on.