Support Questions
Find answers, ask questions, and share your expertise

Livy Executing Spark Job DatabaseNotFound but works in spark-submit

Hey all,

I have a spark job which works fine via spark-submit that fails when executed through the livy rest API with the following exception:

"User class threw exception: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'some_database' not found"

The thing is, the job already works. I've tried modifying all sorts of stuff in the livy conf, in the REST API call (changing the X-Requested-By user, changing the proxyUser, enabling/disabling various proxyUser settings etc to get it to work. I've following all logs from livy (doesn't seem to log anything much) through to YARN, hdfs and the spark logs. The only thing that gives any kind of useful output if the YARN log which tells me I had an exception.

Any ideas? I don't see why livy isn't just getting the spark/scala defaults and running the job as whatever user I specify?