SparkContext did not initialize after waiting for 100000 ms

[#1]

I am trying to run a spark app in hdp cluster. My app keeps failing on:

ERROR ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application.

In the previous lines the only error (which I see a few times in the log) is:

 WARN DataSource: Error while looking for metadata directory.

Any ideas?

[#2 · Accepted solution]

Found the issue. Turns out that since I run my spark application with

--master yarn 

I should remove this part from the SparkSession builder:

.master(masterUrl)