SparkContext did not initialize after waiting for 100000 ms

yogen — Wed, 25 Oct 2017 15:12:41 GMT

I am trying to run a spark app in hdp cluster. My app keeps failing on:

ERROR ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application.

In the previous lines the only error (which I see a few times in the log) is:

 WARN DataSource: Error while looking for metadata directory.

Any ideas?

Re: SparkContext did not initialize after waiting for 100000 ms

yogen — Wed, 25 Oct 2017 21:24:07 GMT

Found the issue. Turns out that since I run my spark application with

--master yarn

I should remove this part from the SparkSession builder:

.master(masterUrl)

question SparkContext did not initialize after waiting for 100000 ms in Archives of Support Questions (Read Only)

SparkContext did not initialize after waiting for 100000 ms

Re: SparkContext did not initialize after waiting for 100000 ms