Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

SparkContext did not initialize after waiting for 100000 ms

Solved Go to solution
Highlighted

SparkContext did not initialize after waiting for 100000 ms

Contributor

I am trying to run a spark app in hdp cluster. My app keeps failing on:


ERROR ApplicationMaster: SparkContext did not initialize after waiting for 100000 ms. Please check earlier log output for errors. Failing the application.

In the previous lines the only error (which I see a few times in the log) is:

 WARN DataSource: Error while looking for metadata directory.

Any ideas?
1 ACCEPTED SOLUTION

Accepted Solutions

Re: SparkContext did not initialize after waiting for 100000 ms

Contributor

Found the issue. Turns out that since I run my spark application with

--master yarn 
I should remove this part from the SparkSession builder:

.master(masterUrl)

View solution in original post

1 REPLY 1

Re: SparkContext did not initialize after waiting for 100000 ms

Contributor

Found the issue. Turns out that since I run my spark application with

--master yarn 
I should remove this part from the SparkSession builder:

.master(masterUrl)

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here