question Re: SparkContext giving error in initialization : org.apache.hadoop.ipc.StandbyException in Support Questions

SparkContext giving error in initialization : org.apache.hadoop.ipc.StandbyException

yukti — Fri, 06 Sep 2019 10:11:50 GMT

while running a normal spark job SparkContext is not getting initialized giving below error:

[Thread-6] ERROR org.apache.spark.SparkContext - Error initializing SparkContext.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby.

I believe it is pointing to standby NN.

This error is coming after we enabled Namenode HA in our cloudera's cluster.

I have checked hive-site.xml, core-site.xml and spark-defaults.xml.

These all are havinf correct configs pointing to nameservice id.

Re: SparkContext giving error in initialization : org.apache.hadoop.ipc.StandbyException

EricL — Fri, 06 Sep 2019 22:38:19 GMT

@yukti,

Are you able to share full spark log so that we can see a bit more context? Hitting SBNN is common, but it should be back to Active NN after that, and it should not be causing failure. I suspect there is something else, so full log might help to understand a bit more.

Cheers
Eric

Re: SparkContext giving error in initialization : org.apache.hadoop.ipc.StandbyException

yukti — Mon, 09 Sep 2019 09:03:22 GMT

Hi Eric
please see the attached error logs.

@EricL

Re: SparkContext giving error in initialization : org.apache.hadoop.ipc.StandbyException

EricL — Wed, 11 Sep 2019 06:54:09 GMT

Hmm, looks like Spark is not able to reach to NN, are both of your NN up and running? Can you run normal HDFS commands and operations?

Can you also share the spark-defaults.conf file which is under /etc/spark/conf or /etc/spark2/conf directory for review?

Thanks

Re: SparkContext giving error in initialization : org.apache.hadoop.ipc.StandbyException

yukti — Wed, 11 Sep 2019 09:08:46 GMT

@EricL

Hi Eric
it was pointing to wrong spark conf so I replaced it with the new one.

But now its giving me another error.
I will open anew thread for another error.