Spark2 Thrift Server not start on ambari cluster

[#1]

in our ambari cluster system ( version 26 ) , we have 3 master node while Spark2 Thrift Server installed on master01 and master03

when we start both or one of then spark thrift server , its start for ashort time - 30 sec and then fail back

we can see from the ambari-agent log the following details about the spark thrift

what could be the problem

INFO 2018-01-01 22:14:51,827 RecoveryManager.py:255 - SPARK2_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2018-01-01 22:15:02,732 RecoveryManager.py:255 - SPARK2_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2018-01-01 22:15:06,054 StatusCommandsExecutor.py:65 - Adding STATUS_COMMAND for component SPARK2_THRIFTSERVER of service SPARK2 of cluster hdp to the queue.
INFO 2018-01-01 22:15:06,153 StatusCommandsExecutor.py:65 - Adding STATUS_COMMAND for component SPARK2_CLIENT of service SPARK2 of cluster hdp to the queue.
INFO 2018-01-01 22:15:06,501 RecoveryManager.py:255 - SPARK2_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2018-01-01 22:15:13,347 RecoveryManager.py:255 - SPARK2_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2018-01-01 22:15:23,356 RecoveryManager.py:255 - SPARK2_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
<br>

[#2]

anything else showing in Ambari? is spark2 running? any other issues?

[#3]

I update my quastion , see the pic with details , second only thrift is the problem all other services are up

[#4 · Accepted solution]

@Michael Bronson

Check if Yarn application is started when you start Spark2 Thrift server. Application name should be "Thrift JDBC/ODBC Server SPARK" and Application Type should be "SPARK".

If the application is not started after few minutes after starting Spark2 server from Ambari GUI, there may be resource crunch on Yarn. Check the max applications allowed in Yarn and memory settings or try adding a Node manager.

Thanks,

Aditya

[#5]

what is the variable that represented the "max applications allowed in Yarn" ?

[#6]

@Michael Bronson

Go to Yarn -> Configs -> Advanced -> Scheduler -> Capacity Scheduler . Check 'yarn.scheduler.capacity.maximum-applications'.

Default should be 10000.

[#7]

in my cluster the - yarn.scheduler.capacity.maximum-applications=10000 , can we change it as decease/increase it to see if it will help?

[#8]

another thing you say "here may be resource crunch on Yarn" , is it possible to validate it ? ( I mean how we can be sure about resource problem ) or verifcation

[#9]

This can happen if spark1 and spark2 are both running on same node.Try to kill the process. Then delete the service and add it to a separate node.It must work.