question Re: Spark2 Thrift Server not start on ambari cluster in Support Questions

Spark2 Thrift Server not start on ambari cluster

mike_bronson7 — Sun, 18 Aug 2019 09:24:45 GMT

in our ambari cluster system ( version 26 ) , we have 3 master node while Spark2 Thrift Server installed on master01 and master03

when we start both or one of then spark thrift server , its start for ashort time - 30 sec and then fail back

we can see from the ambari-agent log the following details about the spark thrift

what could be the problem

INFO 2018-01-01 22:14:51,827 RecoveryManager.py:255 - SPARK2_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2018-01-01 22:15:02,732 RecoveryManager.py:255 - SPARK2_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2018-01-01 22:15:06,054 StatusCommandsExecutor.py:65 - Adding STATUS_COMMAND for component SPARK2_THRIFTSERVER of service SPARK2 of cluster hdp to the queue.
INFO 2018-01-01 22:15:06,153 StatusCommandsExecutor.py:65 - Adding STATUS_COMMAND for component SPARK2_CLIENT of service SPARK2 of cluster hdp to the queue.
INFO 2018-01-01 22:15:06,501 RecoveryManager.py:255 - SPARK2_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2018-01-01 22:15:13,347 RecoveryManager.py:255 - SPARK2_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
INFO 2018-01-01 22:15:23,356 RecoveryManager.py:255 - SPARK2_THRIFTSERVER needs recovery, desired = STARTED, and current = INSTALLED.
<br>

Re: Spark2 Thrift Server not start on ambari cluster

TimothySpann — Tue, 02 Jan 2018 07:43:05 GMT

anything else showing in Ambari? is spark2 running? any other issues?

Re: Spark2 Thrift Server not start on ambari cluster

mike_bronson7 — Tue, 02 Jan 2018 13:55:52 GMT

I update my quastion , see the pic with details , second only thrift is the problem all other services are up

Re: Spark2 Thrift Server not start on ambari cluster

asirna — Tue, 02 Jan 2018 14:41:17 GMT

@Michael Bronson

Check if Yarn application is started when you start Spark2 Thrift server. Application name should be "Thrift JDBC/ODBC Server SPARK" and Application Type should be "SPARK".

If the application is not started after few minutes after starting Spark2 server from Ambari GUI, there may be resource crunch on Yarn. Check the max applications allowed in Yarn and memory settings or try adding a Node manager.

Thanks,

Aditya

Re: Spark2 Thrift Server not start on ambari cluster

mike_bronson7 — Tue, 02 Jan 2018 15:55:40 GMT

what is the variable that represented the "max applications allowed in Yarn" ?

Re: Spark2 Thrift Server not start on ambari cluster

asirna — Tue, 02 Jan 2018 16:33:21 GMT

@Michael Bronson

Go to Yarn -> Configs -> Advanced -> Scheduler -> Capacity Scheduler . Check 'yarn.scheduler.capacity.maximum-applications'.

Default should be 10000.

Re: Spark2 Thrift Server not start on ambari cluster

mike_bronson7 — Tue, 02 Jan 2018 16:56:32 GMT

in my cluster the - yarn.scheduler.capacity.maximum-applications=10000 , can we change it as decease/increase it to see if it will help?

Re: Spark2 Thrift Server not start on ambari cluster

mike_bronson7 — Tue, 02 Jan 2018 20:20:34 GMT

another thing you say "here may be resource crunch on Yarn" , is it possible to validate it ? ( I mean how we can be sure about resource problem ) or verifcation

Re: Spark2 Thrift Server not start on ambari cluster

achandra — Thu, 17 May 2018 01:48:25 GMT

This can happen if spark1 and spark2 are both running on same node.Try to kill the process. Then delete the service and add it to a separate node.It must work.