question Re: Run Spark Thrift server on multiple nodes in Archives of Support Questions (Read Only)

Run Spark Thrift server on multiple nodes

hoangletrung — Fri, 28 Jul 2017 16:50:06 GMT

Currently I running only one Thrift server on my cluster.

I see if I have many client connections, Spark run very slow.

I find the solution for this and I see the doc: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_spark-component-guide/content/install-spark-over-ambari.html

It said: Deploying the Thrift server on multiple hosts increases scalability of the Thrift server; the number of hosts should take into consideration the cluster capacity allocated to Spark.

So If I deploy more Thrift server on my cluster, Can I handle multiple concurrent client connections ? If not, how to I can handle this issue ?.

And other question, Can I running two Spark apps with different configure on each?

Re: Run Spark Thrift server on multiple nodes

youngick — Sat, 29 Jul 2017 17:06:35 GMT

I think you need load balancing for Spark Thrift servers.

Why don't you refer this link?

'https://community.hortonworks.com/questions/29687/how-to-do-load-balancing-spark-thrift-servers-on-h.html'

Re: Run Spark Thrift server on multiple nodes

hoangletrung — Mon, 31 Jul 2017 09:49:03 GMT

@youngick kim

I refer that link suggest, but I can not see any guide to solve my issue.

Best answer saild :

So for now... no load balancing for STS if the cluster is kerberized, otherwise haproxy, httpd +mod_jk or any other load balancer will probably do the work.

P/S: Currently, my cluster do not enable kerberrized

Re: Run Spark Thrift server on multiple nodes

rhardaway — Wed, 02 Aug 2017 12:05:31 GMT

Hoang,

I think what that post is saying is while there is not yet a load balancing solution for STS on a secure cluster, in a non-Kerberos environment any external load balancer, like haproxy, or httpd +mod_jk can be used to distribute requests. The Thrift servers will be independent (no Zk coordination) and the load balancer will simply round robin client requests.

Re: Run Spark Thrift server on multiple nodes

hoangletrung — Thu, 03 Aug 2017 08:30:37 GMT

So I need use third party service like Haproxy to running Thrift server load balancing on HDP

Re: Run Spark Thrift server on multiple nodes

rhardaway — Thu, 03 Aug 2017 10:30:40 GMT

yes, that is my understanding.