Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

[#1]

Hi Guys,

I am following the document https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/startin... which talks about setting up of STS and starting the service.

In our Keerberized cluster, after successful service addition, when I try to start the service, I observed that it fails.

16/05/19 10:27:00 INFO AbstractService: Service:HiveServer2 is started.
16/05/19 10:27:00 INFO HiveThriftServer2: HiveThriftServer2 started
16/05/19 10:27:00 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
16/05/19 10:27:00 INFO Server: jetty-8.y.z-SNAPSHOT
16/05/19 10:27:00 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:10001: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)

When I observed the logs it says, port 10001 is already in use BindingException. In HiveServer2 I notice that for HTTP transport mode, bound port is 10001.

When I manually stopped HS2 on the same host and then started STS, then it works fine.

Here I've a few questions.

1) in the log it says that HS2 is started.. Is it trying to start HS2 ?

2) Port issue: Why still 10001 and not 10015?

For the STS, we have configured the port 10015 but when I try to connect using beeline : (After having a valid ticket)

beeline -u "jdbc:hive2://STSHOST:10015/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm" it fails but when I try to connect on port 10001 it works (previously I used to connect to HS2 using the same command).

beeline -u "jdbc:hive2://STSHOST:10001/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm"

And I can use STS to submit the sqls.

Could anyone please try to explain this behavior?

Can I have HS2 and STS running on the same node?

Thanks.

Tagging experts:

@vshukla, @Timothy Spann, @Jitendra Yadav, @Yuta Imai @Simon Elliston Ball

[#2]

@Smart Solutions

If you use http mode, please specify hive.server2.thrift.http.port to a different port.

[#3]

Hi @Smart Solutions

I am lil late to the party, but I was able to run both HS2 and STS on the same machine on a kerberized cluster using:

HS2 => hive.server2.thrift.http.port: 10001, transportmode: http,

STS => hive.server2.thrift.port:10015, hive.server2.transport.mode: binary

STS does not start and throws the bind exception when I use:

STS => hive.server2.thrift.port:10015, hive.server2.transport.mode: http

So changing transport mode for STS to binary works for me.

Tested on HDP 2.4.2.29-4

Regards

Rakesh

[#4]

And guessing from what is happening here I thought I should try starting up the thrift server in http mode on an http port.

but http port for thrift server is not defined anywhere by Ambari, so created an custom propery for thrift server: 'hive.server2.thrift.http.port: 10013'

and defined 'hive.server2.transport.mode: http'

And it started thrift server on a different port (10013) in http mode!

Regards

Rakesh

[#5]

@Rakesh Gupta Thank you very much. Smart debug sir. Saved me my day today.