Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

[#1]

Hi Guys,

I am following the document https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/startin... which talks about setting up of STS and starting the service.

In our Keerberized cluster, after successful service addition, when I try to start the service, I observed that it fails.

16/05/19 10:27:00 INFO AbstractService: Service:HiveServer2 is started.
16/05/19 10:27:00 INFO HiveThriftServer2: HiveThriftServer2 started
16/05/19 10:27:00 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
16/05/19 10:27:00 INFO Server: jetty-8.y.z-SNAPSHOT
16/05/19 10:27:00 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:10001: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)

When I observed the logs it says, port 10001 is already in use BindingException. In HiveServer2 I notice that for HTTP transport mode, bound port is 10001.

When I manually stopped HS2 on the same host and then started STS, then it works fine.

Here I've a few questions.

1) in the log it says that HS2 is started.. Is it trying to start HS2 ?

2) Port issue: Why still 10001 and not 10015?

For the STS, we have configured the port 10015 but when I try to connect using beeline : (After having a valid ticket)

beeline -u "jdbc:hive2://STSHOST:10015/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm" it fails but when I try to connect on port 10001 it works (previously I used to connect to HS2 using the same command).

beeline -u "jdbc:hive2://STSHOST:10001/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm"

And I can use STS to submit the sqls.

Could anyone please try to explain this behavior?

Can I have HS2 and STS running on the same node?

Thanks.

Tagging experts:

@vshukla, @Timothy Spann, @Jitendra Yadav, @Yuta Imai @Simon Elliston Ball

[#2]

@Smart Solutions please see note from @jzhang, below (thanks Jeff!)

[#3]

@Smart Solutions

STS is listening on 10015 even in secure cluster. Have you tried to start STS through ambari ?

16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab
16/05/24 02:03:32 INFO CompositeService: Operation log root directory is created: /tmp/hive/operation_logs
16/05/24 02:03:32 INFO AbstractService: HiveServer2: Async execution pool size 100
16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is inited.
16/05/24 02:03:32 INFO AbstractService: Service: SessionManager is inited.
16/05/24 02:03:32 INFO AbstractService: Service: CLIService is inited.
16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is inited.
16/05/24 02:03:32 INFO AbstractService: Service: HiveServer2 is inited.
16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is started.
16/05/24 02:03:32 INFO AbstractService: Service:SessionManager is started.
16/05/24 02:03:32 INFO AbstractService: Service:CLIService is started.
16/05/24 02:03:32 INFO ObjectStore: ObjectStore, initialize called
16/05/24 02:03:32 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
16/05/24 02:03:32 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/05/24 02:03:32 INFO ObjectStore: Initialized ObjectStore
16/05/24 02:03:32 INFO HiveMetaStore: 0: get_databases: default
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=get_databases: default
16/05/24 02:03:32 INFO HiveMetaStore: 0: Shutting down the object store...
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=Shutting down the object store...
16/05/24 02:03:32 INFO HiveMetaStore: 0: Metastore shutdown complete.
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=Metastore shutdown complete.
16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is started.
16/05/24 02:03:32 INFO AbstractService: Service:HiveServer2 is started.
16/05/24 02:03:32 INFO HiveThriftServer2: HiveThriftServer2 started
16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab
16/05/24 02:03:32 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=0
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
16/05/24 02:03:33 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=1
16/05/24 02:03:33 INFO ThriftCLIService: Starting ThriftBinaryCLIService on port 10015 with 5...500 worker threads

[#4]

@jzhang, Yes. I did try from Ambari also. The result is same.

[#5]

Could you check whether hive.server2.thrift.port is 10015 on the ambari UI ?

[#6]

Yes it is. 10015

[#7]

You can have both HS2 & Spark Thrift Server running on the same node.

To change the port of STS, modify hive.server2.thrift.port in advanced spark-hive-site-override.conf with Ambari

Bot HS2 & STS can listen in binary (thrift) & HTTP. Binary is the default.

I validated this with HDP 2.4.2.0-258

[#8]

@Smart Solutions Can you see if this solves your issue

[#9]

@vshukla

You suggesting to change hive.server2.thrift.port in advanced spark-hive-site-override.conf? It is already set to 10015.

1) Did you check on Kerberized cluser?

2) Try setting hive.server2.transport.mode to http .

[#10 · Accepted solution]

Confirmed with HWX that its a bug.Closing the issue for now.

[#11]

@Smart Solutions

If you use http mode, please specify hive.server2.thrift.http.port to a different port.

[#12]

Hi @Smart Solutions

I am lil late to the party, but I was able to run both HS2 and STS on the same machine on a kerberized cluster using:

HS2 => hive.server2.thrift.http.port: 10001, transportmode: http,

STS => hive.server2.thrift.port:10015, hive.server2.transport.mode: binary

STS does not start and throws the bind exception when I use:

STS => hive.server2.thrift.port:10015, hive.server2.transport.mode: http

So changing transport mode for STS to binary works for me.

Tested on HDP 2.4.2.29-4

Regards

Rakesh

[#13]

And guessing from what is happening here I thought I should try starting up the thrift server in http mode on an http port.

but http port for thrift server is not defined anywhere by Ambari, so created an custom propery for thrift server: 'hive.server2.thrift.http.port: 10013'

and defined 'hive.server2.transport.mode: http'

And it started thrift server on a different port (10013) in http mode!

Regards

Rakesh

[#14]

@Rakesh Gupta Thank you very much. Smart debug sir. Saved me my day today.