Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

[#1]

Hi Guys,

I am following the document https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/startin... which talks about setting up of STS and starting the service.

In our Keerberized cluster, after successful service addition, when I try to start the service, I observed that it fails.

16/05/19 10:27:00 INFO AbstractService: Service:HiveServer2 is started.
16/05/19 10:27:00 INFO HiveThriftServer2: HiveThriftServer2 started
16/05/19 10:27:00 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
16/05/19 10:27:00 INFO Server: jetty-8.y.z-SNAPSHOT
16/05/19 10:27:00 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:10001: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)

When I observed the logs it says, port 10001 is already in use BindingException. In HiveServer2 I notice that for HTTP transport mode, bound port is 10001.

When I manually stopped HS2 on the same host and then started STS, then it works fine.

Here I've a few questions.

1) in the log it says that HS2 is started.. Is it trying to start HS2 ?

2) Port issue: Why still 10001 and not 10015?

For the STS, we have configured the port 10015 but when I try to connect using beeline : (After having a valid ticket)

beeline -u "jdbc:hive2://STSHOST:10015/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm" it fails but when I try to connect on port 10001 it works (previously I used to connect to HS2 using the same command).

beeline -u "jdbc:hive2://STSHOST:10001/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm"

And I can use STS to submit the sqls.

Could anyone please try to explain this behavior?

Can I have HS2 and STS running on the same node?

Thanks.

Tagging experts:

@vshukla, @Timothy Spann, @Jitendra Yadav, @Yuta Imai @Simon Elliston Ball

[#2]

@Smart Solutions I'll see if I can find out more info.

[#3]

@Smart Solutions please see note from @jzhang, below (thanks Jeff!)

[#4]

@Smart Solutions

STS is listening on 10015 even in secure cluster. Have you tried to start STS through ambari ?

16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab
16/05/24 02:03:32 INFO CompositeService: Operation log root directory is created: /tmp/hive/operation_logs
16/05/24 02:03:32 INFO AbstractService: HiveServer2: Async execution pool size 100
16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is inited.
16/05/24 02:03:32 INFO AbstractService: Service: SessionManager is inited.
16/05/24 02:03:32 INFO AbstractService: Service: CLIService is inited.
16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is inited.
16/05/24 02:03:32 INFO AbstractService: Service: HiveServer2 is inited.
16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is started.
16/05/24 02:03:32 INFO AbstractService: Service:SessionManager is started.
16/05/24 02:03:32 INFO AbstractService: Service:CLIService is started.
16/05/24 02:03:32 INFO ObjectStore: ObjectStore, initialize called
16/05/24 02:03:32 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
16/05/24 02:03:32 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/05/24 02:03:32 INFO ObjectStore: Initialized ObjectStore
16/05/24 02:03:32 INFO HiveMetaStore: 0: get_databases: default
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=get_databases: default
16/05/24 02:03:32 INFO HiveMetaStore: 0: Shutting down the object store...
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=Shutting down the object store...
16/05/24 02:03:32 INFO HiveMetaStore: 0: Metastore shutdown complete.
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=Metastore shutdown complete.
16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is started.
16/05/24 02:03:32 INFO AbstractService: Service:HiveServer2 is started.
16/05/24 02:03:32 INFO HiveThriftServer2: HiveThriftServer2 started
16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab
16/05/24 02:03:32 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=0
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
16/05/24 02:03:33 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=1
16/05/24 02:03:33 INFO ThriftCLIService: Starting ThriftBinaryCLIService on port 10015 with 5...500 worker threads

[#5]

@jzhang, Yes. I did try from Ambari also. The result is same.

[#6]

Could you check whether hive.server2.thrift.port is 10015 on the ambari UI ?

[#7]

Yes it is. 10015

[#8]

You can have both HS2 & Spark Thrift Server running on the same node.

To change the port of STS, modify hive.server2.thrift.port in advanced spark-hive-site-override.conf with Ambari

Bot HS2 & STS can listen in binary (thrift) & HTTP. Binary is the default.

I validated this with HDP 2.4.2.0-258

[#9]

@Smart Solutions Can you see if this solves your issue

[#10]

@vshukla

You suggesting to change hive.server2.thrift.port in advanced spark-hive-site-override.conf? It is already set to 10015.

1) Did you check on Kerberized cluser?

2) Try setting hive.server2.transport.mode to http .

[#11 · Accepted solution]

Confirmed with HWX that its a bug.Closing the issue for now.