Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

[#1]

@Smart Solutions I'll see if I can find out more info.

[#2]

@Smart Solutions please see note from @jzhang, below (thanks Jeff!)

[#3]

@Smart Solutions

STS is listening on 10015 even in secure cluster. Have you tried to start STS through ambari ?

16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab
16/05/24 02:03:32 INFO CompositeService: Operation log root directory is created: /tmp/hive/operation_logs
16/05/24 02:03:32 INFO AbstractService: HiveServer2: Async execution pool size 100
16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is inited.
16/05/24 02:03:32 INFO AbstractService: Service: SessionManager is inited.
16/05/24 02:03:32 INFO AbstractService: Service: CLIService is inited.
16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is inited.
16/05/24 02:03:32 INFO AbstractService: Service: HiveServer2 is inited.
16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is started.
16/05/24 02:03:32 INFO AbstractService: Service:SessionManager is started.
16/05/24 02:03:32 INFO AbstractService: Service:CLIService is started.
16/05/24 02:03:32 INFO ObjectStore: ObjectStore, initialize called
16/05/24 02:03:32 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
16/05/24 02:03:32 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/05/24 02:03:32 INFO ObjectStore: Initialized ObjectStore
16/05/24 02:03:32 INFO HiveMetaStore: 0: get_databases: default
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=get_databases: default
16/05/24 02:03:32 INFO HiveMetaStore: 0: Shutting down the object store...
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=Shutting down the object store...
16/05/24 02:03:32 INFO HiveMetaStore: 0: Metastore shutdown complete.
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=Metastore shutdown complete.
16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is started.
16/05/24 02:03:32 INFO AbstractService: Service:HiveServer2 is started.
16/05/24 02:03:32 INFO HiveThriftServer2: HiveThriftServer2 started
16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab
16/05/24 02:03:32 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=0
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
16/05/24 02:03:33 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=1
16/05/24 02:03:33 INFO ThriftCLIService: Starting ThriftBinaryCLIService on port 10015 with 5...500 worker threads

[#4]

@jzhang, Yes. I did try from Ambari also. The result is same.

[#5]

Could you check whether hive.server2.thrift.port is 10015 on the ambari UI ?

[#6]

Yes it is. 10015

[#7]

You can have both HS2 & Spark Thrift Server running on the same node.

To change the port of STS, modify hive.server2.thrift.port in advanced spark-hive-site-override.conf with Ambari

Bot HS2 & STS can listen in binary (thrift) & HTTP. Binary is the default.

I validated this with HDP 2.4.2.0-258

[#8]

@Smart Solutions Can you see if this solves your issue

[#9]

@vshukla

You suggesting to change hive.server2.thrift.port in advanced spark-hive-site-override.conf? It is already set to 10015.

1) Did you check on Kerberized cluser?

2) Try setting hive.server2.transport.mode to http .

[#10 · Accepted solution]

Confirmed with HWX that its a bug.Closing the issue for now.