Support Questions
Find answers, ask questions, and share your expertise

Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

Solved Go to solution
Highlighted

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

Expert Contributor

@Smart Solutions I'll see if I can find out more info.

Highlighted

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

Expert Contributor

@Smart Solutions please see note from @jzhang, below (thanks Jeff!)

Highlighted

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

Expert Contributor
@Smart Solutions

STS is listening on 10015 even in secure cluster. Have you tried to start STS through ambari ?

16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab
16/05/24 02:03:32 INFO CompositeService: Operation log root directory is created: /tmp/hive/operation_logs
16/05/24 02:03:32 INFO AbstractService: HiveServer2: Async execution pool size 100
16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is inited.
16/05/24 02:03:32 INFO AbstractService: Service: SessionManager is inited.
16/05/24 02:03:32 INFO AbstractService: Service: CLIService is inited.
16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is inited.
16/05/24 02:03:32 INFO AbstractService: Service: HiveServer2 is inited.
16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is started.
16/05/24 02:03:32 INFO AbstractService: Service:SessionManager is started.
16/05/24 02:03:32 INFO AbstractService: Service:CLIService is started.
16/05/24 02:03:32 INFO ObjectStore: ObjectStore, initialize called
16/05/24 02:03:32 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
16/05/24 02:03:32 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/05/24 02:03:32 INFO ObjectStore: Initialized ObjectStore
16/05/24 02:03:32 INFO HiveMetaStore: 0: get_databases: default
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=get_databases: default
16/05/24 02:03:32 INFO HiveMetaStore: 0: Shutting down the object store...
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=Shutting down the object store...
16/05/24 02:03:32 INFO HiveMetaStore: 0: Metastore shutdown complete.
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=Metastore shutdown complete.
16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is started.
16/05/24 02:03:32 INFO AbstractService: Service:HiveServer2 is started.
16/05/24 02:03:32 INFO HiveThriftServer2: HiveThriftServer2 started
16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab
16/05/24 02:03:32 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=0
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
16/05/24 02:03:33 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=1
16/05/24 02:03:33 INFO ThriftCLIService: Starting ThriftBinaryCLIService on port 10015 with 5...500 worker threads
Highlighted

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

Expert Contributor

@jzhang, Yes. I did try from Ambari also. The result is same.

Highlighted

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

Expert Contributor

Could you check whether hive.server2.thrift.port is 10015 on the ambari UI ?

Highlighted

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

Expert Contributor

Yes it is. 10015

Highlighted

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

You can have both HS2 & Spark Thrift Server running on the same node.

To change the port of STS, modify hive.server2.thrift.port in advanced spark-hive-site-override.conf with Ambari

Bot HS2 & STS can listen in binary (thrift) & HTTP. Binary is the default.

I validated this with HDP 2.4.2.0-258

Highlighted

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

@Smart Solutions Can you see if this solves your issue

Highlighted

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

Expert Contributor
@vshukla

You suggesting to change hive.server2.thrift.port in advanced spark-hive-site-override.conf? It is already set to 10015.

1) Did you check on Kerberized cluser?

2) Try setting hive.server2.transport.mode to http .

Highlighted

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

Expert Contributor

Confirmed with HWX that its a bug.Closing the issue for now.

View solution in original post