question Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node? in Archives of Support Questions (Read Only)

Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

smartninja723 — Thu, 19 May 2016 18:58:02 GMT

Hi Guys,

I am following the document https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/starting_sts.html which talks about setting up of STS and starting the service.

In our Keerberized cluster, after successful service addition, when I try to start the service, I observed that it fails.

16/05/19 10:27:00 INFO AbstractService: Service:HiveServer2 is started.
16/05/19 10:27:00 INFO HiveThriftServer2: HiveThriftServer2 started
16/05/19 10:27:00 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
16/05/19 10:27:00 INFO Server: jetty-8.y.z-SNAPSHOT
16/05/19 10:27:00 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:10001: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)

When I observed the logs it says, port 10001 is already in use BindingException. In HiveServer2 I notice that for HTTP transport mode, bound port is 10001.

When I manually stopped HS2 on the same host and then started STS, then it works fine.

Here I've a few questions.

1) in the log it says that HS2 is started.. Is it trying to start HS2 ?

2) Port issue: Why still 10001 and not 10015?

For the STS, we have configured the port 10015 but when I try to connect using beeline : (After having a valid ticket)

beeline -u "jdbc:hive2://STSHOST:10015/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm" it fails but when I try to connect on port 10001 it works (previously I used to connect to HS2 using the same command).

beeline -u "jdbc:hive2://STSHOST:10001/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm"

And I can use STS to submit the sqls.

Could anyone please try to explain this behavior?

Can I have HS2 and STS running on the same node?

Thanks.

Tagging experts:

@vshukla, @Timothy Spann, @Jitendra Yadav, @Yuta Imai @Simon Elliston Ball

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

rajkumar_singh — Thu, 19 May 2016 18:59:14 GMT

you can but you need to change the port of one of the service.

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

krajguru — Thu, 19 May 2016 19:11:00 GMT

@Smart Solutions

How are you starting the spark thrift server, what is the command you are using ?

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

smartninja723 — Thu, 19 May 2016 19:28:19 GMT

Using Ambari,

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

smartninja723 — Thu, 19 May 2016 19:29:06 GMT

Can you be precise, which property to be changed? I do not want to change for HS2.

What I think is : Internally STS uses HS2 and not sure if it inherits port configurations of HS2.

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

smartninja723 — Thu, 19 May 2016 19:31:20 GMT

Do you suggest to use command to start STS ?

./sbin/start-thriftserver.sh --master yarn-client --executor-memory 512m --hiveconf hive.server2.thrift.port=100015

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

jyadav — Thu, 19 May 2016 19:34:09 GMT

@Smart Solutions

I don't think we can run multiple instances of spark thrift from Ambari, Its better to run one instance from Ambari and another from command line.

cd $SPARK_HOME

./sbin/start-thriftserver.sh --master yarn-client --executor-memory 512m --hiveconf hive.server2.thrift.port=100015 &

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

smartninja723 — Thu, 19 May 2016 19:40:00 GMT

@Jitendra Yadav

Its only one instance for STS and one for HS2.

In the Ambari Configuration : hive.server2.thrift.port has been set to 100015 already.

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

jyadav — Thu, 19 May 2016 19:59:39 GMT

@Smart Solutions

which HDP version you are using? there was a bug related to same in HDP 2.3

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

smartninja723 — Thu, 19 May 2016 20:23:40 GMT

@Jitendra Yadav

HDP-2.4.0.0

(2.4.0.0-169)

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

jyadav — Fri, 20 May 2016 21:56:22 GMT

@Smart Solutions

If this is hdp 2.4 then I would suggest you to run HS2 on different port and let Spark run on 10001 as a workaround only. Meantime I will try to reproduce this issue on my HDP 2.4 cluster and open a bug for this if required.

Thanks

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

lgeorge — Sat, 21 May 2016 05:18:38 GMT

From what I understand, STS runs a version of HS2. The two processes need to use different ports. The default port used by Ambari for STS is 10015 (we need to revise the example in the document). The default port used by Ambari for HS2 is 10000. (See Configuring Ports in the HDP Ref. Guide for a list of ports per component.

As a side note, you mentioned that you configured the STS port 100015 but the first beeline command lists 10015. If that's a code mismatch (as opposed to a typo in the post), that would create a mismatch.

Accessing Spark SQL through JDBC in the Spark Guide has additional info about using the STS, and the Spark Guide has some Ambari-specific configuration information in an earlier chapter (Installing STS after Deploying Spark and Customizing the STS Port; links are for HDP 2.4.0).

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

smartninja723 — Mon, 23 May 2016 15:21:52 GMT

@Jitendra Yadav

Did you verify if its a bug? Or any work around.

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

smartninja723 — Mon, 23 May 2016 15:36:16 GMT

@lgeorge

Thanks for the links. And looking through the typo. Yes default ambari port for HS2 is 10000 and when it is Kerberized for http requests it listens on 10001.

I started the STS from the command line

./sbin/start-thriftserver.sh --master yarn-client --executor-memory 512m --hiveconf hive.server2.thrift.port=100015

I tried netstat and I do not see any thing on port 100015 . However, I notice that in the Thrift logs it says : ThriftCLIService started on port 10001. In the log I see following : 16/05/20 15:36:25 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:10001 16/05/20 15:36:25 INFO ThriftCLIService: Started ThriftHttpCLIService in http mode on port 10001 path=/cliservice/* with 5...500 worker threads

It says clearly that STS thrift is trying to start on port 10001.

I would be pleased if I can keep STS and HS2 on the same machine and different ports. I am confused here, because no where in the STS configuration I see 1001 !!!!

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

lgeorge — Mon, 23 May 2016 23:22:17 GMT

@Smart Solutions I'll see if I can find out more info.

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

jyadav — Tue, 24 May 2016 04:31:05 GMT

@Smart Solutions

I'm not able to reproduce this issue on my cluster, But as a workaround you can move the HS2 http port from 10001 to 10002 and let spark occupied the 10001 port.

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

jyadav — Tue, 24 May 2016 05:41:11 GMT

@Smart Solutions

Here is my env info.

sudo -u spark ./sbin/start-thriftserver.sh --master yarn-client --executor-memory 512m --hiveconf hive.server2.thrift.port=10015 
[root@ey spark-thriftserver]# lsof -i:10015
COMMAND   PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    29812 spark  193u  IPv6 246416      0t0  TCP *:10015 (LISTEN)
[root@ey spark-thriftserver]# lsof -i:10001
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    22905 hive  482u  IPv4 139955      0t0  TCP *:scp-config (LISTEN)

Just wanted to check if you have hive-site.xml or some other .xml file inside /usr/hdp/current/spark-thriftserver/conf direcotry which has hive.server2.thrift.port property configured?

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

jzhang — Tue, 24 May 2016 09:26:13 GMT

@Smart Solutions

STS is listening on 10015 even in secure cluster. Have you tried to start STS through ambari ?

16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab
16/05/24 02:03:32 INFO CompositeService: Operation log root directory is created: /tmp/hive/operation_logs
16/05/24 02:03:32 INFO AbstractService: HiveServer2: Async execution pool size 100
16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is inited.
16/05/24 02:03:32 INFO AbstractService: Service: SessionManager is inited.
16/05/24 02:03:32 INFO AbstractService: Service: CLIService is inited.
16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is inited.
16/05/24 02:03:32 INFO AbstractService: Service: HiveServer2 is inited.
16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is started.
16/05/24 02:03:32 INFO AbstractService: Service:SessionManager is started.
16/05/24 02:03:32 INFO AbstractService: Service:CLIService is started.
16/05/24 02:03:32 INFO ObjectStore: ObjectStore, initialize called
16/05/24 02:03:32 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing
16/05/24 02:03:32 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
16/05/24 02:03:32 INFO ObjectStore: Initialized ObjectStore
16/05/24 02:03:32 INFO HiveMetaStore: 0: get_databases: default
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=get_databases: default
16/05/24 02:03:32 INFO HiveMetaStore: 0: Shutting down the object store...
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=Shutting down the object store...
16/05/24 02:03:32 INFO HiveMetaStore: 0: Metastore shutdown complete.
16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM        ip=unknown-ip-addr      cmd=Metastore shutdown complete.
16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is started.
16/05/24 02:03:32 INFO AbstractService: Service:HiveServer2 is started.
16/05/24 02:03:32 INFO HiveThriftServer2: HiveThriftServer2 started
16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab
16/05/24 02:03:32 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=0
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
16/05/24 02:03:33 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=1
16/05/24 02:03:33 INFO ThriftCLIService: Starting ThriftBinaryCLIService on port 10015 with 5...500 worker threads

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

lgeorge — Tue, 24 May 2016 10:42:41 GMT

@Smart Solutions please see note from @jzhang, below (thanks Jeff!)

Re: Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

smartninja723 — Tue, 24 May 2016 23:31:28 GMT

@jzhang, Yes. I did try from Ambari also. The result is same.