Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

[#1]

Hi Guys,

I am following the document https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/startin... which talks about setting up of STS and starting the service.

In our Keerberized cluster, after successful service addition, when I try to start the service, I observed that it fails.

16/05/19 10:27:00 INFO AbstractService: Service:HiveServer2 is started.
16/05/19 10:27:00 INFO HiveThriftServer2: HiveThriftServer2 started
16/05/19 10:27:00 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
16/05/19 10:27:00 INFO Server: jetty-8.y.z-SNAPSHOT
16/05/19 10:27:00 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:10001: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)

When I observed the logs it says, port 10001 is already in use BindingException. In HiveServer2 I notice that for HTTP transport mode, bound port is 10001.

When I manually stopped HS2 on the same host and then started STS, then it works fine.

Here I've a few questions.

1) in the log it says that HS2 is started.. Is it trying to start HS2 ?

2) Port issue: Why still 10001 and not 10015?

For the STS, we have configured the port 10015 but when I try to connect using beeline : (After having a valid ticket)

beeline -u "jdbc:hive2://STSHOST:10015/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm" it fails but when I try to connect on port 10001 it works (previously I used to connect to HS2 using the same command).

beeline -u "jdbc:hive2://STSHOST:10001/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm"

And I can use STS to submit the sqls.

Could anyone please try to explain this behavior?

Can I have HS2 and STS running on the same node?

Thanks.

Tagging experts:

@vshukla, @Timothy Spann, @Jitendra Yadav, @Yuta Imai @Simon Elliston Ball

[#2]

@Smart Solutions

If this is hdp 2.4 then I would suggest you to run HS2 on different port and let Spark run on 10001 as a workaround only. Meantime I will try to reproduce this issue on my HDP 2.4 cluster and open a bug for this if required.

Thanks

[#3]

@Jitendra Yadav

Did you verify if its a bug? Or any work around.

[#4]

@Smart Solutions

I'm not able to reproduce this issue on my cluster, But as a workaround you can move the HS2 http port from 10001 to 10002 and let spark occupied the 10001 port.

[#5]

@Smart Solutions

Here is my env info.

sudo -u spark ./sbin/start-thriftserver.sh --master yarn-client --executor-memory 512m --hiveconf hive.server2.thrift.port=10015 
[root@ey spark-thriftserver]# lsof -i:10015
COMMAND   PID  USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    29812 spark  193u  IPv6 246416      0t0  TCP *:10015 (LISTEN)
[root@ey spark-thriftserver]# lsof -i:10001
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
java    22905 hive  482u  IPv4 139955      0t0  TCP *:scp-config (LISTEN)

Just wanted to check if you have hive-site.xml or some other .xml file inside /usr/hdp/current/spark-thriftserver/conf direcotry which has hive.server2.thrift.port property configured?

[#6]

@Smart Solutions

Did you tried the workaround?

[#7]

@Smart Solutions This seems a bug with HDP 2.4.0 with Kerberos + hive HTTP mode enabled, I'm able to reproduce this locally. Please contact HWX official support for possible fix.

[#8]

@Jitendra Yadav : HS2, we are already using in production hence can't move to different port. However is it possible to move port for STS?

[#9]

@Jitendra YadavThanks for confirming. Have raised a case with HWX.

[#10]

From what I understand, STS runs a version of HS2. The two processes need to use different ports. The default port used by Ambari for STS is 10015 (we need to revise the example in the document). The default port used by Ambari for HS2 is 10000. (See Configuring Ports in the HDP Ref. Guide for a list of ports per component.

As a side note, you mentioned that you configured the STS port 100015 but the first beeline command lists 10015. If that's a code mismatch (as opposed to a typo in the post), that would create a mismatch.

Accessing Spark SQL through JDBC in the Spark Guide has additional info about using the STS, and the Spark Guide has some Ambari-specific configuration information in an earlier chapter (Installing STS after Deploying Spark and Customizing the STS Port; links are for HDP 2.4.0).

[#11]

@lgeorge

Thanks for the links. And looking through the typo. Yes default ambari port for HS2 is 10000 and when it is Kerberized for http requests it listens on 10001.

I started the STS from the command line

./sbin/start-thriftserver.sh --master yarn-client --executor-memory 512m --hiveconf hive.server2.thrift.port=100015 

I tried netstat and I do not see any thing on port 100015 . However, I notice that in the Thrift logs it says : ThriftCLIService started on port 10001. In the log I see following : 16/05/20 15:36:25 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:10001 16/05/20 15:36:25 INFO ThriftCLIService: Started ThriftHttpCLIService in http mode on port 10001 path=/cliservice/* with 5...500 worker threads

It says clearly that STS thrift is trying to start on port 10001.

I would be pleased if I can keep STS and HS2 on the same machine and different ports. I am confused here, because no where in the STS configuration I see 1001 !!!!