Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Can we not have HS2 and Spark Thrift Server (STS) running on the same node?

avatar
Expert Contributor

Hi Guys,

I am following the document https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/startin... which talks about setting up of STS and starting the service.

In our Keerberized cluster, after successful service addition, when I try to start the service, I observed that it fails.

16/05/19 10:27:00 INFO AbstractService: Service:HiveServer2 is started.
16/05/19 10:27:00 INFO HiveThriftServer2: HiveThriftServer2 started
16/05/19 10:27:00 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
16/05/19 10:27:00 INFO Server: jetty-8.y.z-SNAPSHOT
16/05/19 10:27:00 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:10001: java.net.BindException: Address already in use
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)

When I observed the logs it says, port 10001 is already in use BindingException. In HiveServer2 I notice that for HTTP transport mode, bound port is 10001.

When I manually stopped HS2 on the same host and then started STS, then it works fine.

Here I've a few questions.

1) in the log it says that HS2 is started.. Is it trying to start HS2 ?

2) Port issue: Why still 10001 and not 10015?

For the STS, we have configured the port 10015 but when I try to connect using beeline : (After having a valid ticket)

beeline -u "jdbc:hive2://STSHOST:10015/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm" it fails but when I try to connect on port 10001 it works (previously I used to connect to HS2 using the same command).

beeline -u "jdbc:hive2://STSHOST:10001/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm"

And I can use STS to submit the sqls.

Could anyone please try to explain this behavior?

Can I have HS2 and STS running on the same node?

Thanks.

Tagging experts:

@vshukla, @Timothy Spann, @Jitendra Yadav, @Yuta Imai @Simon Elliston Ball

1 ACCEPTED SOLUTION

avatar
Expert Contributor

Confirmed with HWX that its a bug.Closing the issue for now.

View solution in original post

33 REPLIES 33

avatar
Super Guru

you can but you need to change the port of one of the service.

avatar
Expert Contributor

Can you be precise, which property to be changed? I do not want to change for HS2.

What I think is : Internally STS uses HS2 and not sure if it inherits port configurations of HS2.

avatar
Expert Contributor

@Smart Solutions

How are you starting the spark thrift server, what is the command you are using ?

avatar
Expert Contributor

Using Ambari,

avatar
Expert Contributor

Do you suggest to use command to start STS ?

./sbin/start-thriftserver.sh --master yarn-client --executor-memory 512m --hiveconf hive.server2.thrift.port=100015

avatar
Super Guru

@Smart Solutions

I don't think we can run multiple instances of spark thrift from Ambari, Its better to run one instance from Ambari and another from command line.

cd $SPARK_HOME

./sbin/start-thriftserver.sh --master yarn-client --executor-memory 512m --hiveconf hive.server2.thrift.port=100015 &

avatar
Expert Contributor
@Jitendra Yadav

Its only one instance for STS and one for HS2.

In the Ambari Configuration : hive.server2.thrift.port has been set to 100015 already.

avatar
Super Guru
@Smart Solutions

which HDP version you are using? there was a bug related to same in HDP 2.3

avatar
Expert Contributor
@Jitendra Yadav

HDP-2.4.0.0

(2.4.0.0-169)