If this is hdp 2.4 then I would suggest you to run HS2 on different port and let Spark run on 10001 as a workaround only. Meantime I will try to reproduce this issue on my HDP 2.4 cluster and open a bug for this if required.
Here is my env info.
sudo -u spark ./sbin/start-thriftserver.sh --master yarn-client --executor-memory 512m --hiveconf hive.server2.thrift.port=10015 [root@ey spark-thriftserver]# lsof -i:10015 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 29812 spark 193u IPv6 246416 0t0 TCP *:10015 (LISTEN) [root@ey spark-thriftserver]# lsof -i:10001 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 22905 hive 482u IPv4 139955 0t0 TCP *:scp-config (LISTEN)
Just wanted to check if you have hive-site.xml or some other .xml file inside /usr/hdp/current/spark-thriftserver/conf direcotry which has hive.server2.thrift.port property configured?
From what I understand, STS runs a version of HS2. The two processes need to use different ports. The default port used by Ambari for STS is 10015 (we need to revise the example in the document). The default port used by Ambari for HS2 is 10000. (See Configuring Ports in the HDP Ref. Guide for a list of ports per component.
As a side note, you mentioned that you configured the STS port 100015 but the first beeline command lists 10015. If that's a code mismatch (as opposed to a typo in the post), that would create a mismatch.
Accessing Spark SQL through JDBC in the Spark Guide has additional info about using the STS, and the Spark Guide has some Ambari-specific configuration information in an earlier chapter (Installing STS after Deploying Spark and Customizing the STS Port; links are for HDP 2.4.0).
Thanks for the links. And looking through the typo. Yes default ambari port for HS2 is 10000 and when it is Kerberized for http requests it listens on 10001.
I started the STS from the command line
./sbin/start-thriftserver.sh --master yarn-client --executor-memory 512m --hiveconf hive.server2.thrift.port=100015
I tried netstat and I do not see any thing on port 100015 . However, I notice that in the Thrift logs it says : ThriftCLIService started on port 10001. In the log I see following : 16/05/20 15:36:25 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:10001 16/05/20 15:36:25 INFO ThriftCLIService: Started ThriftHttpCLIService in http mode on port 10001 path=/cliservice/* with 5...500 worker threads
It says clearly that STS thrift is trying to start on port 10001.
I would be pleased if I can keep STS and HS2 on the same machine and different ports. I am confused here, because no where in the STS configuration I see 1001 !!!!