Created 05-19-2016 11:58 AM
Hi Guys,
I am following the document https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/startin... which talks about setting up of STS and starting the service.
In our Keerberized cluster, after successful service addition, when I try to start the service, I observed that it fails.
16/05/19 10:27:00 INFO AbstractService: Service:HiveServer2 is started. 16/05/19 10:27:00 INFO HiveThriftServer2: HiveThriftServer2 started 16/05/19 10:27:00 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead. 16/05/19 10:27:00 INFO Server: jetty-8.y.z-SNAPSHOT 16/05/19 10:27:00 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:10001: java.net.BindException: Address already in use java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method)
When I observed the logs it says, port 10001 is already in use BindingException. In HiveServer2 I notice that for HTTP transport mode, bound port is 10001.
When I manually stopped HS2 on the same host and then started STS, then it works fine.
Here I've a few questions.
1) in the log it says that HS2 is started.. Is it trying to start HS2 ?
2) Port issue: Why still 10001 and not 10015?
For the STS, we have configured the port 10015 but when I try to connect using beeline : (After having a valid ticket)
beeline -u "jdbc:hive2://STSHOST:10015/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm" it fails but when I try to connect on port 10001 it works (previously I used to connect to HS2 using the same command).
beeline -u "jdbc:hive2://STSHOST:10001/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm"
And I can use STS to submit the sqls.
Could anyone please try to explain this behavior?
Can I have HS2 and STS running on the same node?
Thanks.
Tagging experts:
@vshukla, @Timothy Spann, @Jitendra Yadav, @Yuta Imai @Simon Elliston Ball
Created 05-27-2016 04:06 PM
Confirmed with HWX that its a bug.Closing the issue for now.
Created 05-24-2016 03:42 AM
@Smart Solutions please see note from @jzhang, below (thanks Jeff!)
Created 05-24-2016 02:26 AM
STS is listening on 10015 even in secure cluster. Have you tried to start STS through ambari ?
16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab 16/05/24 02:03:32 INFO CompositeService: Operation log root directory is created: /tmp/hive/operation_logs 16/05/24 02:03:32 INFO AbstractService: HiveServer2: Async execution pool size 100 16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is inited. 16/05/24 02:03:32 INFO AbstractService: Service: SessionManager is inited. 16/05/24 02:03:32 INFO AbstractService: Service: CLIService is inited. 16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is inited. 16/05/24 02:03:32 INFO AbstractService: Service: HiveServer2 is inited. 16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is started. 16/05/24 02:03:32 INFO AbstractService: Service:SessionManager is started. 16/05/24 02:03:32 INFO AbstractService: Service:CLIService is started. 16/05/24 02:03:32 INFO ObjectStore: ObjectStore, initialize called 16/05/24 02:03:32 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing 16/05/24 02:03:32 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY 16/05/24 02:03:32 INFO ObjectStore: Initialized ObjectStore 16/05/24 02:03:32 INFO HiveMetaStore: 0: get_databases: default 16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM ip=unknown-ip-addr cmd=get_databases: default 16/05/24 02:03:32 INFO HiveMetaStore: 0: Shutting down the object store... 16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM ip=unknown-ip-addr cmd=Shutting down the object store... 16/05/24 02:03:32 INFO HiveMetaStore: 0: Metastore shutdown complete. 16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM ip=unknown-ip-addr cmd=Metastore shutdown complete. 16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is started. 16/05/24 02:03:32 INFO AbstractService: Service:HiveServer2 is started. 16/05/24 02:03:32 INFO HiveThriftServer2: HiveThriftServer2 started 16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab 16/05/24 02:03:32 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens 16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=0 16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) 16/05/24 02:03:33 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens 16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=1 16/05/24 02:03:33 INFO ThriftCLIService: Starting ThriftBinaryCLIService on port 10015 with 5...500 worker threads
Created 05-24-2016 04:31 PM
@jzhang, Yes. I did try from Ambari also. The result is same.
Created 05-25-2016 12:31 AM
Could you check whether hive.server2.thrift.port is 10015 on the ambari UI ?
Created 05-27-2016 09:03 AM
Yes it is. 10015
Created 05-25-2016 12:13 AM
You can have both HS2 & Spark Thrift Server running on the same node.
To change the port of STS, modify hive.server2.thrift.port in advanced spark-hive-site-override.conf with Ambari
Bot HS2 & STS can listen in binary (thrift) & HTTP. Binary is the default.
I validated this with HDP 2.4.2.0-258
Created 05-25-2016 10:26 PM
@Smart Solutions Can you see if this solves your issue
Created 05-27-2016 09:09 AM
You suggesting to change hive.server2.thrift.port in advanced spark-hive-site-override.conf? It is already set to 10015.
1) Did you check on Kerberized cluser?
2) Try setting hive.server2.transport.mode to http .
Created 05-27-2016 04:06 PM
Confirmed with HWX that its a bug.Closing the issue for now.
Created 06-02-2016 03:13 AM
If you use http mode, please specify hive.server2.thrift.http.port to a different port.
Created 02-02-2017 07:35 PM
I am lil late to the party, but I was able to run both HS2 and STS on the same machine on a kerberized cluster using:
HS2 => hive.server2.thrift.http.port: 10001, transportmode: http,
STS => hive.server2.thrift.port:10015, hive.server2.transport.mode: binary
STS does not start and throws the bind exception when I use:
STS => hive.server2.thrift.port:10015, hive.server2.transport.mode: http
So changing transport mode for STS to binary works for me.
Tested on HDP 2.4.2.29-4
Regards
Rakesh
Created 02-02-2017 07:57 PM
And guessing from what is happening here I thought I should try starting up the thrift server in http mode on an http port.
but http port for thrift server is not defined anywhere by Ambari, so created an custom propery for thrift server: 'hive.server2.thrift.http.port: 10013'
and defined 'hive.server2.transport.mode: http'
And it started thrift server on a different port (10013) in http mode!
Regards
Rakesh
Created 09-11-2017 08:54 AM
@Rakesh Gupta Thank you very much. Smart debug sir. Saved me my day today.