Created 05-19-2016 11:58 AM
Hi Guys,
I am following the document https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/startin... which talks about setting up of STS and starting the service.
In our Keerberized cluster, after successful service addition, when I try to start the service, I observed that it fails.
16/05/19 10:27:00 INFO AbstractService: Service:HiveServer2 is started. 16/05/19 10:27:00 INFO HiveThriftServer2: HiveThriftServer2 started 16/05/19 10:27:00 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead. 16/05/19 10:27:00 INFO Server: jetty-8.y.z-SNAPSHOT 16/05/19 10:27:00 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:10001: java.net.BindException: Address already in use java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method)
When I observed the logs it says, port 10001 is already in use BindingException. In HiveServer2 I notice that for HTTP transport mode, bound port is 10001.
When I manually stopped HS2 on the same host and then started STS, then it works fine.
Here I've a few questions.
1) in the log it says that HS2 is started.. Is it trying to start HS2 ?
2) Port issue: Why still 10001 and not 10015?
For the STS, we have configured the port 10015 but when I try to connect using beeline : (After having a valid ticket)
beeline -u "jdbc:hive2://STSHOST:10015/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm" it fails but when I try to connect on port 10001 it works (previously I used to connect to HS2 using the same command).
beeline -u "jdbc:hive2://STSHOST:10001/default;httpPath=cliservice;transportMode=http;principal=hive/_HOST@Realm"
And I can use STS to submit the sqls.
Could anyone please try to explain this behavior?
Can I have HS2 and STS running on the same node?
Thanks.
Tagging experts:
@vshukla, @Timothy Spann, @Jitendra Yadav, @Yuta Imai @Simon Elliston Ball
Created 05-27-2016 04:06 PM
Confirmed with HWX that its a bug.Closing the issue for now.
Created 05-23-2016 04:22 PM
@Smart Solutions I'll see if I can find out more info.
Created 05-24-2016 03:42 AM
@Smart Solutions please see note from @jzhang, below (thanks Jeff!)
Created 05-24-2016 02:26 AM
STS is listening on 10015 even in secure cluster. Have you tried to start STS through ambari ?
16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab 16/05/24 02:03:32 INFO CompositeService: Operation log root directory is created: /tmp/hive/operation_logs 16/05/24 02:03:32 INFO AbstractService: HiveServer2: Async execution pool size 100 16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is inited. 16/05/24 02:03:32 INFO AbstractService: Service: SessionManager is inited. 16/05/24 02:03:32 INFO AbstractService: Service: CLIService is inited. 16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is inited. 16/05/24 02:03:32 INFO AbstractService: Service: HiveServer2 is inited. 16/05/24 02:03:32 INFO AbstractService: Service:OperationManager is started. 16/05/24 02:03:32 INFO AbstractService: Service:SessionManager is started. 16/05/24 02:03:32 INFO AbstractService: Service:CLIService is started. 16/05/24 02:03:32 INFO ObjectStore: ObjectStore, initialize called 16/05/24 02:03:32 INFO Query: Reading in results for query "org.datanucleus.store.rdbms.query.SQLQuery@0" since the connection used is closing 16/05/24 02:03:32 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY 16/05/24 02:03:32 INFO ObjectStore: Initialized ObjectStore 16/05/24 02:03:32 INFO HiveMetaStore: 0: get_databases: default 16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM ip=unknown-ip-addr cmd=get_databases: default 16/05/24 02:03:32 INFO HiveMetaStore: 0: Shutting down the object store... 16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM ip=unknown-ip-addr cmd=Shutting down the object store... 16/05/24 02:03:32 INFO HiveMetaStore: 0: Metastore shutdown complete. 16/05/24 02:03:32 INFO audit: ugi=hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM ip=unknown-ip-addr cmd=Metastore shutdown complete. 16/05/24 02:03:32 INFO AbstractService: Service:ThriftBinaryCLIService is started. 16/05/24 02:03:32 INFO AbstractService: Service:HiveServer2 is started. 16/05/24 02:03:32 INFO HiveThriftServer2: HiveThriftServer2 started 16/05/24 02:03:32 INFO UserGroupInformation: Login successful for user hive/jzhang-test4-1.openstacklocal@EXAMPLE.COM using keytab file /etc/security/keytabs/hive.service.keytab 16/05/24 02:03:32 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens 16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=0 16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) 16/05/24 02:03:33 INFO AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens 16/05/24 02:03:33 INFO TokenStoreDelegationTokenSecretManager: New master key with key id=1 16/05/24 02:03:33 INFO ThriftCLIService: Starting ThriftBinaryCLIService on port 10015 with 5...500 worker threads
Created 05-24-2016 04:31 PM
@jzhang, Yes. I did try from Ambari also. The result is same.
Created 05-25-2016 12:31 AM
Could you check whether hive.server2.thrift.port is 10015 on the ambari UI ?
Created 05-27-2016 09:03 AM
Yes it is. 10015
Created 05-25-2016 12:13 AM
You can have both HS2 & Spark Thrift Server running on the same node.
To change the port of STS, modify hive.server2.thrift.port in advanced spark-hive-site-override.conf with Ambari
Bot HS2 & STS can listen in binary (thrift) & HTTP. Binary is the default.
I validated this with HDP 2.4.2.0-258
Created 05-25-2016 10:26 PM
@Smart Solutions Can you see if this solves your issue
Created 05-27-2016 09:09 AM
You suggesting to change hive.server2.thrift.port in advanced spark-hive-site-override.conf? It is already set to 10015.
1) Did you check on Kerberized cluser?
2) Try setting hive.server2.transport.mode to http .
Created 05-27-2016 04:06 PM
Confirmed with HWX that its a bug.Closing the issue for now.