Created 11-03-2016 06:57 AM
0favorite | If anybody have done Spark Streaming 1.6.1 (Scala) with Phoenix with Kerberos enabled. Please share some sample code. I am using HDP 2.4.2 |
Created 11-03-2016 07:08 AM
I think this test case in Phoenix code help you to write scala code.
Make sure you suffix principle and keytab file to url. For ex:
jdbc:phoenix:<<em>Zookeeper_host_name</em>>:<<em>port_number</em>>:<<em>secured_Zookeeper_node</em>>:<<em>principal_name</em>>:<<em>HBase_headless_keytab_file</em>>
can get more info here:
http://phoenix.apache.org/phoenix_spark.html
You can validate phoenix connection with kerberos as below
Hope it helps.
Created 11-03-2016 07:18 AM
Thank You @Rajeshbabu, I more help where can I find phoenix driver to run this jdbc code on hortonworks 2.4.2
Created 11-03-2016 09:29 AM
In phoenix-client jar you can see phoenix driver class
org.apache.phoenix.jdbc.PhoenixDriver
Created 11-03-2016 07:27 AM
One more doubt I have is .. we specify keytab location in JDBC url, so when yarn will allocate executor for spark does this jdbc url will look locallly for the keytab file on each executor.
Created 11-03-2016 09:05 AM
Instead of specifying keytab location in the jdbc url you can try this. This might help you.
https://community.hortonworks.com/questions/56848/spark-cant-connect-to-secure-phoenix.html
Created 11-04-2016 12:33 PM
I am getting below exception now
client-ip/192.168.212.100:16000 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to client-ip/192.168.212.100:16000 is closing. Call id=33, waitTime=2 Fri Nov 04 12:48:00 CET 2016, RpcRetryingCaller{globalStartTime=1478259550578, pause=100, retries=35}, org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to client-ip/192.168.212.100:16000 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to client-ip/192.168.212.100:16000 is closing. Call id=34, waitTime=3 at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:4083) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:528) at org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:550) at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:810) ... 36 more Caused by: org.apache.hadoop.hbase.MasterNotRunningException: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to client-ip/192.168.212.100:16000 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to client-ip/192.168.212.100:16000 is closing. Call id=34, waitTime=3 at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1540) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.makeStub(ConnectionManager.java:1560) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.getKeepAliveMasterService(ConnectionManager.java:1711) at org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:38) at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:124) ... 40 more Caused by: com.google.protobuf.ServiceException: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to client-ip/192.168.212.100:16000 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to client-ip/192.168.212.100:16000 is closing. Call id=34, waitTime=3 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:223) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287) at org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$BlockingStub.isMasterRunning(MasterProtos.java:58152) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$MasterServiceStubMaker.isMasterRunning(ConnectionManager.java:1571) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStubNoRetries(ConnectionManager.java:1509) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation$StubMaker.makeStub(ConnectionManager.java:1531) ... 44 more Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to client-ip/192.168.212.100:16000 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to client-ip/192.168.212.100:16000 is closing. Call id=34, waitTime=3 at org.apache.hadoop.hbase.ipc.RpcClientImpl.wrapException(RpcClientImpl.java:1259) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1230) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213) ... 49 more Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to client-ip/192.168.212.100:16000 is closing. Call id=34, waitTime=3 at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1047) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:846) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:574) 16/11/04 12:48:01 INFO CoarseGrainedExecutorBackend: Got assigned task 76 16/11/04 12:48:01 INFO Executor: Running task 0.1 in stage 7.0 (TID 76) 16/11/04 12:48:01 INFO BlockManager: Found block input-0-1478259544400 locally 16/11/04 12:48:01 INFO deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 16/11/04 12:48:01 INFO deprecation: io.bytes.per.checksum is deprecated. Instead, use dfs.bytes-per-checksum 16/11/04 12:48:01 INFO RecoverableZooKeeper: Process identifier=hconnection-0x45e8f7b3 connecting to ZooKeeper ensemble=zk-ip:2181 16/11/04 12:48:01 INFO ZooKeeper: Initiating client connection, connectString=zk-ip:2181 sessionTimeout=90000 watcher=hconnection-0x45e8f7b30x0, quorum=zk-ip:2181, baseZNode=/hbase-secure 16/11/04 12:48:01 WARN ClientCnxn: SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: './key.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 16/11/04 12:48:01 INFO ClientCnxn: Opening socket connection to server client-ip/192.168.212.104:2181 16/11/04 12:48:01 INFO ClientCnxn: Socket connection established to client-ip/192.168.212.104:2181, initiating session 16/11/04 12:48:01 INFO ClientCnxn: Session establishment complete on server client-ip/192.168.212.104:2181, sessionid = 0x3574704ea4103d4, negotiated timeout = 40000 16/11/04 12:48:01 INFO RecoverableZooKeeper: Process identifier=hconnection-0x27ab4ba3 connecting to ZooKeeper ensemble=zk-ip:2181 16/11/04 12:48:01 INFO ZooKeeper: Initiating client connection, connectString=zk-ip:2181 sessionTimeout=90000 watcher=hconnection-0x27ab4ba30x0, quorum=zk-ip:2181, baseZNode=/hbase-secure 16/11/04 12:48:01 WARN ClientCnxn: SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: './key.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it. 16/11/04 12:48:01 INFO ClientCnxn: Opening socket connection to server client-ip/192.168.212.104:2181 16/11/04 12:48:01 INFO ClientCnxn: Socket connection established to client-ip/192.168.212.104:2181, initiating session 16/11/04 12:48:01 INFO ClientCnxn: Session establishment complete on server client-ip/192.168.212.104:2181, sessionid = 0x3574704ea4103d5, negotiated timeout = 40000