Support Questions
Find answers, ask questions, and share your expertise

Phoenix connection failure

Highlighted

Phoenix connection failure

New Contributor

HDP Version: 3.0.0

env: Phoenix+Kerberos

classOf[org.apache.phoenix.jdbc.PhoenixDriver]
import java.sql.DriverManager
val conn = DriverManager.getConnection("jdbc:phoenix:dev-dmp3.fengdai.org,dev-dmp4.fengdai.org,dev-dmp5.fengdai.org:2181:/hbase-secure:" + classPath + "hbase.headless.keytab:hbase-dev_dmp@DEVDIP.ORG")

produce error

Exception in thread "main" org.apache.phoenix.exception.PhoenixIOException: callTimeout=1200000, callDuration=1329436: Failed after attempts=16, exceptions:
Sun Dec 16 16:12:45 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:45 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:45 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:46 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:46 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:47 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:49 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:53 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:03 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:13 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:23 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:33 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:53 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:14:13 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:14:34 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:14:54 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
 
	at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:138)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1204)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.createTable(ConnectionQueryServicesImpl.java:1501)
	at org.apache.phoenix.schema.MetaDataClient.createTableInternal(MetaDataClient.java:2721)
	at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:1114)
	at org.apache.phoenix.compile.CreateTableCompiler$1.execute(CreateTableCompiler.java:192)
	at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:408)
	at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:391)
	at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)
	at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:390)
	at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:378)
	at org.apache.phoenix.jdbc.PhoenixStatement.executeUpdate(PhoenixStatement.java:1806)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2569)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl$12.call(ConnectionQueryServicesImpl.java:2532)
	at org.apache.phoenix.util.PhoenixContextExecutor.call(PhoenixContextExecutor.java:76)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.init(ConnectionQueryServicesImpl.java:2532)
	at org.apache.phoenix.jdbc.PhoenixDriver.getConnectionQueryServices(PhoenixDriver.java:255)
	at org.apache.phoenix.jdbc.PhoenixEmbeddedDriver.createConnection(PhoenixEmbeddedDriver.java:150)
	at org.apache.phoenix.jdbc.PhoenixDriver.connect(PhoenixDriver.java:221)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:270)
	at com.tairanchina.csp.dmp.examples.kerberos.phoenix.PhoenixZKMain$.main(PhoenixZKMain.scala:39)
	at com.tairanchina.csp.dmp.examples.kerberos.phoenix.PhoenixZKMain.main(PhoenixZKMain.scala)
Caused by: java.net.SocketTimeoutException: callTimeout=1200000, callDuration=1329436: Failed after attempts=16, exceptions:
Sun Dec 16 16:12:45 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:45 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:45 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:46 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:46 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:47 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:49 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:53 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:03 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:13 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:23 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:33 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:53 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:14:13 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:14:34 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:14:54 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
 
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:159)
	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3055)
	at org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3047)
	at org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:434)
	at org.apache.phoenix.query.ConnectionQueryServicesImpl.ensureTableCreated(ConnectionQueryServicesImpl.java:1105)
	... 21 more
Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=16, exceptions:
Sun Dec 16 16:12:45 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:45 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:45 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:46 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:46 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:47 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:49 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:12:53 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:03 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:13 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:23 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:33 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:13:53 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:14:13 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:14:34 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
Sun Dec 16 16:14:54 CST 2018, RpcRetryingCaller{globalStartTime=1544947965386, pause=100, maxAttempts=16}, java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed


	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:386)
	at org.apache.hadoop.hbase.client.HTable.get(HTable.java:360)
	at org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1076)
	at org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:399)
	at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:437)
	at org.apache.hadoop.hbase.client.HBaseAdmin$6.rpcCall(HBaseAdmin.java:434)
	at org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:58)
	at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:107)
	... 25 more
Caused by: java.io.IOException: Call to dev-dmp4.fengdai.org/10.200.168.6:16020 failed on local exception: java.io.IOException: Connection closed
	at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:180)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410)
	at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406)
	at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
	at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
	at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:202)
	at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelInactive(NettyRpcDuplexHandler.java:210)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
	at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:377)
	at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:342)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
	at org.apache.hbase.thirdparty.io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:75)
	at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:277)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:224)
	at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1354)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:231)
	at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:917)
	at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:822)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
	at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
	at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Connection closed
	... 25 more

how to fix this problem please.

1 REPLY 1
Highlighted

Re: Phoenix connection failure

Explorer

Hi @jason

Are you using correct jars, Please check the below for reference.

https://streever.atlassian.net/wiki/spaces/HADOOP/pages/11993090/Phoenix+JDBC+Client+Setup