I am using hdp 2.6. And I am trying to fetch data from phoenix. I have tried follwoing link.;- https://community.hortonworks.com/questions/60413/hbase-master-and-regionserver-goes-down-citing-lea.html https://community.hortonworks.com/questions/41122/during-an-import-of-hbase-using-importtsv-hdfs-is.html Getting following error. ERROR SparkSQLDriver: Failed in [select * from rasdb.dim_account] java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.spark.sql.types.Decimal$.createUnsafe(Decimal.scala:456) at org.apache.spark.sql.types.Decimal.createUnsafe(Decimal.scala) at org.apache.spark.sql.catalyst.expressions.UnsafeRow.getDecimal(UnsafeRow.java:404) at org.apache.spark.sql.catalyst.CatalystTypeConverters$DecimalConverter.toScalaImpl(CatalystTypeConverters.scala:324) at org.apache.spark.sql.catalyst.CatalystTypeConverters$DecimalConverter.toScalaImpl(CatalystTypeConverters.scala:304) at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toScala(CatalystTypeConverters.scala:111) at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:264) at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:231) at org.apache.spark.sql.catalyst.CatalystTypeConverters$anonfun$createToScalaConverter$2.apply(CatalystTypeConverters.scala:396) at org.apache.spark.sql.execution.SparkPlan$anonfun$executeCollectPublic$1.apply(SparkPlan.scala:298) at org.apache.spark.sql.execution.SparkPlan$anonfun$executeCollectPublic$1.apply(SparkPlan.scala:298) at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) at org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:298) at org.apache.spark.sql.execution.QueryExecution$anonfun$hiveResultString$4.apply(QueryExecution.scala:139) at org.apache.spark.sql.execution.QueryExecution$anonfun$hiveResultString$4.apply(QueryExecution.scala:138) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57) at org.apache.spark.sql.execution.QueryExecution.hiveResultString(QueryExecution.scala:138) at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:63) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:335) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:247) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:751) 17/12/12 05:14:49 INFO ClientCnxn: Client session timed out, have not heard from server in 48847ms for sessionid 0x2604a1e0a641d82, closing socket connection and attempting reconnect 17/12/12 05:14:49 INFO ClientCnxn: Client session timed out, have not heard from server in 47326ms for sessionid 0x3604a1e09c01d5f, closing socket connection and attempting reconnect 17/12/12 05:14:49 INFO ClientCnxn: Client session timed out, have not heard from server in 55591ms for sessionid 0x2604a1e0a641d85, closing socket connection and attempting reconnect 17/12/12 05:14:49 INFO ContextCleaner: Cleaned accumulator 49 17/12/12 05:14:49 INFO BlockManagerInfo: Removed broadcast_2_piece0 on 192.168.180.54:44265 in memory (size: 33.4 KB, free: 409.5 MB) java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.spark.sql.types.Decimal$.createUnsafe(Decimal.scala:456) at org.apache.spark.sql.types.Decimal.createUnsafe(Decimal.scala) at org.apache.spark.sql.catalyst.expressions.UnsafeRow.getDecimal(UnsafeRow.java:404) at org.apache.spark.sql.catalyst.CatalystTypeConverters$DecimalConverter.toScalaImpl(CatalystTypeConverters.scala:324) at org.apache.spark.sql.catalyst.CatalystTypeConverters$DecimalConverter.toScalaImpl(CatalystTypeConverters.scala:304) at org.apache.spark.sql.catalyst.CatalystTypeConverters$CatalystTypeConverter.toScala(CatalystTypeConverters.scala:111) at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:264) at org.apache.spark.sql.catalyst.CatalystTypeConverters$StructConverter.toScala(CatalystTypeConverters.scala:231) at org.apache.spark.sql.catalyst.CatalystTypeConverters$anonfun$createToScalaConverter$2.apply(CatalystTypeConverters.scala:396) at org.apache.spark.sql.execution.SparkPlan$anonfun$executeCollectPublic$1.apply(SparkPlan.scala:298) at org.apache.spark.sql.execution.SparkPlan$anonfun$executeCollectPublic$1.apply(SparkPlan.scala:298) at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) at org.apache.spark.sql.execution.SparkPlan.executeCollectPublic(SparkPlan.scala:298) at org.apache.spark.sql.execution.QueryExecution$anonfun$hiveResultString$4.apply(QueryExecution.scala:139) at org.apache.spark.sql.execution.QueryExecution$anonfun$hiveResultString$4.apply(QueryExecution.scala:138) at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57) at org.apache.spark.sql.execution.QueryExecution.hiveResultString(QueryExecution.scala:138) at org.apache.spark.sql.hive.thriftserver.SparkSQLDriver.run(SparkSQLDriver.scala:63) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.processCmd(SparkSQLCLIDriver.scala:335) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:247) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:751) spark-sql> 17/12/12 05:14:49 WARN NettyRpcEnv: Ignored failure: java.util.concurrent.TimeoutException: Cannot receive any reply in 10 seconds 17/12/12 05:14:49 WARN NettyRpcEndpointRef: Error sending message [message = Heartbeat(driver,[Lscala.Tuple2;@5dd1e183,BlockManagerId(driver xxx, 44265, None))] in 1 attempts org.apache.spark.rpc.RpcTimeoutException: Futures timed out after [10 seconds]. This timeout is controlled by spark.executor.heartbeatInterval at org.apache.spark.rpc.RpcTimeout.org$apache$spark$rpc$RpcTimeout$createRpcTimeoutException(RpcTimeout.scala:48) at org.apache.spark.rpc.RpcTimeout$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:63) at org.apache.spark.rpc.RpcTimeout$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83) at org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102) at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$reportHeartBeat(Executor.scala:689) at org.apache.spark.executor.Executor$anon$1$anonfun$run$1.apply$mcV$sp(Executor.scala:718) at org.apache.spark.executor.Executor$anon$1$anonfun$run$1.apply(Executor.scala:718) at org.apache.spark.executor.Executor$anon$1$anonfun$run$1.apply(Executor.scala:718) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1963) at org.apache.spark.executor.Executor$anon$1.run(Executor.scala:718) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.util.concurrent.TimeoutException: Futures timed out after [10 seconds] at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219) at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223) at scala.concurrent.Await$anonfun$result$1.apply(package.scala:190) at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53) at scala.concurrent.Await$.result(package.scala:190) at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:81) ... 14 more 17/12/12 05:14:49 INFO ClientCnxn: Opening socket connection to server ip-192-168-180-21.ca-central-1.compute.internal/192.168.180.21:2181. Will not attempt to authenticate using SASL (unknown error) 17/12/12 05:14:49 INFO ClientCnxn: Socket connection established, initiating session, client: /192.168.180.54:53080, server: ip-192-168-180-21.ca-central-1.compute.internal/192.168.180.21:2181 17/12/12 05:14:49 INFO ClientCnxn: Unable to reconnect to ZooKeeper service, session 0x3604a1e09c01d5f has expired, closing socket connection 17/12/12 05:14:49 WARN ConnectionManager$HConnectionImplementation: This client just lost it's session with ZooKeeper, closing it. It will be recreated next time someone needs it org.apache.phoenix.shaded.org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:634) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:566) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:40) at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:534) at org.apache.phoenix.shaded.org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 17/12/12 05:14:49 INFO ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x3604a1e09c01d5f 17/12/12 05:14:49 INFO ClientCnxn: EventThread shut down 17/12/12 05:14:50 INFO ClientCnxn: Opening socket connection to server ip-xxxca-central-1.compute.internal/xxx:2181. Will not attempt to authenticate using SASL (unknown error) 17/12/12 05:14:50 INFO ClientCnxn: Socket connection established, initiating session, client: /xxx:53088, server: ip-xxx.ca-central-1.compute.internal/xxxx:2181 17/12/12 05:14:50 INFO ClientCnxn: Session establishment complete on server ip-xxx.ca-central-1.compute.internal/xxx:2181, sessionid = 0x2604a1e0a641d85, negotiated timeout = 60000 17/12/12 05:14:50 INFO ClientCnxn: Opening socket connection to server ip-192-168-181-26.ca-central-1.compute.internal/192.168.181.26:2181. Will not attempt to authenticate using SASL (unknown error) 17/12/12 05:14:50 INFO ClientCnxn: Socket connection established, initiating session, client: /192.168.180.54:59278, server: ip-xxx.ca-central-1.compute.internal/xxx:2181 17/12/12 05:14:50 INFO ClientCnxn: Session establishment complete on server ip-xxxx.ca-central-1.compute.internal/xxx:2181, sessionid = 0x2604a1e0a641d82, negotiated timeout = 60000 ... View more

Hi, I am using HDP 2.5.3.0 version. I have created schema in phoenix and under that schema I have created table. After creating table in phoenix, I have tried to created external table on hive. But while creating table I am getting following error. Failed with exception java.io.IOException:java.lang.RuntimeException: org.apache.phoenix.schema.TableNotFoundException: ERROR 1012 (42M03): Table undefined. tableName=ACCOUNT1 DDL:- CREATE EXTERNAL TABLE account1 ( acct_no Int , acct_descr string ) ROW FORMAT SERDE 'org.apache.phoenix.hive.PhoenixSerDe' STORED BY 'org.apache.phoenix.hive.PhoenixStorageHandler' TBLPROPERTIES ( "phoenix.table.name" = "account1", "phoenix.zookeeper.quorum"="<ZKNODE>", "phoenix.zookeeper.znode.parent" = "/hbase-unsecure", "phoenix.zookeeper.client.port" = "2181", "phoenix.rowkeys" = "acct_no", "phoenix.column.mapping" = "acct_no:acct_no, acct_descr:acct_descr"); Tried following steps also to resolve the issue. 1)jar uf /usr/hdp/current/hive-server2/auxlib/phoenix-4.7.0.2.5.6.0-40-client.jar /etc/hive/conf/hive-site.xml 2) jar uf /usr/hdp/current/hive-server2/auxlib/phoenix-4.7.0.2.5.6.0-40-client.jar /etc/hbase/conf/hbase-site.xml 3) jar uf /usr/hdp/current/hive-server2/auxlib/phoenix-hive-4.7.0.2.5.3.0-37.jar /etc/hbase/conf/hbase-site.xml 4) jar uf /usr/hdp/current/hive-server2/auxlib/phoenix-hive-4.7.0.2.5.3.0-37.jar /etc/hive/conf/hive-site.xml 5) jar -uf /usr/hdp/2.5.6.0-40/phoenix/phoenix-4.7.0.2.5.6.0-40-client.jar /etc/hive/conf/hive-site.xml 6) jar -uf /usr/hdp/2.5.6.0-40/phoenix/phoenix-4.7.0.2.5.6.0-40-client.jar /etc/hbase/conf/hbase-site.xml 7) Edit Ambari->Hive->Configs->Advanced->Custom hive-site->Add Property..., add the following properties based on your HBase configurations(you can search in Ambari->HBase->Configs): custom hive-site.xml hbase.zookeeper.quorum= ZKNODEt zookeeper.znode.parent=/hbase-unsecure phoenix.schema.mapSystemTablesToNamespace=true phoenix.schema.isNamespaceMappingEnabled=true 😎 Restart Hive via Ambari 9) Copy jar to /usr/hdp/current/hive-server2/auxlib from /usr/hdp/2.5.6.0-40/phoenix/phoenix-4.7.0.2.5.6.0-40-hive.jar /usr/hdp/2.5.6.0-40/phoenix/phoenix-hive-4.7.0.2.5.6.0-40-sources.jar download jar phoenix-hive-4.7.0.2.5.3.0-37.jar HIVE_AUX_JARS_PATH=/usr/hdp/current/hive-server2/auxlib 10)phoenix.schema.mapSystemTablesToNamespace=true phoenix.schema.isNamespaceMappingEnabled=true Please help on this. Need urgent help. ... View more