Member since
04-26-2017
9
Posts
0
Kudos Received
0
Solutions
10-20-2020
01:10 AM
You can check the explain plan . It provided the order of the execution. It is not a rule that certain reucers would start after all the mapper phase. The ceratin mapper phase depends on the results from the reducers too. You can check the explain plan . If you want map join then you can set property map.auto.conevrt.join =true
... View more
10-07-2020
04:28 AM
Its a service that's running, not a job. So avoid killing it.
... View more
10-03-2020
09:40 AM
I have not installed HBase. I am on HDP 3.1.5 and I am running hive queries using Tez engine. I see the below error when Tez container fails an attempt. 2020-10-03 12:27:12,685 INFO storage.HBaseTimelineReaderImpl (HBaseTimelineReaderImpl.java:run(170)) - Running HBase liveness monitor 2020-10-03 12:27:12,985 INFO zookeeper.ReadOnlyZKClient (ReadOnlyZKClient.java:run(315)) - 0x712625fd no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2020-10-03 12:27:42,917 WARN storage.HBaseTimelineReaderImpl (HBaseTimelineReaderImpl.java:run(183)) - Got failure attempting to read from timeline storage, assuming HBase down java.io.UncheckedIOException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=4, exceptions: Sat Oct 03 12:27:36 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 Sat Oct 03 12:27:37 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.io.IOException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: visa-dn01-hs.cluster.com/10.1.0.163:17020 Sat Oct 03 12:27:39 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 Sat Oct 03 12:27:42 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55) at org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReader.readEntities(TimelineEntityReader.java:283) at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl$HBaseMonitor.run(HBaseTimelineReaderImpl.java:174) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=4, exceptions: Sat Oct 03 12:27:36 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 Sat Oct 03 12:27:37 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.io.IOException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: visa-dn01-hs.cluster.com/10.1.0.163:17020 Sat Oct 03 12:27:39 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 Sat Oct 03 12:27:42 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80) ... 3 more Caused by: java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:269) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:263) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ... 7 more Caused by: java.net.ConnectException: Connection refused ... 11 more
... View more
- Tags:
- HBase
- Tez engine
09-22-2020
03:47 AM
Hi All,
Few queries under https://github.com/hortonworks/hive-testbench are resulting in "Unsupported SubQuery Expression" error.
I am using HDP 3.1.5, Hive 3.1.0, Tez 0.9.1, NO LLAP.
0: jdbc:hive2://visa-dn01-hs.cluster.com:2181> source query1.sql; 20/09/22 05:54:41 [main]: WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist 20/09/22 05:54:41 [main]: WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist Error: Error while compiling statement: FAILED: SemanticException Line 0:-1 Unsupported SubQuery Expression 'ctr_store_sk': Only SubQuery expressions that are top level conjuncts are allowed (state=42000,code=40000) 0: jdbc:hive2://visa-dn01-hs.cluster.com:2181> source query10.sql; 20/09/22 05:54:48 [main]: WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist 20/09/22 05:54:48 [main]: WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist Error: Error while compiling statement: FAILED: SemanticException [Error 10249]: Line 28:4 Unsupported SubQuery Expression '3': Only 1 SubQuery expression is supported. (state=42000,code=10249)
... View more
07-13-2019
04:49 PM
Any luck on this @Santoshkumar Hiremath? I am facing a similar issue. Any help/pointer is appreciated. Thanks.
... View more
04-26-2017
12:26 PM
Hi, I have setup single node and multinode (1MN+3DN) cluster with HDP 2.6. I am using xfs filesystem on RHEL72. After deployment the setup looks good but after some time few nodes stops responding with the below error (can be seen in the console). sda5 is a root disk, hence I cant login and fix the filesystem. Reboot fixes the issue. Did check the disks and dint notice any issue with the disks. [16613.582270] systemd[14024]: Failed at step EXEC spawning /sbin/agetty: Input/output error
[16613.831856] systemd[14027]: Failed at step EXEC spawning /sbin/agetty: Input/output error
[16614.082272] systemd[14030]: Failed at step EXEC spawning /sbin/agetty: Input/output error
[16614.332070] systemd[14033]: Failed at step EXEC spawning /sbin/agetty: Input/output error
[16614.581558] systemd[14036]: Failed at step EXEC spawning /sbin/agetty: Input/output error
[16616.436262] XFS (sda5): xfs_log_force: error -5 returned.
[16646.515588] XFS (sda5): xfs_log_force: error -5 returned.
[16676.594914] XFS (sda5): xfs_log_force: error -5 returned.
... View more
- Tags:
- FileSystem