In My HDP 3.1.5 cluster, Tez is not following the reducer order in the VERTICES (shown in the first pic). I am interested in knowing how Tez decides the order and number of containers in a VERTICES. I can even see single-digit containers in a few VERTICES. Resources are free on the cluster but it has very few containers running.      ... View more

I have not installed HBase. I am on HDP 3.1.5 and I am running hive queries using Tez engine. I see the below error when Tez container fails an attempt.      2020-10-03 12:27:12,685 INFO storage.HBaseTimelineReaderImpl (HBaseTimelineReaderImpl.java:run(170)) - Running HBase liveness monitor 2020-10-03 12:27:12,985 INFO zookeeper.ReadOnlyZKClient (ReadOnlyZKClient.java:run(315)) - 0x712625fd no activities for 60000 ms, close active connection. Will reconnect next time when there are new requests. 2020-10-03 12:27:42,917 WARN storage.HBaseTimelineReaderImpl (HBaseTimelineReaderImpl.java:run(183)) - Got failure attempting to read from timeline storage, assuming HBase down java.io.UncheckedIOException: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=4, exceptions: Sat Oct 03 12:27:36 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 Sat Oct 03 12:27:37 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.io.IOException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: visa-dn01-hs.cluster.com/10.1.0.163:17020 Sat Oct 03 12:27:39 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 Sat Oct 03 12:27:42 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:55) at org.apache.hadoop.yarn.server.timelineservice.storage.reader.TimelineEntityReader.readEntities(TimelineEntityReader.java:283) at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineReaderImpl$HBaseMonitor.run(HBaseTimelineReaderImpl.java:174) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=4, exceptions: Sat Oct 03 12:27:36 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 Sat Oct 03 12:27:37 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.io.IOException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on local exception: org.apache.hadoop.hbase.ipc.FailedServerException: This server is in the failed servers list: visa-dn01-hs.cluster.com/10.1.0.163:17020 Sat Oct 03 12:27:39 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 Sat Oct 03 12:27:42 EDT 2020, RpcRetryingCaller{globalStartTime=1601742456884, pause=1000, maxAttempts=4}, java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:145) at org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:80) ... 3 more Caused by: java.net.ConnectException: Call to visa-dn01-hs.cluster.com/10.1.0.163:17020 failed on connection exception: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:165) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:390) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:95) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:410) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:406) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118) at org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.userEventTriggered(BufferCallBeforeInitHandler.java:92) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:307) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1377) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:329) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:315) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:929) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.failInit(NettyRpcConnection.java:179) at org.apache.hadoop.hbase.ipc.NettyRpcConnection.access$500(NettyRpcConnection.java:71) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:269) at org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:263) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:507) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:500) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:479) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:122) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:327) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:343) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:633) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) ... 1 more Caused by: org.apache.hbase.thirdparty.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: visa-dn01-hs.cluster.com/10.1.0.163:17020 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hbase.thirdparty.io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ... 7 more Caused by: java.net.ConnectException: Connection refused ... 11 more ... View more

Hi All,   Few queries under https://github.com/hortonworks/hive-testbench are resulting in "Unsupported SubQuery Expression" error.    I am using HDP 3.1.5, Hive 3.1.0, Tez 0.9.1, NO LLAP.    0: jdbc:hive2://visa-dn01-hs.cluster.com:2181> source query1.sql; 20/09/22 05:54:41 [main]: WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist 20/09/22 05:54:41 [main]: WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist Error: Error while compiling statement: FAILED: SemanticException Line 0:-1 Unsupported SubQuery Expression 'ctr_store_sk': Only SubQuery expressions that are top level conjuncts are allowed (state=42000,code=40000) 0: jdbc:hive2://visa-dn01-hs.cluster.com:2181> source query10.sql; 20/09/22 05:54:48 [main]: WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist 20/09/22 05:54:48 [main]: WARN conf.HiveConf: HiveConf of name hive.heapsize does not exist Error: Error while compiling statement: FAILED: SemanticException [Error 10249]: Line 28:4 Unsupported SubQuery Expression '3': Only 1 SubQuery expression is supported. (state=42000,code=10249) ... View more

2019-03-19 04:48:41,934 INFO server.AuthenticationFilter (AuthenticationFilter.java:constructSecretProvider(240)) - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2019-03-19 04:48:41,940 INFO server.KerberosAuthenticationHandler (KerberosAuthenticationHandler.java:init(158)) - Using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/ppc64le-rhel75-50g-1.novalocal@EXAMPLE.COM 2019-03-19 04:48:41,951 INFO server.AbstractConnector (AbstractConnector.java:doStart(278)) - Started ServerConnector@2c5529ab{HTTP/1.1,[http/1.1]}{ppc64le-rhel75-50g-1.novalocal:50070} 2019-03-19 04:48:41,960 INFO server.AbstractConnector (AbstractConnector.java:doStop(318)) - Stopped ServerConnector@2c5529ab{HTTP/1.1,[http/1.1]}{ppc64le-rhel75-50g-1.novalocal:50070} 2019-03-19 04:48:41,963 INFO handler.ContextHandler (ContextHandler.java:doStop(910)) - Stopped o.e.j.s.ServletContextHandler@77258e59{/logs,file:///var/log/hadoop/hdfs/,UNAVAILABLE} 2019-03-19 04:48:41,967 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(210)) - Stopping NameNode metrics system... 2019-03-19 04:48:41,968 INFO impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted. 2019-03-19 04:48:41,969 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(216)) - NameNode metrics system stopped. 2019-03-19 04:48:41,970 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - NameNode metrics system shutdown complete. 2019-03-19 04:48:41,970 ERROR namenode.NameNode (NameNode.java:main(1715)) - Failed to start namenode. java.io.IOException: Problem starting http server at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1165) at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:177) at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:869) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:691) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710) Caused by: javax.servlet.ServletException: Public key for signature validation must be provisioned. at org.apache.hadoop.security.authentication.server.JWTRedirectAuthenticationHandler.init(JWTRedirectAuthenticationHandler.java:125) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:194) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:180) at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:349) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.server.Server.start(Server.java:427) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.server.Server.doStart(Server.java:394) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1134) ... 7 more 2019-03-19 04:48:41,972 INFO timeline.HadoopTimelineMetricsSink (AbstractTimelineMetricsSink.java:getCurrentCollectorHost(291)) - No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 2019-03-19 04:48:41,973 INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: java.io.IOException: Problem starting http server 2019-03-19 04:48:41,977 INFO namenode.NameNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG: ... View more

Hi, I have setup single node and multinode (1MN+3DN) cluster with HDP 2.6. I am using xfs filesystem on RHEL72. After deployment the setup looks good but after some time few nodes stops responding with the below error (can be seen in the console). sda5 is a root disk, hence I cant login and fix the filesystem. Reboot fixes the issue. Did check the disks and dint notice any issue with the disks. [16613.582270] systemd[14024]: Failed at step EXEC spawning /sbin/agetty: Input/output error [16613.831856] systemd[14027]: Failed at step EXEC spawning /sbin/agetty: Input/output error [16614.082272] systemd[14030]: Failed at step EXEC spawning /sbin/agetty: Input/output error [16614.332070] systemd[14033]: Failed at step EXEC spawning /sbin/agetty: Input/output error [16614.581558] systemd[14036]: Failed at step EXEC spawning /sbin/agetty: Input/output error [16616.436262] XFS (sda5): xfs_log_force: error -5 returned. [16646.515588] XFS (sda5): xfs_log_force: error -5 returned. [16676.594914] XFS (sda5): xfs_log_force: error -5 returned. ... View more