Member since
10-31-2019
6
Posts
0
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1322 | 11-11-2019 08:56 PM | |
| 2827 | 11-11-2019 07:58 PM |
12-09-2020
09:57 PM
Use impala-shell to refresh hive partition table: Query: REFRESH hdm.hdpos_v_zddb_xsmx
Query submitted at: 2020-12-10 05:07:51 (Coordinator: http://realtimeanalysis-kudutablet01:25000)
Query progress can be monitored at: http://realtimeanalysis-kudutablet01:25000/query_plan?query_id=4541f98e8a7de9eb:2ab15a3400000000
Fetched 0 row(s) in 0.12s Then use jdbc connection query, no latest data Max Per-Host Resource Reservation: Memory=68.00MB Threads=3
Per-Host Resource Estimates: Memory=256MB
WARNING: The following tables are missing relevant table and/or column statistics.
hdm.hdpos_v_zddb_xsmx
Analyzed query: SELECT * FROM (SELECT accdate, busno, compid, wareid,
sum(wareqty) wareqty, sum(nvl(netsum, CAST(0 AS DECIMAL(16,4)))) netsum,
sum(nvl(stdsum, CAST(0 AS DECIMAL(16,4)))) stdsum, sum(nvl(puramount, CAST(0 AS
DECIMAL(16,4)))) puramount FROM hdm.hdpos_v_zddb_xsmx WHERE dt = '20201209' AND
accdate = TIMESTAMP '2020-12-09 00:00:00' GROUP BY accdate, busno, compid,
wareid) a WHERE a.wareqty > CAST(0 AS DECIMAL(3,0))
F02:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
| Per-Host Resources: mem-estimate=108.00KB mem-reservation=0B thread-reservation=1
PLAN-ROOT SINK
| mem-estimate=0B mem-reservation=0B thread-reservation=0
|
04:EXCHANGE [UNPARTITIONED]
| mem-estimate=108.00KB mem-reservation=0B thread-reservation=0
| tuple-ids=1 row-size=104B cardinality=0
| in pipelines: 03(GETNEXT)
|
F01:PLAN FRAGMENT [HASH(accdate,busno,compid,wareid)] hosts=1 instances=1
Per-Host Resources: mem-estimate=128.11MB mem-reservation=34.00MB thread-reservation=1
03:AGGREGATE [FINALIZE]
| output: sum:merge(wareqty), sum:merge(nvl(netsum, 0)), sum:merge(nvl(stdsum, 0)), sum:merge(nvl(puramount, 0))
| group by: accdate, busno, compid, wareid
| having: sum(wareqty) > CAST(0 AS DECIMAL(3,0))
| mem-estimate=128.00MB mem-reservation=34.00MB spill-buffer=2.00MB thread-reservation=0
| tuple-ids=1 row-size=104B cardinality=0
| in pipelines: 03(GETNEXT), 00(OPEN)
|
02:EXCHANGE [HASH(accdate,busno,compid,wareid)]
| mem-estimate=108.00KB mem-reservation=0B thread-reservation=0
| tuple-ids=1 row-size=104B cardinality=0
| in pipelines: 00(GETNEXT)
|
F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1
Per-Host Resources: mem-estimate=128.00MB mem-reservation=34.00MB thread-reservation=1
01:AGGREGATE [STREAMING]
| output: sum(wareqty), sum(nvl(netsum, CAST(0 AS DECIMAL(16,4)))), sum(nvl(stdsum, CAST(0 AS DECIMAL(16,4)))), sum(nvl(puramount, CAST(0 AS DECIMAL(16,4))))
| group by: accdate, busno, compid, wareid
| mem-estimate=128.00MB mem-reservation=34.00MB spill-buffer=2.00MB thread-reservation=0
| tuple-ids=1 row-size=104B cardinality=0
| in pipelines: 00(GETNEXT)
|
00:SCAN HDFS [hdm.hdpos_v_zddb_xsmx, RANDOM]
partition predicates: dt = '20201209'
partitions=0/155 files=0 size=0B
predicates: accdate = TIMESTAMP '2020-12-09 00:00:00'
stored statistics:
table: rows=unavailable size=unavailable
partitions: 0/0 rows=unavailable
columns missing stats: accdate, busno, compid, wareid, wareqty, netsum, stdsum, puramount
extrapolated-rows=disabled max-scan-range-rows=0
mem-estimate=0B mem-reservation=0B thread-reservation=0
tuple-ids=0 row-size=72B cardinality=0
in pipelines: 00(GETNEXT) I checked when hdfs generates the partition table before refreshing the partition. how can I solve this problem, please give suggestions, thanks!
... View more
Labels:
- Labels:
-
Apache Impala
11-11-2019
08:56 PM
The problem has been solved because the yarn service component has a client configuration with no value. There is a warning in the yarn component. After restoring it, hive and yarn can operate normally.
... View more
11-11-2019
07:58 PM
My problem has been solved because the yarn service component has a client configuration that initializes itself without value.
... View more
11-10-2019
07:59 PM
The version of CDH5.16.1 I used recently has the following error, which makes it impossible to use the cm management platform to execute some commands, such as distributing the Gateway configuration. I hope to provide some solutions. 2019-11-07 14:00:02,161 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.service.AbstractRoleHandler: (1 skipped) Unable to generate configuration for GATEWAY ba se group 2019-11-07 14:00:02,161 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.service.AbstractRoleHandler: Unable to generate configuration for HIVESERVER2: hive-HIVE SERVER2-0f0c09243092e98c6d762d9b05f72d25 2019-11-07 14:00:02,163 ERROR com.cloudera.cmf.scheduler-1_Worker-1:com.cloudera.cmf.scheduler.CommandDispatcherJob: Failed to invoke command (GlobalPoolsRefresh) for schedu le (1) java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NullPointerException at com.cloudera.cmf.service.AbstractRoleHandler.generateConfigFiles(AbstractRoleHandler.java:968) at com.cloudera.cmf.service.AbstractRoleHandler.generateConfiguration(AbstractRoleHandler.java:902) at com.cloudera.cmf.service.hive.BaseHiveRoleHandler.makeProcess(BaseHiveRoleHandler.java:93) at com.cloudera.cmf.command.GlobalPoolsRefreshCommand$1.apply(GlobalPoolsRefreshCommand.java:104) at com.cloudera.cmf.command.GlobalPoolsRefreshCommand$1.apply(GlobalPoolsRefreshCommand.java:82) at com.cloudera.cmf.cluster.PoolsRefreshCmdWorkCreator.constructRoleRefreshes(PoolsRefreshCmdWorkCreator.java:80) at com.cloudera.cmf.command.GlobalPoolsRefreshCommand.constructWork(GlobalPoolsRefreshCommand.java:123) at com.cloudera.cmf.command.GlobalPoolsRefreshCommand.constructWork(GlobalPoolsRefreshCommand.java:75) at com.cloudera.cmf.command.GlobalPoolsRefreshCommand.constructWork(GlobalPoolsRefreshCommand.java:34) at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:54) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeGlobalCommandHelper(ServiceHandlerRegistry.java:781) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeGlobalCommand(ServiceHandlerRegistry.java:749) at com.cloudera.server.cmf.components.OperationsManagerImpl.executeGlobalCmd(OperationsManagerImpl.java:1752) at com.cloudera.cmf.scheduler.CommandDispatcherJob.dispatchCommand(CommandDispatcherJob.java:248) at com.cloudera.cmf.scheduler.CommandDispatcherJob.executeCommand(CommandDispatcherJob.java:191) at com.cloudera.cmf.scheduler.CommandDispatcherJob.access$500(CommandDispatcherJob.java:61) at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:168) at com.cloudera.cmf.scheduler.CommandDispatcherJob$1.call(CommandDispatcherJob.java:165) at com.cloudera.server.common.RetryWrapper.executeWithRetry(RetryWrapper.java:32) at com.cloudera.server.common.RetryUtils.executeWithRetryHelper(RetryUtils.java:210) at com.cloudera.server.common.RetryUtils.executeWithRetryConstantSleep(RetryUtils.java:73) at com.cloudera.cmf.scheduler.CommandDispatcherJob.execute(CommandDispatcherJob.java:162) at org.quartz.core.JobRunShell.run(JobRunShell.java:206) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:548) Caused by: java.lang.RuntimeException: java.lang.NullPointerException at com.cloudera.cmf.service.AbstractRoleHandler.generateConfigFiles(AbstractRoleHandler.java:968) at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfigFiles(AbstractClientConfigHandler.java:124) at com.cloudera.cmf.service.AbstractClientConfigHandler.buildClientConfigFiles(AbstractClientConfigHandler.java:110) at com.cloudera.cmf.service.components.ConfigHelper.getClientConfigs(ConfigHelper.java:147) at com.cloudera.cmf.service.AbstractRoleHandler.generateClientConfigsForDependencies(AbstractRoleHandler.java:1006) at com.cloudera.cmf.service.AbstractRoleHandler.generateConfigFiles(AbstractRoleHandler.java:952) ... 23 more Caused by: java.lang.NullPointerException at com.cloudera.cmf.service.hadoopcommon.HadoopCommonHelpers.makeJavaOpts(HadoopCommonHelpers.java:242) Thank you.
... View more
Labels:
- Labels:
-
Apache Hive
-
Cloudera Manager
10-31-2019
07:39 PM
Hi, everyone, my CDH cluster can't execute commands on the cm management platform, such as pushing the client configuration, killing the jobs on the yarn, etc. The error log of the server is as follows. Thank you for your suggestions. Thank you. com.cloudera.cmf.command.CmdExecException: com.cloudera.cmf.command.CmdExecException: java.lang.RuntimeException: java.lang.NullPointerException A server error has occurred. The full stack trace is not shown here due to security reasons. See Cloudera Manager Server log for details. WARN 986168735@scm-web-253:com.cloudera.cmf.command.flow.CmdStep: Unexpected exception during command work com.cloudera.cmf.command.CmdExecException: java.lang.RuntimeException: java.lang.NullPointerException at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:119) at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommandHelper(ServiceHandlerRegistry.java:885) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeCommand(ServiceHandlerRegistry.java:845) at com.cloudera.cmf.command.flow.CmdWorkCtx.execSvcCmd(CmdWorkCtx.java:159) at com.cloudera.cmf.command.flow.work.ExecSvcCmdWork.doWork(ExecSvcCmdWork.java:91) at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177) at com.cloudera.cmf.command.flow.work.ScatterCmdWork.doWork(ScatterCmdWork.java:86) at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177) at com.cloudera.cmf.command.flow.SeqCmdWork.doWork(SeqCmdWork.java:107) at com.cloudera.cmf.command.flow.CmdStep.doWork(CmdStep.java:177) at com.cloudera.cmf.command.flow.SeqFlowCmd.run(SeqFlowCmd.java:117) at com.cloudera.cmf.command.CmdWorkCommand.execute(CmdWorkCommand.java:94) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeClusterCommandHelper(ServiceHandlerRegistry.java:833) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeClusterCommand(ServiceHandlerRegistry.java:793) at com.cloudera.cmf.service.ServiceHandlerRegistry.executeClusterCommand(ServiceHandlerRegistry.java:788) at com.cloudera.server.cmf.components.OperationsManagerImpl.executeClusterCmd(OperationsManagerImpl.java:1771) at com.cloudera.server.web.cmf.cluster.ClusterController.dispatchClusterCommand(ClusterController.java:213) at com.cloudera.server.web.cmf.cluster.ClusterController$$FastClassByCGLIB$$2b29d5cd.invoke() at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191) at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621) at com.cloudera.server.web.cmf.cluster.ClusterController$$EnhancerByCGLIB$$e0bcbdd8.dispatchClusterCommand() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606)
... View more
Labels:
- Labels:
-
Cloudera Manager