I got CDH 5.7.2, ... View more

I find the issue occurs when the table is written and then read with minimal time in between.   A workaround I used was to put a sleep for some seconds. ... View more

Thanks everyone for their input. 🙂   My peer, Rohit Narra, suggested that we *may* get "WARNINGS: Unknown disk id" when the stats on the paritions are NOT computed incrementally, and is reproduable even if we limit to 10 rows. However, when we compute stats incrementally, this error goes away. Stragne as it may seem, I can confirm I was able to reproduce the problem and the solution as shared below -- I masked some info for anonymity. It is a scary but really non-specific warning -- could be anything from catalogd unable to communicate impalad to some host problems. Cloudera might as well say that "something may be wrong" 😛   Anyways, In this case, the warning was a false alarm. @Bharathv, thanks for confirming the warning has improved with CDH 5.12.x.   = process = I simplified the query to generate the "WARNINGS: Unknown disk id" while querying a single partition.   [prodserver.ca:21000] > select dt_skey from serv_video_esd.cdn_integrated where dt_skey=20170824 and org_nm='media' and service='media' and cdn_nm='nakami' limit 3; Query: select dt_skey from serv_video_esd.cdn_integrated where dt_skey=20170824 and org_nm='media' and service='media' and cdn_nm='nakami' limit 3 +----------+ | dt_skey | +----------+ | 20170824 | | 20170824 | | 20170824 | +----------+ WARNINGS: Unknown disk id. This will negatively affect performance. Check your hdfs settings to enable block location metadata.   The table is partitioned on four attributes: | PARTITIONED BY ( | dt_skey INT, | org_nm STRING, | service STRING, | cdn_nm STRING   Notice, the stats is "true" on this partition: | dt_skey | org_nm | service | cdn_nm | #Rows | #Files | Size | Bytes Cached | Cache Replication | Format | Incremental stats | Location | 20170824 | media | media | nakami | 257006421 | 1549 | 12.91GB | NOT CACHED | NOT CACHED | PARQUET | true | hdfs://nameservice1/user/hive/warehouse/serv_video_esd.db/cdn_integrated/dt_skey=20170824/org_nm=media/service=media/cdn_nm=nakami   I recomputed stats anyway. COMPUTE INCREMENTAL STATS serv_video_esd.cdn_integrated PARTITION (dt_skey=20170824,org_nm='media',service='media',cdn_nm='nakami'); Query: compute INCREMENTAL STATS serv_video_esd.cdn_integrated PARTITION (dt_skey=20170824,org_nm='media',service='media',cdn_nm='nakami') +------------------------------------------+ | summary | +------------------------------------------+ | Updated 1 partition(s) and 26 column(s). | +------------------------------------------+ Fetched 1 row(s) in 27.21s   [prodserver.ca:21000] > select dt_skey from serv_video_esd.cdn_integrated where dt_skey=20170824 and org_nm='media' and service='media' and cdn_nm='nakami' limit 3; Query: select dt_skey from serv_video_esd.cdn_integrated where dt_skey=20170824 and org_nm='media' and service='media' and cdn_nm='nakami' limit 3 +----------+ | dt_skey | +----------+ | 20170824 | | 20170824 | | 20170824 | +----------+ Fetched 3 row(s) in 0.12s <================================ "WARNINGS: Unknown disk id." disappers!!! ... View more

Thanks everyone for their replies. @Tim HDFS Rebalance process finishes under a minute. Seems that data is balanced.   @Bharathv re DN being busy. do you mean the data node daemon or machine? For the coordinator host, - JVM Heap Memory of DN process coordinator is 50% occupied. - CPU of DN process is close to zero. - HOST CPU is around 30%. uptime is low (as 24 core machine) 10:47:49 up 364 days, 22:34, 1 user, load average: 3.38, 5.46, 5.20 - HOST Physical Memory Used is around 130gb out of 250gb I am inclined to rule out possibility of being busy.       ... View more

We perform "compute stats" fairly regularly but still get this message. Besides, don't seems a very relevant messaged for dated stats. 😛   Has this jira been resolved?   https://issues.apache.org/jira/browse/IMPALA-1427   Need to confirm if this not a "rogue" message. We have impala daemon versions.   [root@hostname~]# impalad --version impalad version 2.5.0-cdh5.7.2 RELEASE (build 1140f8289dc0d2b1517bcf70454bb4575eb8cc70) Built on Fri, 22 Jul 2016 12:30:57 PST   [root@hostname~]# catalogd --version catalogd version 2.5.0-cdh5.7.2 RELEASE (build 1140f8289dc0d2b1517bcf70454bb4575eb8cc70) Built on Fri, 22 Jul 2016 12:30:57 PST ... View more

Users are getting *non-deterministic* "Memory limit exceeded" for impala queries. Impala Daemon Memory Limit: 100 GB. spill to disk is enabled    However, a query failed with the above memory with Aggregate Peak Memory Usage: 125 MiB   Explored query profile by CM -> Impala -> Queries -> {failed query "oom=true AND stats_missing=false"}   Want help in narrowing down the cause of the failure:  inaccurate stats, congestion, hdfs disk rebalancing, or  something else?   Where can I find the detailed of the failure? /var/log/impalad and catalogd state the "Query ID" but not the failure details. For example, impala logs stated the Query ID only: a24fb2eae077b513:45c8d35936a35e9e   impalad.w0575oslphcda11.bell.corp.bce.ca.impala.log.INFO.20170124-031735.13521:I0124 03:57:06.889070 44232 plan-fragment-executor.cc:92] Prepare(): query_id=a24fb2eae077b513:45c8d35936a35e93 instance_id=a24fb2eae077b513:45c8d35936a35e9e     ... View more