Member since
07-01-2015
460
Posts
78
Kudos Received
43
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1346 | 11-26-2019 11:47 PM | |
| 1304 | 11-25-2019 11:44 AM | |
| 9471 | 08-07-2019 12:48 AM | |
| 2175 | 04-17-2019 03:09 AM | |
| 3484 | 02-18-2019 12:23 AM |
05-15-2017
06:18 AM
Hi, I have an old table where data was created by Impala (2.x). The table is accessible by Impala and the data returned by Impala is valid and correct. However when I try to read the same table (partition) by SparkSQL or Hive, I got in 3 out of 30 columns NULL values. The data are there, the columns are varchars as the others, nullability is the same, and they contain non null values. When I created a new table via CTAS in Impala: create table tmp.sample stored as parquet as select * from orig_table where partcol = xxx; Then the data are correctly read by Hive and SparkSQL, all 30 columns. The old table has these properties: | ROW FORMAT SERDE | | 'parquet.hive.serde.ParquetHiveSerDe' | | STORED AS INPUTFORMAT | | 'parquet.hive.DeprecatedParquetInputFormat' | | OUTPUTFORMAT | | 'parquet.hive.DeprecatedParquetOutputFormat' | The new table has different row format serde and input/output formats: | ROW FORMAT SERDE | | 'parquet.hive.serde.ParquetHiveSerDe' | | STORED AS INPUTFORMAT | | 'parquet.hive.DeprecatedParquetInputFormat' | | OUTPUTFORMAT | | 'parquet.hive.DeprecatedParquetOutputFormat' | How can I tweak Hive/Beeline/Spark to read the data correctly from the original table? Thanks PS. Using CDH5.7.1
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Impala
03-08-2017
03:52 AM
I have found out what was the problem. The solution is to set the singleSession property to true in command line. Because setting it in a program seems to not work properly. /bin/spark-shell --conf spark.sql.hive.thriftServer.singleSession=true WORKS. /bin/spark-shell ... sql.setConf("spark.sql.hive.thriftServer.singleSession","true") ... DOES NOT WORK
... View more
03-06-2017
04:27 AM
Hi, I have found a general template how to access spark temporary data (id data frame) via an external tool using JDBC. What I have found that it should be quite simple: 1. Run spark-shell or submit spark job 2. Configure HiveContext and then run HiveThirftServer from the job. In a separate session access the thrift server via beeline and query data. Here is my code Spark 2.1: import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.thriftserver.HiveThriftServer2
import org.apache.spark.sql.hive.thriftserver._
val sql = new HiveContext(sc)
sql.setConf("hive.server2.thrift.port", "10002")
sql.setConf("hive.server2.authentication","KERBEROS" )
sql.setConf("hive.server2.authentication.kerberos.principal","hive/host1.lab.hadoop.net@LAB.HADOOP.NET" )
sql.setConf("hive.server2.authentication.kerberos.keytab","/home/h.keytab" )
sql.setConf("spark.sql.hive.thriftServer.singleSession","true")
val data = sql.sql("select 112 as id")
data.collect
data.createOrReplaceTempView("yyy")
sql.sql("show tables").show
HiveThriftServer2.startWithContext(sql)
WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException Connect to the JDBC server: beeline -u "jdbc:hive2://localhost:10002/default;principal=hive/host1.lab.hadoop.net@LAB.HADOOP.NET" However when I try to launch the HiveThriftServer2 I can access the spark thrift but do not see the temporary table. Command "show tables" do not show any temporary table. Trying to query "yyy" throws an error: scala> sql.sql("show tables").collect
res11: Array[org.apache.spark.sql.Row] = Array([,sometablename,true], [,yyy,true])
scala> 17/03/06 11:15:50 ERROR thriftserver.SparkExecuteStatementOperation: Error executing query, currentState RUNNING,
org.apache.spark.sql.AnalysisException: Table or view not found: yyy; line 1 pos 14
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.org$apache$spark$sql$catalyst$analysis$Analyzer$ResolveRelations$$lookupTableFromCatalog(Analyzer.scala:459)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:478)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$8.applyOrElse(Analyzer.scala:463)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:61)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:61) If I create a table from beeline via "create table t as select 100 as id" the table is created and I can see it in spark-shell (data stored locally in spark-warehouse directory) So the other direction is working. So the question what I am missing, why I can't see the temporary table? Thanks
... View more
Labels:
- Labels:
-
Apache Spark
02-19-2017
11:37 PM
I dont fully understand your goal, but sqoop just transfers the data from Oracle to HDFS and updates the metadata. As mentioned above, Impala is absolutely fine with this, just do invalidate metadata or refresh on that table which was imported. Regarding your case, if you need to extend the table - add one column - I would do it by transfering the imported table into a managed table (via Hive or Impala) and in that simple insert into ... select statement I would do the additional column. For this case wou don't need row level insert neither update or delete. And if you need to change some records (or remove) - you can still overwrite the partition. This approach is common for Hive and Impala (unless you are using Kudu, or Hive new transactions). But the basics remain the same - HDFS is a append only system, si Hive does nothing else, just keeps the list of changes (inserts, deletes, updates) in separate file and then time by time overwrites the tables in the background. Subquery is fully supported as far as I know, the data types can be easily changed via alter table replace command.
... View more
02-15-2017
05:40 AM
Yes setting the memory limit per query. My point was, that even when I set a much higher limit via SET MEM_LIMIT the query fails. Example: I set 3GB. The query fails and in the error message there is only 1GB consumption. I assume that the metrics about the memory consumption are not accurate, so ok, I assume that the query eated more than 3 gigs. But then I raise the limit to 5GB and the query succeeds. So I check the real execution plan, and there is no evidence of using more than 3GB, lets say there is a peak memory usage 1.5GB. I dont have real exec plans or profiles, but beleive me, the output is not correct. Regarding the admission control, this is the link: https://blog.cloudera.com/blog/2016/12/resource-management-for-apache-impala-incubating/ "Cloudera recommends that you set memory limits for queries whenever possible. Default memory limits can be set on all queries in a pool (see below), and explicitly specifying the query memory limit will override the default. If you are using admission control pools with restrictions on the maximum memory available to queries in a pool, setting default per-query memory limits is required." https://blog.cloudera.com/blog/2016/12/resource-management-for-apache-impala-incubating/ Another cloudera documentation states clearly, that if DO NOT SET the mem limit, then the default query limit is "allocated". So If I have 100GB of total memory, 5 nodes, and I set 10GB as a default, then if two users run a query in the same time (even tiny ones) then the third query will go to the queue and will wait. " A dynamic resource pool has Max Memory set to 100 GB. The Default Query Memory Limit for the pool is 10 GB. Therefore, any query running in this pool could use up to 50 GB of memory (default query memory limit * number of Impala nodes). The maximum number of queries that Impala executes concurrently within this dynamic resource pool is two, which is the most that could be accomodated within the 100 GB Max Memory cluster-wide limit." So what I would like to do is to measure EXACTLY the peak memory and SET it to the value that is safe enough to go through. The problem is, that explain plans, and error messages are not correct, or I am not understanding the whole memory limit concept at all. T.
... View more
02-14-2017
01:41 AM
And even more confusing is this: Limit is way bigger than the consumption, but still failing.. Memory Limit Exceeded
Query(a04dc11b92580986:99f26ad466311b9f) Limit: Limit=4.00 GB Consumption=1.48 GB
Fragment a04dc11b92580986:99f26ad466311baa: Consumption=48.03 MB
SELECT_NODE (id=29): Consumption=0
ANALYTIC_EVAL_NODE (id=28): Consumption=0
SORT_NODE (id=27): Consumption=48.02 MB
EXCHANGE_NODE (id=44): Consumption=0
DataStreamRecvr: Consumption=0
Block Manager: Limit=3.20 GB Consumption=528.00 MB
Fragment a04dc11b92580986:99f26ad466311bb0: Consumption=988.46 MB
HASH_JOIN_NODE (id=26): Consumption=0
HASH_JOIN_NODE (id=25): Consumption=0
HASH_JOIN_NODE (id=24): Consumption=0
NESTED_LOOP_JOIN_NODE (id=23): Consumption=494.08 MB
Exprs: Consumption=494.00 MB
HASH_JOIN_NODE (id=22): Consumption=72.00 KB
HASH_JOIN_NODE (id=21): Consumption=64.00 KB
NESTED_LOOP_JOIN_NODE (id=20): Consumption=494.06 MB
Exprs: Consumption=494.00 MB
HASH_JOIN_NODE (id=19): Consumption=76.00 KB
Exprs: Consumption=28.00 KB
HASH_JOIN_NODE (id=18): Consumption=40.00 KB
HASH_JOIN_NODE (id=17): Consumption=32.00 KB
NESTED_LOOP_JOIN_NODE (id=16): Consumption=32.00 KB
HASH_JOIN_NODE (id=15): Consumption=16.00 KB
EXCHANGE_NODE (id=31): Consumption=0
DataStreamRecvr: Consumption=0
EXCHANGE_NODE (id=32): Consumption=0
DataStreamRecvr: Consumption=0
EXCHANGE_NODE (id=33): Consumption=0
... View more
02-14-2017
01:24 AM
Followed the Cloudera guide to set up and configure Impala Admission control via Dynamic Resource Pools in CDH 5.7 (Impala 2.5.0), one of the first thing is that "admin should know what kind of workflow is executed and what are the resource requests". I started with this, to capping the queries wiht try and error approach. I set up a limit for a particular query to 2GB via set MEM_LIMIT option and it worked fine. Later the query failed on memory limit exceeded error, however the explain clearly shows, that there are NO peak with more than few MBs. Everything is under 1GB (except the scan which has 1.08Gig) I had to raise the limit from 2GB to 4GB. Now the query runs ok. What concerns me is the fact, that relying on explain summary is not enough and I dont know where to look for a safe threshold. Exec Summary
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
---------------------------------------------------------------------------------------------------------------------------------
37:EXCHANGE 5 74.246ms 367.995ms 810.69K 3.62M 0 0 HASH(CAST(substring(field...
23:SELECT 5 15.465ms 23.090ms 810.69K 3.62M 3.01 MB 0
22:ANALYTIC 5 191.768ms 263.507ms 811.66K 3.62M 23.01 MB 0
21:SORT 5 1s492ms 1s802ms 811.66K 3.62M 112.64 MB 240.00 MB
36:EXCHANGE 5 66.684ms 74.809ms 811.66K 3.62M 0 0 HASH(d.eid)
20:HASH JOIN 5 233.074ms 404.226ms 811.66K 3.62M 2.10 MB 331.00 B LEFT OUTER JOIN, BROADCAST
|--35:EXCHANGE 5 12.292us 14.792us 71 7 0 0 BROADCAST
| 11:SCAN HDFS 1 6.251ms 6.251ms 71 7 84.00 KB 64.00 MB trg.table1 tp
19:HASH JOIN 5 45.483ms 57.714ms 811.66K 3.62M 2.10 MB 103.00 B LEFT OUTER JOIN, BROADCAST
|--34:EXCHANGE 5 15.827us 17.612us 441 2 0 0 BROADCAST
| 10:SCAN HDFS 1 23.226ms 23.226ms 441 2 115.00 KB 80.00 MB trg.table2 o
18:HASH JOIN 5 81.817ms 103.238ms 811.66K 3.62M 2.27 MB 2.53 KB LEFT OUTER JOIN, BROADCAST
|--33:EXCHANGE 5 10.905us 16.528us 116 78 0 0 BROADCAST
| 09:SCAN HDFS 1 21.119ms 21.119ms 116 78 77.00 KB 48.00 MB trg.table3...
17:HASH JOIN 5 238.710ms 310.216ms 810.69K 3.62M 3.09 MB 34.79 KB LEFT OUTER JOIN, BROADCAST
|--32:EXCHANGE 5 14.653us 17.272us 506 506 0 0 BROADCAST
| 08:SCAN HDFS 1 5.079ms 5.079ms 506 506 121.17 KB 48.00 MB trg.table4 s1
16:HASH JOIN 5 37.020ms 48.938ms 810.69K 3.62M 2.06 MB 154.41 KB LEFT OUTER JOIN, BROADCAST
|--31:EXCHANGE 5 7.393us 12.096us 4 3.78K 0 0 BROADCAST
| 07:SCAN HDFS 1 6.101ms 6.101ms 4 3.78K 958.15 KB 64.00 MB tmp.table8 s2
15:HASH JOIN 5 21.639ms 29.371ms 810.69K 3.62M 2.05 MB 154.41 KB LEFT OUTER JOIN, BROADCAST
|--30:EXCHANGE 5 8.122us 11.664us 1 3.78K 0 0 BROADCAST
| 06:SCAN HDFS 1 8.067ms 8.067ms 1 3.78K 958.15 KB 64.00 MB tmp.table8 s3
14:HASH JOIN 5 82.441ms 99.157ms 810.69K 3.62M 2.53 MB 154.41 KB LEFT OUTER JOIN, BROADCAST
|--29:EXCHANGE 5 220.902us 235.376us 11.35K 3.78K 0 0 BROADCAST
| 05:SCAN HDFS 1 5.741ms 5.741ms 11.35K 3.78K 1.09 MB 64.00 MB tmp.table8 s4
13:HASH JOIN 5 254.708ms 304.909ms 810.69K 3.62M 152.05 MB 8.64 MB RIGHT OUTER JOIN, PARTITIONED
|--28:EXCHANGE 5 40.477ms 58.477ms 810.69K 82.11K 0 0 HASH(CAST(d.entid AS INT))
| 12:HASH JOIN 7 226.257ms 359.085ms 810.69K 82.11K 2.09 MB 184.00 B LEFT OUTER JOIN, BROADCAST
| |--26:EXCHANGE 7 9.622us 11.480us 9 9 0 0 BROADCAST
| | 03:SCAN HDFS 1 4.366ms 4.366ms 9 9 47.00 KB 32.00 MB trg.table5 ro
| 02:HASH JOIN 7 23.812ms 35.389ms 810.69K 82.11K 2.20 MB 2.16 MB LEFT OUTER JOIN, PARTITIONED
| |--25:EXCHANGE 7 91.400us 131.127us 10.44K 82.11K 0 0 HASH(d_sur.reference,d_sur....
| | 01:SCAN HDFS 7 410.599ms 520.945ms 10.44K 82.11K 13.21 MB 168.00 MB tmp.table8_detail d_sur
| 24:EXCHANGE 7 14.274ms 20.817ms 810.69K 82.11K 0 0 HASH(d.reference,d.mnid,d...
| 00:SCAN HDFS 7 495.925ms 697.044ms 810.69K 82.11K 52.63 MB 1.08 GB tmp.table8_detail d
27:EXCHANGE 5 26.006ms 34.625ms 3.62M 3.62M 0 0 HASH(c.intid)
04:SCAN HDFS 5 239.460ms 369.856ms 3.62M 3.62M 35.71 MB 48.00 MB tmp.table7 c Based on this explain summary some would say 1.08GB is a peak, so 1.5GB limit would be safe. But not even close. Any thoughts where to look for a real consumption? Thanks
... View more
Labels:
- Labels:
-
Apache Impala
02-08-2017
05:26 AM
It looks like the problem is really in the timestamp field. Running a similar query on table without timestamp show much better results on the new environment. Thanks for the help
... View more
02-03-2017
01:24 AM
I tried to disable prefetch and streaming, but did not helped. Here is the profile of old Impala on old cluster: +---------------------+----------+
| part_col | count(*) |
+---------------------+----------+
| 2017-01-01 00:00:00 | 18428899 |
| 2017-01-02 00:00:00 | 20974112 |
| 2017-01-03 00:00:00 | 13178213 |
| 2017-01-04 00:00:00 | 26768773 |
| 2017-01-05 00:00:00 | 24148741 |
+---------------------+----------+
Query Runtime Profile:
Query (id=a7442ed1a43b8e8d:8ad38fb17f29baa0):
Summary:
Session ID: f94cc4d4c075f287:175ee4069895a6b0
Session Type: BEESWAX
Start Time: 2017-01-31 09:07:47.514559000
End Time: 2017-01-31 09:07:51.159487000
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 2.2.0-cdh5.4.8 RELEASE (build 137d99e9f751c454a0c79f3b00302938e4984f9c)
User: hadoop
Connected User: hadoop
Delegated User:
Network Address: ::ffff:xx.xx.xx.xx:33929
Default Db: default
Sql Statement: select part_col, count(*) from perf_test group by part_col order by 1
Coordinator: worker2.some.domain.net:22000
Plan:
----------------
Estimated Per-Host Requirements: Memory=256.00MB VCores=2
WARNING: The following tables are missing relevant table and/or column statistics.
default.perf_test
F02:PLAN FRAGMENT [UNPARTITIONED]
05:MERGING-EXCHANGE [UNPARTITIONED]
order by: part_col ASC
hosts=4 per-host-mem=unavailable
tuple-ids=2 row-size=24B cardinality=unavailable
F01:PLAN FRAGMENT [HASH(part_col)]
DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=05, UNPARTITIONED]
02:SORT
| order by: part_col ASC
| hosts=4 per-host-mem=0B
| tuple-ids=2 row-size=24B cardinality=unavailable
|
04:AGGREGATE [FINALIZE]
| output: count:merge(*)
| group by: part_col
| hosts=4 per-host-mem=128.00MB
| tuple-ids=1 row-size=24B cardinality=unavailable
|
03:EXCHANGE [HASH(part_col)]
hosts=4 per-host-mem=0B
tuple-ids=1 row-size=24B cardinality=unavailable
F00:PLAN FRAGMENT [RANDOM]
DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=03, HASH(part_col)]
01:AGGREGATE
| output: count(*)
| group by: part_col
| hosts=4 per-host-mem=128.00MB
| tuple-ids=1 row-size=24B cardinality=unavailable
|
00:SCAN HDFS [default.perf_test, RANDOM]
partitions=1/1 files=31 size=6.89GB
table stats: unavailable
column stats: unavailable
hosts=4 per-host-mem=88.00MB
tuple-ids=0 row-size=16B cardinality=unavailable
----------------
Estimated Per-Host Mem: 268435456
Estimated Per-Host VCores: 2
Tables Missing Stats: default.perf_test
Admission result: Admitted immediately
Request Pool: root.hadoop
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
---------------------------------------------------------------------------------------------------------------------
05:MERGING-EXCHANGE 1 119.314us 119.314us 5 -1 0 -1.00 B UNPARTITIONED
02:SORT 4 435.85us 588.641us 5 -1 8.02 MB 0
04:AGGREGATE 4 255.672ms 272.347ms 5 -1 3.15 MB 128.00 MB FINALIZE
03:EXCHANGE 4 62.142us 87.60us 20 -1 0 0 HASH(part_col)
01:AGGREGATE 4 2s803ms 3s270ms 20 -1 3.17 MB 128.00 MB
00:SCAN HDFS 4 53.678ms 56.787ms 103.50M -1 1.82 MB 88.00 MB default.perf_test
Planner Timeline: 3.584ms
- Analysis finished: 1.132ms (1.132ms)
- Equivalence classes computed: 1.236ms (103.831us)
- Single node plan created: 1.823ms (587.16us)
- Distributed plan created: 2.300ms (477.253us)
- Lineage info computed: 2.664ms (363.854us)
- Planning finished: 3.584ms (919.713us)
Query Timeline: 3s649ms
- Start execution: 42.805us (42.805us)
- Planning finished: 4.458ms (4.415ms)
- Submit for admission: 4.775ms (317.243us)
- Completed admission: 4.838ms (63.556us)
- Ready to start remote fragments: 5.352ms (514.114us)
- Remote fragments started: 511.206ms (505.853ms)
- Rows available: 3s612ms (3s101ms)
- First row fetched: 3s639ms (26.709ms)
- Unregister query: 3s644ms (5.608ms)
ImpalaServer:
- ClientFetchWaitTimer: 30.950ms
- RowMaterializationTimer: 72.117us
Execution Profile a7442ed1a43b8e8d:8ad38fb17f29baa0:(Total: 3s608ms, non-child: 0ns, % non-child: 0.00%)
Fragment start latencies: count: 8, last: 0.231475ns, min: 0.160659ns, max: 0.273263ns, mean: 0.221710ns, stddev: 0.041526ns
Per Node Peak Memory Usage: worker4.some.domain.net:22000(8.41 MB) worker1.some.domain.net:22000(8.10 MB) worker3.some.domain.net:22000(8.41 MB) worker2.some.domain.net:22000(8.23 MB)
- FinalizationTimer: 0ns
Coordinator Fragment F02:(Total: 3s100ms, non-child: 251.349us, % non-child: 0.01%)
MemoryUsage(500.0ms): 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB
ThreadUsage(500.0ms): 1
- AverageThreadTokens: 1.00
- PeakMemoryUsage: 24.00 KB (24576)
- PerHostPeakMemUsage: 0
- PrepareTime: 49.521us
- RowsProduced: 5 (5)
- TotalCpuTime: 3s634ms
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 0ns
- TotalStorageWaitTime: 0ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 96 (96)
- BlocksRecycled: 1 (1)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 38.40 GB (41231687680)
- PeakMemoryUsage: 8.01 MB (8396800)
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
EXCHANGE_NODE (id=5):(Total: 3s100ms, non-child: 119.314us, % non-child: 0.00%)
BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0
- BytesReceived: 134.00 B (134)
- ConvertRowBatchTime: 0ns
- DeserializeRowBatchTimer: 31.363us
- FirstBatchArrivalWaitTime: 3s100ms
- MergeGetNext: 51.896us
- MergeGetNextBatch: 45.734us
- PeakMemoryUsage: 0
- RowsReturned: 5 (5)
- RowsReturnedRate: 1.00 /sec
- SendersBlockedTimer: 0ns
- SendersBlockedTotalTimer(*): 0ns
Averaged Fragment F01:(Total: 3s604ms, non-child: 0ns, % non-child: 0.00%)
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:3s334ms max:3s367ms mean: 3s350ms stddev:15.818906.818.905508ms
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec
num instances: 4
- AverageThreadTokens: 1.00
- PeakMemoryUsage: 8.26 MB (8665088)
- PerHostPeakMemUsage: 8.29 MB (8689408)
- PrepareTime: 255.347ms
- RowsProduced: 1 (1)
- TotalCpuTime: 257.307ms
- TotalNetworkReceiveTime: 3s347ms
- TotalNetworkSendTime: 314.714us
- TotalStorageWaitTime: 0ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 96 (96)
- BlocksRecycled: 1 (1)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 38.40 GB (41231687680)
- PeakMemoryUsage: 8.01 MB (8399530)
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
CodeGen:(Total: 254.686ms, non-child: 254.686ms, % non-child: 100.00%)
- CodegenTime: 478.430us
- CompileTime: 323ns
- LoadTime: 117.38us
- ModuleFileSize: 2.23 MB (2333204)
- OptimizationTime: 0ns
- PrepareTime: 252.960ms
DataStreamSender (dst_id=5):(Total: 40.576us, non-child: 40.576us, % non-child: 100.00%)
- BytesSent: 33.00 B (33)
- NetworkThroughput(*): 111.39 KB/sec
- OverallThroughput: 713.32 KB/sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 14.713us
- ThriftTransmitTime(*): 233.563us
- UncompressedRowBatchSize: 48.00 B (48)
SORT_NODE (id=2):(Total: 3s603ms, non-child: 435.86us, % non-child: 0.01%)
- InMemorySortTime: 2.144us
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 8.01 MB (8402944)
- RowsReturned: 1 (1)
- RowsReturnedRate: 0
- SortDataSize: 40.00 B (40)
- TotalMergesPerformed: 0 (0)
AGGREGATION_NODE (id=4):(Total: 3s603ms, non-child: 255.672ms, % non-child: 7.10%)
- BuildTime: 21.426us
- GetNewBlockTime: 55.516us
- GetResultsTime: 2.938us
- HashBuckets: 1.28K (1280)
- LargestPartitionPercent: 12 (12)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.14 MB (3297280)
- PinTime: 0ns
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
- UnpinTime: 870ns
EXCHANGE_NODE (id=3):(Total: 3s347ms, non-child: 3s347ms, % non-child: 100.00%)
- BytesReceived: 134.00 B (134)
- ConvertRowBatchTime: 8.18us
- DeserializeRowBatchTimer: 31.103us
- FirstBatchArrivalWaitTime: 2s437ms
- PeakMemoryUsage: 0
- RowsReturned: 5 (5)
- RowsReturnedRate: 1.00 /sec
- SendersBlockedTimer: 0ns
- SendersBlockedTotalTimer(*): 0ns
Averaged Fragment F00:(Total: 2s857ms, non-child: 0ns, % non-child: 0.00%)
split sizes: min: 1.27 GB, max: 1.92 GB, avg: 1.72 GB, stddev: 272.90 MB
completion times: min:1s886ms max:3s146ms mean: 2s675ms stddev:477.663985.663.984856ms
execution rates: min:624.97 MB/sec max:690.60 MB/sec mean:663.26 MB/sec stddev:23.89 MB/sec
num instances: 4
- AverageThreadTokens: 7.44
- PeakMemoryUsage: 4.93 MB (5166592)
- PerHostPeakMemUsage: 8.07 MB (8457728)
- PrepareTime: 185.387ms
- RowsProduced: 5 (5)
- TotalCpuTime: 22s474ms
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 1.299ms
- TotalStorageWaitTime: 87.376ms
CodeGen:(Total: 184.807ms, non-child: 184.807ms, % non-child: 100.00%)
- CodegenTime: 290.866us
- CompileTime: 233ns
- LoadTime: 45.102us
- ModuleFileSize: 2.23 MB (2333204)
- OptimizationTime: 0ns
- PrepareTime: 183.547ms
DataStreamSender (dst_id=3):(Total: 24.337us, non-child: 24.337us, % non-child: 100.00%)
- BytesSent: 134.00 B (134)
- NetworkThroughput(*): 120.91 KB/sec
- OverallThroughput: 5.27 MB/sec
- PeakMemoryUsage: 40.00 KB (40960)
- SerializeBatchTime: 23.984us
- ThriftTransmitTime(*): 1.114ms
- UncompressedRowBatchSize: 192.00 B (192)
AGGREGATION_NODE (id=1):(Total: 2s856ms, non-child: 2s803ms, % non-child: 98.12%)
- BuildTime: 2s601ms
- GetNewBlockTime: 44.889us
- GetResultsTime: 4.697us
- HashBuckets: 4.10K (4096)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.17 MB (3325952)
- PinTime: 0ns
- RowsRepartitioned: 0 (0)
- RowsReturned: 5 (5)
- RowsReturnedRate: 1.00 /sec
- SpilledPartitions: 0 (0)
- UnpinTime: 792ns
HDFS_SCAN_NODE (id=0):(Total: 53.678ms, non-child: 53.678ms, % non-child: 100.00%)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 6.65
- BytesRead: 777.91 KB (796579)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 777.91 KB (796579)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 777.91 KB (796579)
- DecompressionTime: 136.633us
- MaxCompressedTextFileLength: 0
- NumColumns: 1 (1)
- NumDisksAccessed: 1 (1)
- NumScannerThreadsStarted: 7 (7)
- PeakMemoryUsage: 1.74 MB (1824256)
- PerReadThreadRawHdfsThroughput: 748.79 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 25.87M (25874684)
- RowsReturned: 25.87M (25874684)
- RowsReturnedRate: 479.03 M/sec
- ScanRangesComplete: 7 (7)
- ScannerThreadsInvoluntaryContextSwitches: 10 (10)
- ScannerThreadsTotalWallClockTime: 19s703ms
- MaterializeTupleTime(*): 19s566ms
- ScannerThreadsSysTime: 24.0ms
- ScannerThreadsUserTime: 1s204ms
- ScannerThreadsVoluntaryContextSwitches: 25.27K (25265)
- TotalRawHdfsReadTime(*): 1.19ms
- TotalReadThroughput: 274.84 KB/sec
Fragment F01:
Instance a7442ed1a43b8e8d:8ad38fb17f29baa5 (host=worker4.some.domain.net:22000):(Total: 3s605ms, non-child: 0ns, % non-child: 0.00%)
MemoryUsage(500.0ms): 0, 3.14 MB, 3.14 MB, 3.14 MB, 3.14 MB, 3.15 MB, 3.15 MB
ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.00
- PeakMemoryUsage: 8.41 MB (8818688)
- PerHostPeakMemUsage: 8.41 MB (8818688)
- PrepareTime: 271.989ms
- RowsProduced: 2 (2)
- TotalCpuTime: 273.763ms
- TotalNetworkReceiveTime: 3s332ms
- TotalNetworkSendTime: 359.786us
- TotalStorageWaitTime: 0ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 96 (96)
- BlocksRecycled: 1 (1)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 38.40 GB (41231687680)
- PeakMemoryUsage: 8.02 MB (8404992)
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
CodeGen:(Total: 271.331ms, non-child: 271.331ms, % non-child: 100.00%)
- CodegenTime: 437.945us
- CompileTime: 267ns
- LoadTime: 132.3us
- ModuleFileSize: 2.23 MB (2333204)
- OptimizationTime: 0ns
- PrepareTime: 269.916ms
DataStreamSender (dst_id=5):(Total: 41.665us, non-child: 41.665us, % non-child: 100.00%)
- BytesSent: 53.00 B (53)
- NetworkThroughput(*): 214.21 KB/sec
- OverallThroughput: 1.21 MB/sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 16.673us
- ThriftTransmitTime(*): 241.624us
- UncompressedRowBatchSize: 76.00 B (76)
SORT_NODE (id=2):(Total: 3s604ms, non-child: 313.224us, % non-child: 0.01%)
- InMemorySortTime: 2.746us
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 8.02 MB (8404992)
- RowsReturned: 2 (2)
- RowsReturnedRate: 0
- SortDataSize: 64.00 B (64)
- TotalMergesPerformed: 0 (0)
AGGREGATION_NODE (id=4):(Total: 3s604ms, non-child: 272.347ms, % non-child: 7.56%)
- BuildTime: 28.572us
- GetNewBlockTime: 86.550us
- GetResultsTime: 5.612us
- HashBuckets: 2.05K (2048)
- LargestPartitionPercent: 12 (12)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.15 MB (3301376)
- PinTime: 0ns
- RowsRepartitioned: 0 (0)
- RowsReturned: 2 (2)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
- UnpinTime: 827ns
EXCHANGE_NODE (id=3):(Total: 3s332ms, non-child: 69.672us, % non-child: 0.00%)
BytesReceived(500.0ms): 0, 0, 0, 0, 52.00 B, 52.00 B
- BytesReceived: 208.00 B (208)
- ConvertRowBatchTime: 8.324us
- DeserializeRowBatchTimer: 44.682us
- FirstBatchArrivalWaitTime: 2s117ms
- PeakMemoryUsage: 0
- RowsReturned: 8 (8)
- RowsReturnedRate: 2.00 /sec
- SendersBlockedTimer: 0ns
- SendersBlockedTotalTimer(*): 0ns
Instance a7442ed1a43b8e8d:8ad38fb17f29baa3 (host=worker2.some.domain.net:22000):(Total: 3s604ms, non-child: 0ns, % non-child: 0.00%)
MemoryUsage(500.0ms): 3.14 MB, 3.14 MB, 3.14 MB, 3.14 MB, 3.14 MB, 3.15 MB, 3.16 MB
ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.00
- PeakMemoryUsage: 8.21 MB (8613888)
- PerHostPeakMemUsage: 8.23 MB (8626176)
- PrepareTime: 238.304ms
- RowsProduced: 1 (1)
- TotalCpuTime: 240.469ms
- TotalNetworkReceiveTime: 3s364ms
- TotalNetworkSendTime: 332.998us
- TotalStorageWaitTime: 0ns
CodeGen:(Total: 237.499ms, non-child: 237.499ms, % non-child: 100.00%)
- CodegenTime: 621.766us
- CompileTime: 524ns
- LoadTime: 61.381us
- ModuleFileSize: 2.23 MB (2333204)
- OptimizationTime: 0ns
- PrepareTime: 234.905ms
DataStreamSender (dst_id=5):(Total: 62.686us, non-child: 62.686us, % non-child: 100.00%)
- BytesSent: 31.00 B (31)
- NetworkThroughput(*): 121.00 KB/sec
- OverallThroughput: 482.94 KB/sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 25.91us
- ThriftTransmitTime(*): 250.198us
- UncompressedRowBatchSize: 40.00 B (40)
SORT_NODE (id=2):(Total: 3s604ms, non-child: 588.641us, % non-child: 0.02%)
- InMemorySortTime: 1.479us
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 8.02 MB (8404992)
- RowsReturned: 1 (1)
- RowsReturnedRate: 0
- SortDataSize: 32.00 B (32)
- TotalMergesPerformed: 0 (0)
AGGREGATION_NODE (id=4):(Total: 3s603ms, non-child: 238.727ms, % non-child: 6.62%)
- BuildTime: 30.821us
- GetNewBlockTime: 57.183us
- GetResultsTime: 3.80us
- HashBuckets: 1.02K (1024)
- LargestPartitionPercent: 25 (25)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.15 MB (3301376)
- PinTime: 0ns
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
- UnpinTime: 1.96us
EXCHANGE_NODE (id=3):(Total: 3s364ms, non-child: 87.60us, % non-child: 0.00%)
BytesReceived(500.0ms): 0, 0, 0, 0, 0, 30.00 B, 90.00 B
- BytesReceived: 120.00 B (120)
- ConvertRowBatchTime: 11.349us
- DeserializeRowBatchTimer: 44.852us
- FirstBatchArrivalWaitTime: 2s150ms
- PeakMemoryUsage: 0
- RowsReturned: 4 (4)
- RowsReturnedRate: 1.00 /sec
- SendersBlockedTimer: 0ns
- SendersBlockedTotalTimer(*): 0ns
Instance a7442ed1a43b8e8d:8ad38fb17f29baa4 (host=worker1.some.domain.net:22000):(Total: 3s604ms, non-child: 0ns, % non-child: 0.00%)
MemoryUsage(500.0ms): 0, 3.14 MB, 3.14 MB, 3.14 MB, 3.14 MB, 3.14 MB, 3.14 MB, 3.14 MB
ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.00
- PeakMemoryUsage: 8.02 MB (8409088)
- PerHostPeakMemUsage: 8.10 MB (8494080)
- PrepareTime: 240.672ms
- RowsProduced: 0 (0)
- TotalCpuTime: 242.522ms
- TotalNetworkReceiveTime: 3s362ms
- TotalNetworkSendTime: 230ns
- TotalStorageWaitTime: 0ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 96 (96)
- BlocksRecycled: 1 (1)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 38.40 GB (41231687680)
- PeakMemoryUsage: 8.00 MB (8388608)
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
CodeGen:(Total: 240.75ms, non-child: 240.75ms, % non-child: 100.00%)
- CodegenTime: 430.880us
- CompileTime: 253ns
- LoadTime: 153.349us
- ModuleFileSize: 2.23 MB (2333204)
- OptimizationTime: 0ns
- PrepareTime: 238.587ms
DataStreamSender (dst_id=5):(Total: 14.671us, non-child: 14.671us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
SORT_NODE (id=2):(Total: 3s603ms, non-child: 364.504us, % non-child: 0.01%)
- InMemorySortTime: 1.118us
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 8.01 MB (8396800)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- SortDataSize: 0
- TotalMergesPerformed: 0 (0)
AGGREGATION_NODE (id=4):(Total: 3s603ms, non-child: 240.841ms, % non-child: 6.68%)
- BuildTime: 1.836us
- GetNewBlockTime: 39.494us
- GetResultsTime: 0ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.13 MB (3284992)
- PinTime: 0ns
- RowsRepartitioned: 0 (0)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
- UnpinTime: 686ns
EXCHANGE_NODE (id=3):(Total: 3s362ms, non-child: 5.944us, % non-child: 0.00%)
BytesReceived(500.0ms): 0, 0, 0, 0, 0, 0, 0
- BytesReceived: 0
- ConvertRowBatchTime: 817ns
- DeserializeRowBatchTimer: 0ns
- FirstBatchArrivalWaitTime: 3s362ms
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- SendersBlockedTimer: 0ns
- SendersBlockedTotalTimer(*): 0ns
Instance a7442ed1a43b8e8d:8ad38fb17f29baa2 (host=worker3.some.domain.net:22000):(Total: 3s603ms, non-child: 0ns, % non-child: 0.00%)
MemoryUsage(500.0ms): 0, 3.14 MB, 3.14 MB, 3.14 MB, 3.14 MB, 3.15 MB, 3.15 MB, 3.16 MB
ThreadUsage(500.0ms): 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.00
- PeakMemoryUsage: 8.41 MB (8818688)
- PerHostPeakMemUsage: 8.41 MB (8818688)
- PrepareTime: 270.423ms
- RowsProduced: 2 (2)
- TotalCpuTime: 272.476ms
- TotalNetworkReceiveTime: 3s331ms
- TotalNetworkSendTime: 565.845us
- TotalStorageWaitTime: 0ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 96 (96)
- BlocksRecycled: 1 (1)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 38.40 GB (41231687680)
- PeakMemoryUsage: 8.02 MB (8404992)
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
CodeGen:(Total: 269.840ms, non-child: 269.840ms, % non-child: 100.00%)
- CodegenTime: 423.130us
- CompileTime: 250ns
- LoadTime: 121.419us
- ModuleFileSize: 2.23 MB (2333204)
- OptimizationTime: 0ns
- PrepareTime: 268.433ms
DataStreamSender (dst_id=5):(Total: 43.283us, non-child: 43.283us, % non-child: 100.00%)
- BytesSent: 50.00 B (50)
- NetworkThroughput(*): 110.36 KB/sec
- OverallThroughput: 1.10 MB/sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 17.90us
- ThriftTransmitTime(*): 442.430us
- UncompressedRowBatchSize: 76.00 B (76)
SORT_NODE (id=2):(Total: 3s602ms, non-child: 473.974us, % non-child: 0.01%)
- InMemorySortTime: 3.235us
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 8.02 MB (8404992)
- RowsReturned: 2 (2)
- RowsReturnedRate: 0
- SortDataSize: 64.00 B (64)
- TotalMergesPerformed: 0 (0)
AGGREGATION_NODE (id=4):(Total: 3s602ms, non-child: 270.772ms, % non-child: 7.52%)
- BuildTime: 24.478us
- GetNewBlockTime: 38.840us
- GetResultsTime: 3.61us
- HashBuckets: 2.05K (2048)
- LargestPartitionPercent: 12 (12)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 3.15 MB (3301376)
- PinTime: 0ns
- RowsRepartitioned: 0 (0)
- RowsReturned: 2 (2)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
- UnpinTime: 874ns
EXCHANGE_NODE (id=3):(Total: 3s331ms, non-child: 85.894us, % non-child: 0.00%)
BytesReceived(500.0ms): 0, 0, 0, 0, 52.00 B, 52.00 B, 156.00 B
- BytesReceived: 208.00 B (208)
- ConvertRowBatchTime: 11.584us
- DeserializeRowBatchTimer: 34.881us
- FirstBatchArrivalWaitTime: 2s117ms
- PeakMemoryUsage: 0
- RowsReturned: 8 (8)
- RowsReturnedRate: 2.00 /sec
- SendersBlockedTimer: 0ns
- SendersBlockedTotalTimer(*): 0ns
Fragment F00:
Instance a7442ed1a43b8e8d:8ad38fb17f29baa8 (host=worker1.some.domain.net:22000):(Total: 3s327ms, non-child: 0ns, % non-child: 0.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:8/1.92 GB
MemoryUsage(500.0ms): 4.73 MB, 4.76 MB, 4.77 MB, 4.76 MB, 4.76 MB, 4.73 MB, 4.23 MB The new cluster: +---------------------+----------+
| part_col | count(*) |
+---------------------+----------+
| 2017-01-01 00:00:00 | 18428899 |
| 2017-01-02 00:00:00 | 20974112 |
| 2017-01-03 00:00:00 | 13178213 |
| 2017-01-04 00:00:00 | 26768773 |
| 2017-01-05 00:00:00 | 24148741 |
+---------------------+----------+
Query Runtime Profile:
Query (id=814dc21ba8355459:ac7e36724f454893):
Summary:
Session ID: 694ced478c51b5e5:321e569a7a0513b5
Session Type: BEESWAX
Start Time: 2017-01-31 09:08:07.217250000
End Time: 2017-01-31 09:08:12.161790000
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 2.6.0-cdh5.8.3 RELEASE (build c644f476b774db9db87a619628f7a6ecc5f843e0)
User: test_user@SOME.DOMAIN.NET
Connected User: test_user@SOME.DOMAIN.NET
Delegated User:
Network Address: ::ffff:xx.xx.xx.xx:55766
Default Db: default
Sql Statement: select part_col, count(*) from perf_test group by part_col order by 1
Coordinator: worker3.some.domain.net:22000
Query Options (non default):
Plan:
----------------
Estimated Per-Host Requirements: Memory=344.00MB VCores=2
WARNING: The following tables are missing relevant table and/or column statistics.
default.perf_test
05:MERGING-EXCHANGE [UNPARTITIONED]
| order by: part_col ASC
| hosts=5 per-host-mem=unavailable
| tuple-ids=2 row-size=24B cardinality=unavailable
|
02:SORT
| order by: part_col ASC
| hosts=5 per-host-mem=0B
| tuple-ids=2 row-size=24B cardinality=unavailable
|
04:AGGREGATE [FINALIZE]
| output: count:merge(*)
| group by: part_col
| hosts=5 per-host-mem=128.00MB
| tuple-ids=1 row-size=24B cardinality=unavailable
|
03:EXCHANGE [HASH(part_col)]
| hosts=5 per-host-mem=0B
| tuple-ids=1 row-size=24B cardinality=unavailable
|
01:AGGREGATE [STREAMING]
| output: count(*)
| group by: part_col
| hosts=5 per-host-mem=128.00MB
| tuple-ids=1 row-size=24B cardinality=unavailable
|
00:SCAN HDFS [default.perf_test, RANDOM]
partitions=1/1 files=32 size=7.22GB
table stats: unavailable
column stats: unavailable
hosts=5 per-host-mem=88.00MB
tuple-ids=0 row-size=16B cardinality=unavailable
----------------
Estimated Per-Host Mem: 360710144
Estimated Per-Host VCores: 2
Tables Missing Stats: default.perf_test
Request Pool: root.test_user
Admission result: Admitted immediately
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
---------------------------------------------------------------------------------------------------------------------
05:MERGING-EXCHANGE 1 82.553us 82.553us 5 -1 0 -1.00 B UNPARTITIONED
02:SORT 5 434.217us 491.025us 5 -1 8.02 MB 0
04:AGGREGATE 5 279.461ms 316.351ms 5 -1 2.30 MB 128.00 MB FINALIZE
03:EXCHANGE 5 28.378us 55.658us 25 -1 0 0 HASH(part_col)
01:AGGREGATE 5 3s965ms 4s739ms 25 -1 1.59 MB 128.00 MB STREAMING
00:SCAN HDFS 5 83.393ms 118.345ms 103.50M -1 12.04 MB 88.00 MB default.perf_test
Planner Timeline: 23.087ms
- Analysis finished: 10.147ms (10.147ms)
- Equivalence classes computed: 10.654ms (506.930us)
- Single node plan created: 14.745ms (4.091ms)
- Runtime filters computed: 14.787ms (41.721us)
- Distributed plan created: 17.268ms (2.481ms)
- Lineage info computed: 18.728ms (1.460ms)
- Planning finished: 23.087ms (4.358ms)
Query Timeline: 4s949ms
- Start execution: 56.064us (56.064us)
- Planning finished: 26.811ms (26.755ms)
- Submit for admission: 27.851ms (1.039ms)
- Completed admission: 27.929ms (77.878us)
- Ready to start 10 remote fragments: 28.364ms (434.946us)
- All 10 remote fragments started: 29.638ms (1.273ms)
- Rows available: 4s860ms (4s831ms)
- First row fetched: 4s942ms (81.424ms)
- Unregister query: 4s944ms (2.126ms)
ImpalaServer:
- ClientFetchWaitTimer: 82.476ms
- RowMaterializationTimer: 51.064us
Execution Profile 814dc21ba8355459:ac7e36724f454893:(Total: 4s833ms, non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Targets Target type Partition filter Pending (Expected) First arrived Completed
-------------------------------------------------------------------------------------------------------------------
Fragment start latencies: Count: 10, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms
Per Node Peak Memory Usage: worker2.some.domain.net:22000(23.93 MB) worker5.some.domain.net:22000(23.32 MB) worker1.some.domain.net:22000(23.85 MB) worker4.some.domain.net:22000(23.94 MB) worker3.some.domain.net:22000(23.86 MB)
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
Coordinator Fragment F02:(Total: 4s831ms, non-child: 255.854us, % non-child: 0.01%)
MemoryUsage(500.000ms): 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 20.04 KB (20520)
- PerHostPeakMemUsage: 0
- PrepareTime: 45.494us
- RowsProduced: 5 (5)
- TotalCpuTime: 4s914ms
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 49 (49)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 61.98 GB (66552700928)
- PeakMemoryUsage: 8.77 MB (9191424)
- TotalBufferWaitTime: 0.000ns
- TotalEncryptionTime: 0.000ns
- TotalIntegrityCheckTime: 0.000ns
- TotalReadBlockTime: 0.000ns
EXCHANGE_NODE (id=5):(Total: 4s831ms, non-child: 82.553us, % non-child: 0.00%)
BytesReceived(500.000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0
- BytesReceived: 134.00 B (134)
- ConvertRowBatchTime: 0.000ns
- DeserializeRowBatchTimer: 31.107us
- FirstBatchArrivalWaitTime: 4s830ms
- MergeGetNext: 25.594us
- MergeGetNextBatch: 15.618us
- PeakMemoryUsage: 0
- RowsReturned: 5 (5)
- RowsReturnedRate: 1.00 /sec
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
Averaged Fragment F01:(Total: 4s721ms, non-child: 0.000ns, % non-child: 0.00%)
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:4s831ms max:4s832ms mean: 4s832ms stddev:302.637us
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec
num instances: 5
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 10.30 MB (10802320)
- PerHostPeakMemUsage: 23.78 MB (24934251)
- PrepareTime: 278.764ms
- RowsProduced: 1 (1)
- TotalCpuTime: 390.687ms
- TotalNetworkReceiveTime: 4s440ms
- TotalNetworkSendTime: 107.306us
- TotalStorageWaitTime: 0.000ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 49 (49)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 61.98 GB (66552700928)
- PeakMemoryUsage: 8.77 MB (9191424)
- TotalBufferWaitTime: 0.000ns
- TotalEncryptionTime: 0.000ns
- TotalIntegrityCheckTime: 0.000ns
- TotalReadBlockTime: 0.000ns
CodeGen:(Total: 381.581ms, non-child: 381.581ms, % non-child: 100.00%)
- CodegenTime: 6.404ms
- CompileTime: 7.476ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.15 MB (2254856)
- NumFunctions: 29 (29)
- NumInstructions: 550 (550)
- OptimizationTime: 102.403ms
- PrepareTime: 270.719ms
DataStreamSender (dst_id=5):(Total: 68.002us, non-child: 68.002us, % non-child: 100.00%)
- BytesSent: 26.00 B (26)
- NetworkThroughput(*): 787.80 KB/sec
- OverallThroughput: 295.48 KB/sec
- PeakMemoryUsage: 4.00 KB (4096)
- RowsReturned: 1 (1)
- SerializeBatchTime: 19.405us
- TransmitDataRPCTime: 21.326us
- UncompressedRowBatchSize: 38.00 B (38)
SORT_NODE (id=2):(Total: 4s720ms, non-child: 434.218us, % non-child: 0.01%)
- InMemorySortTime: 1.484us
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 8.01 MB (8401715)
- RowsReturned: 1 (1)
- RowsReturnedRate: 0
- SortDataSize: 32.00 B (32)
- SpilledRuns: 0 (0)
- TotalMergesPerformed: 0 (0)
AGGREGATION_NODE (id=4):(Total: 4s720ms, non-child: 279.461ms, % non-child: 5.92%)
- BuildTime: 13.858us
- GetNewBlockTime: 145.320us
- GetResultsTime: 1.887us
- HTResizeTime: 3.177us
- HashBuckets: 1.02K (1024)
- LargestPartitionPercent: 8 (8)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 2.29 MB (2401408)
- PinTime: 0.000ns
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
- UnpinTime: 1.046us
EXCHANGE_NODE (id=3):(Total: 4s440ms, non-child: 4s440ms, % non-child: 100.00%)
- BytesReceived: 133.00 B (133)
- ConvertRowBatchTime: 4.099us
- DeserializeRowBatchTimer: 27.477us
- FirstBatchArrivalWaitTime: 2s625ms
- PeakMemoryUsage: 0
- RowsReturned: 5 (5)
- RowsReturnedRate: 1.00 /sec
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
Averaged Fragment F00:(Total: 4s051ms, non-child: 0.000ns, % non-child: 0.00%)
split sizes: min: 1.33 GB, max: 1.54 GB, avg: 1.44 GB, stddev: 78.16 MB
completion times: min:1s806ms max:4s831ms mean: 4s052ms stddev:1s134ms
execution rates: min:317.74 MB/sec max:756.58 MB/sec mean:412.43 MB/sec stddev:172.21 MB/sec
num instances: 5
- AverageThreadTokens: 6.25
- BloomFilterBytes: 0
- PeakMemoryUsage: 13.49 MB (14146688)
- PerHostPeakMemUsage: 23.78 MB (24934251)
- PrepareTime: 281.243ms
- RowsProduced: 5 (5)
- TotalCpuTime: 26s501ms
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 506.296us
- TotalStorageWaitTime: 136.659ms
CodeGen:(Total: 274.654ms, non-child: 274.654ms, % non-child: 100.00%)
- CodegenTime: 6.016ms
- CompileTime: 0.000ns
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.15 MB (2254856)
- NumFunctions: 0 (0)
- NumInstructions: 0 (0)
- OptimizationTime: 0.000ns
- PrepareTime: 273.688ms
DataStreamSender (dst_id=3):(Total: 147.360us, non-child: 147.360us, % non-child: 100.00%)
- BytesSent: 133.00 B (133)
- NetworkThroughput(*): 1.84 MB/sec
- OverallThroughput: 885.74 KB/sec
- PeakMemoryUsage: 32.00 KB (32768)
- RowsReturned: 5 (5)
- SerializeBatchTime: 48.621us
- TransmitDataRPCTime: 73.238us
- UncompressedRowBatchSize: 192.00 B (192)
AGGREGATION_NODE (id=1):(Total: 4s049ms, non-child: 3s965ms, % non-child: 97.94%)
- GetNewBlockTime: 78.103us
- GetResultsTime: 9.538us
- HTResizeTime: 523.269us
- HashBuckets: 8.19K (8192)
- LargestPartitionPercent: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 1.59 MB (1668224)
- PinTime: 0.000ns
- ReductionFactorEstimate: 0.00
- ReductionFactorThresholdToExpand: 0.00
- RowsPassedThrough: 0 (0)
- RowsReturned: 5 (5)
- RowsReturnedRate: 1.00 /sec
- StreamingTime: 3s666ms
- UnpinTime: 0.000ns
HDFS_SCAN_NODE (id=0):(Total: 83.393ms, non-child: 83.393ms, % non-child: 100.00%)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 5.63
- BytesRead: 650.78 KB (666403)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 650.78 KB (666403)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 650.78 KB (666403)
- DecompressionTime: 345.570us
- MaxCompressedTextFileLength: 0
- NumColumns: 1 (1)
- NumDisksAccessed: 6 (6)
- NumRowGroups: 6 (6)
- NumScannerThreadsStarted: 6 (6)
- PeakMemoryUsage: 11.89 MB (12465356)
- PerReadThreadRawHdfsThroughput: 686.45 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 20.70M (20699747)
- RowsReturned: 20.70M (20699747)
- RowsReturnedRate: 282.05 M/sec
- ScanRangesComplete: 6 (6)
- ScannerThreadsInvoluntaryContextSwitches: 12 (12)
- ScannerThreadsTotalWallClockTime: 22s586ms
- MaterializeTupleTime(*): 22s440ms
- ScannerThreadsSysTime: 32.800ms
- ScannerThreadsUserTime: 666.440ms
- ScannerThreadsVoluntaryContextSwitches: 20.07K (20074)
- TotalRawHdfsReadTime(*): 1.368ms
- TotalReadThroughput: 181.83 KB/sec
Fragment F01:
Instance 814dc21ba8355459:ac7e36724f454896 (host=worker1.some.domain.net:22000):(Total: 4s773ms, non-child: 0.000ns, % non-child: 0.00%)
MemoryUsage(500.000ms): 10.28 MB, 10.28 MB, 10.28 MB, 10.28 MB, 10.29 MB, 10.29 MB, 10.29 MB, 10.29 MB, 10.29 MB, 10.31 MB
ThreadUsage(500.000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 10.31 MB (10810528)
- PerHostPeakMemUsage: 23.85 MB (25013504)
- PrepareTime: 140.565ms
- RowsProduced: 2 (2)
- TotalCpuTime: 198.707ms
- TotalNetworkReceiveTime: 4s632ms
- TotalNetworkSendTime: 144.042us
- TotalStorageWaitTime: 0.000ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 49 (49)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 61.98 GB (66552700928)
- PeakMemoryUsage: 8.77 MB (9191424)
- TotalBufferWaitTime: 0.000ns
- TotalEncryptionTime: 0.000ns
- TotalIntegrityCheckTime: 0.000ns
- TotalReadBlockTime: 0.000ns
CodeGen:(Total: 193.769ms, non-child: 193.769ms, % non-child: 100.00%)
- CodegenTime: 3.318ms
- CompileTime: 3.802ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.15 MB (2254856)
- NumFunctions: 29 (29)
- NumInstructions: 550 (550)
- OptimizationTime: 53.125ms
- PrepareTime: 136.382ms
DataStreamSender (dst_id=5):(Total: 119.588us, non-child: 119.588us, % non-child: 100.00%)
- BytesSent: 50.00 B (50)
- NetworkThroughput(*): 1.84 MB/sec
- OverallThroughput: 408.30 KB/sec
- PeakMemoryUsage: 4.00 KB (4096)
- RowsReturned: 2 (2)
- SerializeBatchTime: 73.063us
- TransmitDataRPCTime: 25.979us
- UncompressedRowBatchSize: 76.00 B (76)
SORT_NODE (id=2):(Total: 4s773ms, non-child: 271.839us, % non-child: 0.01%)
ExecOption: Codegen Enabled
- InMemorySortTime: 1.324us
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 8.02 MB (8404992)
- RowsReturned: 2 (2)
- RowsReturnedRate: 0
- SortDataSize: 64.00 B (64)
- SpilledRuns: 0 (0)
- TotalMergesPerformed: 0 (0)
AGGREGATION_NODE (id=4):(Total: 4s773ms, non-child: 140.842ms, % non-child: 2.95%)
ExecOption: Codegen Disabled: HashTableCtx::CodegenEvalRow(): type TIMESTAMP NYI
- BuildTime: 12.344us
- GetNewBlockTime: 87.985us
- GetResultsTime: 1.797us
- HTResizeTime: 1.646us
- HashBuckets: 2.05K (2048)
- LargestPartitionPercent: 10 (10)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 2.30 MB (2409600)
- PinTime: 0.000ns
- RowsRepartitioned: 0 (0)
- RowsReturned: 2 (2)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
- UnpinTime: 616.000ns
EXCHANGE_NODE (id=3):(Total: 4s632ms, non-child: 28.353us, % non-child: 0.00%)
BytesReceived(500.000ms): 0, 0, 0, 0, 52.00 B, 52.00 B, 52.00 B, 52.00 B, 52.00 B, 208.00 B
- BytesReceived: 260.00 B (260)
- ConvertRowBatchTime: 4.176us
- DeserializeRowBatchTimer: 25.676us
- FirstBatchArrivalWaitTime: 1s607ms
- PeakMemoryUsage: 0
- RowsReturned: 10 (10)
- RowsReturnedRate: 2.00 /sec
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
Instance 814dc21ba8355459:ac7e36724f454897 (host=worker5.some.domain.net:22000):(Total: 4s709ms, non-child: 0.000ns, % non-child: 0.00%)
MemoryUsage(500.000ms): 0, 10.28 MB, 10.28 MB, 10.28 MB, 10.29 MB, 10.29 MB, 10.29 MB, 10.29 MB, 10.29 MB, 10.30 MB
ThreadUsage(500.000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
- AverageThreadTokens: 1.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 10.31 MB (10810528)
- PerHostPeakMemUsage: 23.32 MB (24449296)
- PrepareTime: 311.018ms
- RowsProduced: 2 (2)
- TotalCpuTime: 435.142ms
- TotalNetworkReceiveTime: 4s396ms
- TotalNetworkSendTime: 187.827us
- TotalStorageWaitTime: 0.000ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 49 (49)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 61.98 GB (66552700928)
- PeakMemoryUsage: 8.77 MB (9191424)
- TotalBufferWaitTime: 0.000ns
- TotalEncryptionTime: 0.000ns
- TotalIntegrityCheckTime: 0.000ns
- TotalReadBlockTime: 0.000ns
CodeGen:(Total: 425.412ms, non-child: 425.412ms, % non-child: 100.00%)
- CodegenTime: 6.872ms
- CompileTime: 8.123ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 2.15 MB (2254856)
- NumFunctions: 29 (29)
- NumInstructions: 550 (550)
- OptimizationTime: 113.870ms
- PrepareTime: 302.324ms
DataStreamSender (dst_id=5):(Total: 69.893us, non-child: 69.893us, % non-child: 100.00%)
- BytesSent: 53.00 B (53)
- NetworkThroughput(*): 1.31 MB/sec
- OverallThroughput: 740.53 KB/sec
- PeakMemoryUsage: 4.00 KB (4096)
- RowsReturned: 2 (2)
- SerializeBatchTime: 11.380us
- TransmitDataRPCTime: 38.673us
- UncompressedRowBatchSize: 76.00 B (76)
SORT_NODE (id=2):(Total: 4s708ms, non-child: 491.025us, % non-child: 0.01%)
ExecOption: Codegen Enabled
- InMemorySortTime: 2.603us
- InitialRunsCreated: 1 (1)
- PeakMemoryUsage: 8.02 MB (8404992)
- RowsReturned: 2 (2)
- RowsReturnedRate: 0
- SortDataSize: 64.00 B (64)
- SpilledRuns: 0 (0)
- TotalMergesPerformed: 0 (0)
AGGREGATION_NODE (id=4):(Total: 4s708ms, non-child: 311.656ms, % non-child: 6.62%)
ExecOption: Codegen Disabled: HashTableCtx::CodegenEvalRow(): type TIMESTAMP NYI
- BuildTime: 21.607us
- GetNewBlockTime: 144.739us
- GetResultsTime: 5.404us
- HTResizeTime: 3.560us
- HashBuckets: 2.05K (2048)
- LargestPartitionPercent: 10 (10)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 16 (16)
- PeakMemoryUsage: 2.30 MB (2409600)
- PinTime: 0.000ns
- RowsRepartitioned: 0 (0)
- RowsReturned: 2 (2)
- RowsReturnedRate: 0
- SpilledPartitions: 0 (0)
- UnpinTime: 1.295us
EXCHANGE_NODE (id=3):(Total: 4s396ms, non-child: 42.905us, % non-child: 0.00%)
BytesReceived(500.000ms): 0, 0, 0, 51.00 B, 51.00 B, 51.00 B, 51.00 B, 51.00 B, 154.00 B
- BytesReceived: 258.00 B (258)
- ConvertRowBatchTime: 6.460us
- DeserializeRowBatchTimer: 67.686us
- FirstBatchArrivalWaitTime: 1s371ms
- PeakMemoryUsage: 0
- RowsReturned: 10 (10)
- RowsReturnedRate: 2.00 /sec
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
... View more
02-02-2017
05:21 AM
Hi, I have an issue after upgrade of cluster and migrate to a newer hardware. The same query "Select part_col, count(*) from table group by part_col" runs significantly slower on the new environment. In this case there are two clusters, one 4 nodes (CDH 5.4 Impala 2.2), one with 5 nodes (CDH5.8 Impala 2.6 kerberized). The worker nodes are newer and has more data disks, the processor is the same Xeon E5-2640 v3 vs E5-2650 v2. The connectivity is 1GBit (old) vs 10Gbit(new). So the old cluster has less nodes, less disks per node (5 vs 16), less RAM, but still manages to execute the query faster than the new cluster. I digged into the profile of the queries and found out that the step AGGREGATE takes much longer and consumes all the exec time. And in the new version it has a tag "STREAMING". The HDFS scan is in this case more or less the same, but on larger table much faster on the new cluster. So I assume this is not an IO issue. I assume that during AGGREGATE the data is already in RAM and hashes are computed by CPU.. Does it mean that the new cluster has less powerfull CPU? Or what other factors can affect the AGGREGATE (STREAMING) duration? Is there any other tools or papers describing how to tune Impala in these situations? Thanks for any advice OLD CLUSTER - Impala 2.2
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
---------------------------------------------------------------------------------------------------------------------
05:MERGING-EXCHANGE 1 119.314us 119.314us 5 -1 0 -1.00 B UNPARTITIONED
02:SORT 4 435.85us 588.641us 5 -1 8.02 MB 0
04:AGGREGATE 4 255.672ms 272.347ms 5 -1 3.15 MB 128.00 MB FINALIZE
03:EXCHANGE 4 62.142us 87.60us 20 -1 0 0 HASH(part_col)
01:AGGREGATE 4 2s803ms 3s270ms 20 -1 3.17 MB 128.00 MB
00:SCAN HDFS 4 53.678ms 56.787ms 103.50M -1 1.82 MB 88.00 MB default.table
NEW CLUSTER - Impala 2.6
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
---------------------------------------------------------------------------------------------------------------------
05:MERGING-EXCHANGE 1 82.553us 82.553us 5 -1 0 -1.00 B UNPARTITIONED
02:SORT 5 434.217us 491.025us 5 -1 8.02 MB 0
04:AGGREGATE 5 279.461ms 316.351ms 5 -1 2.30 MB 128.00 MB FINALIZE
03:EXCHANGE 5 28.378us 55.658us 25 -1 0 0 HASH(part_col)
01:AGGREGATE 5 3s965ms 4s739ms 25 -1 1.59 MB 128.00 MB STREAMING
00:SCAN HDFS 5 83.393ms 118.345ms 103.50M -1 12.04 MB 88.00 MB default.table
... View more
Labels:
- Labels:
-
Apache Impala