Reply
Expert Contributor
Posts: 321
Registered: ‎01-25-2017

impala submission control issue

Hello All,

Will appreciate any insight of what i'm missing in my configuration to get the impala submission control working as expected ....

a- Here is my placement Placement Rule:
1- Use the pool Specified at run time., only if the pool exists.
2- Use the pool root.[secondary group].
3- Use the pool root.default.

b- Here is the impala resource pools

Name Max Memory Max Running Queries Max Queued Queries Queue Timeout Default Query Memory Limit Access Control
root.impiaas Unlimited Unlimited 200 1 minute(s) No Default Custom
root.impreports Unlimited Unlimited 200 1 minute(s) No Default Custom


C- Here is my Access control for root.impiaas pool:

Allow these users and groups to submit to this pool

Users: fawzea

The same for impreports the allowed user and groups to submit to pool is only nira.


LDAP groups for fawzea user:

Member in groups:

- dba
- impiaas
- mysql

LDAP group for nira user
Member in groups:
- dba
- impreports
- mysql

 

Impala Queries UI shows:

1- select count(*) from dv_entersiteevent_prq_remote where year=2017

User: fawzea
Pool: root.impiaas

2- select count(*) from dv_entersiteevent_prq_remote where year=2017 and month=10
User: nira
Pool: root.impreports

As shown above that the queries submitted based on the secondary group rule.

When i'm running the following with fawzea user:

set request_pool=impreports;
select count(*) from dv_entersiteevent_prq_remote where year=2017 and month=10;

I can see that the query is running with impreports pool.


What i'm missing here in my configuration to get this works as expected.
==================================================================================

The config:

[root@aopr-dhc001 ~]# cat /var/run/cloudera-scm-agent/process/4558-impala-IMPALAD/impala-conf/fair-scheduler.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<allocations>
<queue name="root">
<weight>1.0</weight>
<schedulingPolicy>fair</schedulingPolicy>
<queue name="impiaas">
<weight>1.0</weight>
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>fawzea </aclSubmitApps>
</queue>
<queue name="impreports">
<weight>1.0</weight>
<schedulingPolicy>fair</schedulingPolicy>
<aclSubmitApps>nira </aclSubmitApps>
</queue>
</queue>
<defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
<queuePlacementPolicy>
<rule name="specified" create="false"/>
<rule name="secondaryGroupExistingQueue"/>
<rule name="default"/>
</queuePlacementPolicy>
</allocations>

==============================================

Query profile and log:

Query Runtime Profile:
Query (id=c843dae48ce9b289:6ad3ffc000000000):
Summary:
Session ID: ca420ac477772b56:51ae460ce9e44999
Session Type: BEESWAX
Start Time: 2018-02-08 01:02:01.023558000
End Time: 2018-02-08 01:02:01.501506000
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 2.11.0-cdh5.14.0 RELEASE (build d68206561bce6b26762d62c01a78e6cd27aa7690)
User: fawzea
Connected User: fawzea
Delegated User:
Network Address: ::ffff:10.16.144.131:50078
Default Db: default
Sql Statement: select count(*) from dv_entersiteevent_prq_remote where year=2017 and month=10
Coordinator: aopr-dhc001.lpdomain.com:22000
Query Options (set by configuration): REQUEST_POOL=impreports
Query Options (set by configuration and planner): REQUEST_POOL=impreports,MT_DOP=0
Plan:
----------------
Max Per-Host Resource Reservation: Memory=0B
Per-Host Resource Estimates: Memory=21.00MB
WARNING: The following tables are missing relevant table and/or column statistics.
default.dv_entersiteevent_prq_remote

F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
| Per-Host Resources: mem-estimate=10.00MB mem-reservation=0B
PLAN-ROOT SINK
| mem-estimate=0B mem-reservation=0B
|
03:AGGREGATE [FINALIZE]
| output: count:merge(*)
| mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
| tuple-ids=1 row-size=8B cardinality=1
|
02:EXCHANGE [UNPARTITIONED]
| mem-estimate=0B mem-reservation=0B
| tuple-ids=1 row-size=8B cardinality=1
|
F00:PLAN FRAGMENT [RANDOM] hosts=5 instances=5
Per-Host Resources: mem-estimate=11.00MB mem-reservation=0B
01:AGGREGATE
| output: sum_init_zero(default.dv_entersiteevent_prq_remote.parquet-stats: num_rows)
| mem-estimate=10.00MB mem-reservation=0B spill-buffer=2.00MB
| tuple-ids=1 row-size=8B cardinality=1
|
00:SCAN HDFS [default.dv_entersiteevent_prq_remote, RANDOM]
partitions=31/441 files=2292 size=1.39GB
stats-rows=unavailable extrapolated-rows=disabled
table stats: rows=unavailable size=unavailable
column stats: all
mem-estimate=1.00MB mem-reservation=0B
tuple-ids=0 row-size=8B cardinality=unavailable
----------------
Estimated Per-Host Mem: 22020096
Tables Missing Stats: default.dv_entersiteevent_prq_remote
Per Host Min Reservation: aopr-dhc001.lpdomain.com:22000(0) aopr-dhc002.lpdomain.com:22000(0) aopr-dhc003.lpdomain.com:22000(0) aopr-dhc004.lpdomain.com:22000(0) aopr-dhc005.lpdomain.com:22000(0)
Request Pool: root.impreports
Admission result: Admitted immediately
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
03:AGGREGATE 1 222.958us 222.958us 1 1 36.00 KB 10.00 MB FINALIZE
02:EXCHANGE 1 357.508ms 357.508ms 5 1 0 0 UNPARTITIONED
01:AGGREGATE 5 1.027ms 1.541ms 5 1 20.00 KB 10.00 MB
00:SCAN HDFS 5 282.797ms 333.715ms 2.29K -1 2.62 MB 1.00 MB default.dv_entersiteevent_prq_remote
Errors:
Planner Timeline: 13.508ms
- Analysis finished: 3.303ms (3.303ms)
- Value transfer graph computed: 3.465ms (162.667us)
- Single node plan created: 10.352ms (6.886ms)
- Runtime filters computed: 10.677ms (325.364us)
- Distributed plan created: 10.733ms (55.854us)
- Lineage info computed: 10.898ms (164.937us)
- Planning finished: 13.508ms (2.610ms)
Query Timeline: 481.950ms
- Query submitted: 79.971us (79.971us)
- Planning finished: 33.947ms (33.867ms)
- Submit for admission: 40.067ms (6.120ms)
- Completed admission: 40.197ms (130.146us)
- Ready to start on 5 backends: 40.905ms (707.852us)
- All 5 execution backends (6 fragment instances) started: 46.417ms (5.512ms)
- Rows available: 424.548ms (378.131ms)
- First row fetched: 473.488ms (48.939ms)
- Unregister query: 477.951ms (4.462ms)
- ComputeScanRangeAssignmentTimer: 3.980ms
ImpalaServer:
- ClientFetchWaitTimer: 51.659ms
- RowMaterializationTimer: 1.726ms
Execution Profile c843dae48ce9b289:6ad3ffc000000000:(Total: 385.029ms, non-child: 0.000ns, % non-child: 0.00%)
Number of filters: 0
Filter routing table:
ID Src. Node Tgt. Node(s) Target type Partition filter Pending (Expected) First arrived Completed Enabled
-------------------------------------------------------------------------------------------------------------------

Backend startup latencies: Count: 5, min / max: 3ms / 4ms, 25th %-ile: 3ms, 50th %-ile: 4ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th %-ile: 4ms, 99.9th %-ile: 4ms
Per Node Peak Memory Usage: aopr-dhc001.lpdomain.com:22000(2.68 MB) aopr-dhc002.lpdomain.com:22000(2.54 MB) aopr-dhc003.lpdomain.com:22000(2.54 MB) aopr-dhc004.lpdomain.com:22000(2.41 MB) aopr-dhc005.lpdomain.com:22000(2.56 MB)
- FiltersReceived: 0 (0)
- FinalizationTimer: 0.000ns
- NumBackends: 5 (5)
- NumFragmentInstances: 6 (6)
- NumFragments: 2 (2)
Averaged Fragment F01:(Total: 427.530ms, non-child: 49.428ms, % non-child: 11.56%)
split sizes: min: 0, max: 0, avg: 0, stddev: 0
completion times: min:428.693ms max:428.693ms mean: 428.693ms stddev:0.000ns
execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec
num instances: 1
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 44.02 KB (45072)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.68 MB (2806113)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 357.481ms
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 406.708ms
- TotalThreadsSysTime: 0.000ns
- TotalThreadsUserTime: 0.000ns
- TotalThreadsVoluntaryContextSwitches: 7 (7)
Fragment Instance Lifecycle Timings:
- ExecTime: 49.055ms
- ExecTreeExecTime: 4.251us
- OpenTime: 357.663ms
- ExecTreeOpenTime: 357.649ms
- PrepareTime: 20.764ms
- ExecTreePrepareTime: 81.759us
PLAN_ROOT_SINK:
- PeakMemoryUsage: 0
CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child: 100.00%)
- CodegenTime: 0.000ns
- CompileTime: 0.000ns
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 0 (0)
- NumInstructions: 0 (0)
- OptimizationTime: 0.000ns
- PeakMemoryUsage: 0
- PrepareTime: 19.685ms
AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child: 222.958us, % non-child: 0.06%)
- BuildTime: 7.368us
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 36.00 KB (36864)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 2.00 /sec
- SpilledPartitions: 0 (0)
EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, % non-child: 100.00%)
- ConvertRowBatchTime: 8.881us
- PeakMemoryUsage: 0
- RowsReturned: 5 (5)
- RowsReturnedRate: 13.00 /sec
DataStreamReceiver:
- BytesReceived: 80.00 B (80)
- DeserializeRowBatchTimer: 24.165us
- FirstBatchArrivalWaitTime: 277.194ms
- PeakMemoryUsage: 4.02 KB (4112)
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
Coordinator Fragment F01:
Instance c843dae48ce9b289:6ad3ffc000000000 (host=aopr-dhc001.lpdomain.com:22000):(Total: 427.530ms, non-child: 49.428ms, % non-child: 11.56%)
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 44.02 KB (45072)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.68 MB (2806113)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 357.481ms
- TotalNetworkSendTime: 0.000ns
- TotalStorageWaitTime: 0.000ns
- TotalThreadsInvoluntaryContextSwitches: 0 (0)
- TotalThreadsTotalWallClockTime: 406.708ms
- TotalThreadsSysTime: 0.000ns
- TotalThreadsUserTime: 0.000ns
- TotalThreadsVoluntaryContextSwitches: 7 (7)
Fragment Instance Lifecycle Timings:
- ExecTime: 49.055ms
- ExecTreeExecTime: 4.251us
- OpenTime: 357.663ms
- ExecTreeOpenTime: 357.649ms
- PrepareTime: 20.764ms
- ExecTreePrepareTime: 81.759us
PLAN_ROOT_SINK:
- PeakMemoryUsage: 0
CodeGen:(Total: 20.369ms, non-child: 20.369ms, % non-child: 100.00%)
- CodegenTime: 0.000ns
- CompileTime: 0.000ns
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 0 (0)
- NumInstructions: 0 (0)
- OptimizationTime: 0.000ns
- PeakMemoryUsage: 0
- PrepareTime: 19.685ms
AGGREGATION_NODE (id=3):(Total: 357.731ms, non-child: 222.958us, % non-child: 0.06%)
- BuildTime: 7.368us
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 36.00 KB (36864)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 2.00 /sec
- SpilledPartitions: 0 (0)
EXCHANGE_NODE (id=2):(Total: 357.508ms, non-child: 357.508ms, % non-child: 100.00%)
- ConvertRowBatchTime: 8.881us
- PeakMemoryUsage: 0
- RowsReturned: 5 (5)
- RowsReturnedRate: 13.00 /sec
DataStreamReceiver:
- BytesReceived: 80.00 B (80)
- DeserializeRowBatchTimer: 24.165us
- FirstBatchArrivalWaitTime: 277.194ms
- PeakMemoryUsage: 4.02 KB (4112)
- SendersBlockedTimer: 0.000ns
- SendersBlockedTotalTimer(*): 0.000ns
Averaged Fragment F00:(Total: 327.360ms, non-child: 1.578ms, % non-child: 0.48%)
split sizes: min: 283.31 MB, max: 284.33 MB, avg: 284.04 MB, stddev: 382.00 KB
completion times: min:299.457ms max:378.950ms mean: 328.881ms stddev:31.507ms
execution rates: min:750.18 MB/sec max:949.01 MB/sec mean:871.27 MB/sec stddev:79.74 MB/sec
num instances: 5
- AverageThreadTokens: 15.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 2.54 MB (2666852)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.55 MB (2669309)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 303.541us
- TotalStorageWaitTime: 4s821ms
- TotalThreadsInvoluntaryContextSwitches: 104 (104)
- TotalThreadsTotalWallClockTime: 6s835ms
- TotalThreadsSysTime: 31.784ms
- TotalThreadsUserTime: 682.084ms
- TotalThreadsVoluntaryContextSwitches: 2.12K (2121)
Fragment Instance Lifecycle Timings:
- ExecTime: 500.816us
- ExecTreeExecTime: 8.200us
- OpenTime: 302.848ms
- ExecTreeOpenTime: 281.309ms
- PrepareTime: 23.946ms
- ExecTreePrepareTime: 2.513ms
DataStreamSender (dst_id=2):(Total: 269.902us, non-child: 269.902us, % non-child: 100.00%)
- BytesSent: 16.00 B (16)
- NetworkThroughput(*): 373.13 KB/sec
- OverallThroughput: 63.53 KB/sec
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- SerializeBatchTime: 14.525us
- TransmitDataRPCTime: 43.122us
- UncompressedRowBatchSize: 16.00 B (16)
CodeGen:(Total: 41.687ms, non-child: 41.687ms, % non-child: 100.00%)
- CodegenTime: 1.194ms
- CompileTime: 7.360ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 15 (15)
- NumInstructions: 214 (214)
- OptimizationTime: 13.529ms
- PeakMemoryUsage: 107.00 KB (109568)
- PrepareTime: 20.084ms
AGGREGATION_NODE (id=1):(Total: 283.825ms, non-child: 1.027ms, % non-child: 0.36%)
- BuildTime: 1.259ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 20.00 KB (20480)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 2.00 /sec
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 282.797ms, non-child: 282.797ms, % non-child: 100.00%)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 14.40
- BytesRead: 44.52 MB (46686028)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 44.52 MB (46686028)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 44.52 MB (46686028)
- CachedFileHandle**bleep**Count: 0 (0)
- CachedFileHandlesMissCount: 458 (458)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 0 (0)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 458 (458)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.50 MB (2625539)
- PerReadThreadRawHdfsThroughput: 1.29 GB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 269.787ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 0 (0)
- RowsReturned: 458 (458)
- RowsReturnedRate: 1.64 K/sec
- ScanRangesComplete: 458 (458)
- ScannerThreadsInvoluntaryContextSwitches: 99 (99)
- ScannerThreadsTotalWallClockTime: 6s532ms
- MaterializeTupleTime(*): 70.664us
- ScannerThreadsSysTime: 26.184ms
- ScannerThreadsUserTime: 650.089ms
- ScannerThreadsVoluntaryContextSwitches: 1.67K (1674)
- TotalRawHdfsReadTime(*): 34.416ms
- TotalReadThroughput: 36.86 MB/sec
Fragment F00:
Instance c843dae48ce9b289:6ad3ffc000000004 (host=aopr-dhc004.lpdomain.com:22000):(Total: 377.384ms, non-child: 1.619ms, % non-child: 0.43%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB 1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB 4:42/26.73 MB 8:34/21.25 MB
MemoryUsage(500.000ms): 1.91 MB
ThreadUsage(500.000ms): 25
- AverageThreadTokens: 25.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 2.41 MB (2531689)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.41 MB (2531689)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 382.629us
- TotalStorageWaitTime: 5s790ms
- TotalThreadsInvoluntaryContextSwitches: 115 (115)
- TotalThreadsTotalWallClockTime: 8s122ms
- TotalThreadsSysTime: 33.984ms
- TotalThreadsUserTime: 702.879ms
- TotalThreadsVoluntaryContextSwitches: 2.04K (2037)
Fragment Instance Lifecycle Timings:
- ExecTime: 604.005us
- ExecTreeExecTime: 8.338us
- OpenTime: 352.927ms
- ExecTreeOpenTime: 331.921ms
- PrepareTime: 23.783ms
- ExecTreePrepareTime: 2.559ms
DataStreamSender (dst_id=2):(Total: 331.818us, non-child: 331.818us, % non-child: 100.00%)
- BytesSent: 16.00 B (16)
- NetworkThroughput(*): 289.00 KB/sec
- OverallThroughput: 47.09 KB/sec
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- SerializeBatchTime: 14.466us
- TransmitDataRPCTime: 54.066us
- UncompressedRowBatchSize: 16.00 B (16)
CodeGen:(Total: 40.950ms, non-child: 40.950ms, % non-child: 100.00%)
- CodegenTime: 1.122ms
- CompileTime: 7.318ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 15 (15)
- NumInstructions: 214 (214)
- OptimizationTime: 13.106ms
- PeakMemoryUsage: 107.00 KB (109568)
- PrepareTime: 19.773ms
AGGREGATION_NODE (id=1):(Total: 334.482ms, non-child: 766.527us, % non-child: 0.23%)
ExecOption: Codegen Enabled
- BuildTime: 1.192ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 20.00 KB (20480)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 2.00 /sec
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 333.715ms, non-child: 333.715ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:36/21.95 MB 11:33/19.87 MB 9:27/15.16 MB 5:44/28.07 MB 6:56/33.32 MB 1:38/22.24 MB 10:44/27.90 MB 3:34/20.26 MB 2:45/28.21 MB 7:31/19.33 MB 4:42/26.73 MB 8:34/21.25 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 464 out of 464
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/Unknown(Skipped):464
BytesRead(500.000ms): 12.27 MB
- FooterProcessingTime: (Avg: 13.555ms ; Min: 5.433ms ; Max: 62.957ms ; Number of samples: 464)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 24.00
- BytesRead: 45.01 MB (47193988)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 45.01 MB (47193988)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 45.01 MB (47193988)
- CachedFileHandle**bleep**Count: 0 (0)
- CachedFileHandlesMissCount: 464 (464)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 0 (0)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 464 (464)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.38 MB (2490376)
- PerReadThreadRawHdfsThroughput: 1016.44 MB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 320.860ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 0 (0)
- RowsReturned: 464 (464)
- RowsReturnedRate: 1.39 K/sec
- ScanRangesComplete: 464 (464)
- ScannerThreadsInvoluntaryContextSwitches: 108 (108)
- ScannerThreadsTotalWallClockTime: 7s769ms
- MaterializeTupleTime(*): 75.514us
- ScannerThreadsSysTime: 29.984ms
- ScannerThreadsUserTime: 669.884ms
- ScannerThreadsVoluntaryContextSwitches: 1.60K (1597)
- TotalRawHdfsReadTime(*): 44.279ms
- TotalReadThroughput: 24.53 MB/sec
Instance c843dae48ce9b289:6ad3ffc000000002 (host=aopr-dhc003.lpdomain.com:22000):(Total: 351.014ms, non-child: 1.641ms, % non-child: 0.47%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB 2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB 4:60/37.59 MB 1:44/27.19 MB
MemoryUsage(500.000ms): 1.16 MB
ThreadUsage(500.000ms): 25
- AverageThreadTokens: 25.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 2.54 MB (2662753)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.54 MB (2662753)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 342.746us
- TotalStorageWaitTime: 4s796ms
- TotalThreadsInvoluntaryContextSwitches: 93 (93)
- TotalThreadsTotalWallClockTime: 7s286ms
- TotalThreadsSysTime: 27.988ms
- TotalThreadsUserTime: 671.887ms
- TotalThreadsVoluntaryContextSwitches: 2.08K (2079)
Fragment Instance Lifecycle Timings:
- ExecTime: 524.006us
- ExecTreeExecTime: 7.920us
- OpenTime: 327.802ms
- ExecTreeOpenTime: 307.154ms
- PrepareTime: 22.617ms
- ExecTreePrepareTime: 2.650ms
DataStreamSender (dst_id=2):(Total: 348.990us, non-child: 348.990us, % non-child: 100.00%)
- BytesSent: 16.00 B (16)
- NetworkThroughput(*): 354.47 KB/sec
- OverallThroughput: 44.77 KB/sec
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- SerializeBatchTime: 15.602us
- TransmitDataRPCTime: 44.080us
- UncompressedRowBatchSize: 16.00 B (16)
CodeGen:(Total: 39.215ms, non-child: 39.215ms, % non-child: 100.00%)
- CodegenTime: 1.183ms
- CompileTime: 7.168ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 15 (15)
- NumInstructions: 214 (214)
- OptimizationTime: 12.929ms
- PeakMemoryUsage: 107.00 KB (109568)
- PrepareTime: 18.453ms
AGGREGATION_NODE (id=1):(Total: 309.807ms, non-child: 948.677us, % non-child: 0.31%)
ExecOption: Codegen Enabled
- BuildTime: 1.552ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 20.00 KB (20480)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 3.00 /sec
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 308.859ms, non-child: 308.859ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:35/20.89 MB 7:44/26.64 MB 11:27/17.21 MB 9:32/18.61 MB 5:25/14.56 MB 2:38/24.32 MB 6:31/19.85 MB 8:50/30.76 MB 10:41/25.92 MB 3:31/19.78 MB 4:60/37.59 MB 1:44/27.19 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 458 out of 458
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/Unknown(Skipped):458
BytesRead(500.000ms): 36.74 MB
- FooterProcessingTime: (Avg: 11.502ms ; Min: 3.668ms ; Max: 23.564ms ; Number of samples: 458)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 24.00
- BytesRead: 44.51 MB (46668432)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 44.51 MB (46668432)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 44.51 MB (46668432)
- CachedFileHandle**bleep**Count: 0 (0)
- CachedFileHandlesMissCount: 458 (458)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 0 (0)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 458 (458)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.50 MB (2621440)
- PerReadThreadRawHdfsThroughput: 1.26 GB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 295.595ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 0 (0)
- RowsReturned: 458 (458)
- RowsReturnedRate: 1.48 K/sec
- ScanRangesComplete: 458 (458)
- ScannerThreadsInvoluntaryContextSwitches: 91 (91)
- ScannerThreadsTotalWallClockTime: 6s958ms
- MaterializeTupleTime(*): 71.630us
- ScannerThreadsSysTime: 16.989ms
- ScannerThreadsUserTime: 645.891ms
- ScannerThreadsVoluntaryContextSwitches: 1.63K (1630)
- TotalRawHdfsReadTime(*): 34.377ms
- TotalReadThroughput: 73.49 MB/sec
Instance c843dae48ce9b289:6ad3ffc000000001 (host=aopr-dhc005.lpdomain.com:22000):(Total: 308.195ms, non-child: 1.784ms, % non-child: 0.58%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB 9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB 4:40/24.29 MB 11:33/20.77 MB
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 2.56 MB (2683241)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.56 MB (2683241)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 292.765us
- TotalStorageWaitTime: 4s482ms
- TotalThreadsInvoluntaryContextSwitches: 113 (113)
- TotalThreadsTotalWallClockTime: 6s310ms
- TotalThreadsSysTime: 35.983ms
- TotalThreadsUserTime: 682.884ms
- TotalThreadsVoluntaryContextSwitches: 2.14K (2143)
Fragment Instance Lifecycle Timings:
- ExecTime: 532.572us
- ExecTreeExecTime: 8.377us
- OpenTime: 280.241ms
- ExecTreeOpenTime: 258.652ms
- PrepareTime: 27.342ms
- ExecTreePrepareTime: 2.863ms
DataStreamSender (dst_id=2):(Total: 309.794us, non-child: 309.794us, % non-child: 100.00%)
- BytesSent: 16.00 B (16)
- NetworkThroughput(*): 356.91 KB/sec
- OverallThroughput: 50.44 KB/sec
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- SerializeBatchTime: 14.262us
- TransmitDataRPCTime: 43.778us
- UncompressedRowBatchSize: 16.00 B (16)
CodeGen:(Total: 44.584ms, non-child: 44.584ms, % non-child: 100.00%)
- CodegenTime: 1.382ms
- CompileTime: 7.346ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 15 (15)
- NumInstructions: 214 (214)
- OptimizationTime: 13.462ms
- PeakMemoryUsage: 107.00 KB (109568)
- PrepareTime: 23.006ms
AGGREGATION_NODE (id=1):(Total: 261.516ms, non-child: 836.656us, % non-child: 0.32%)
ExecOption: Codegen Enabled
- BuildTime: 1.280ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 20.00 KB (20480)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 3.00 /sec
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 260.680ms, non-child: 260.680ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:46/28.97 MB 8:39/24.34 MB 10:33/21.03 MB 3:43/26.50 MB 5:32/20.27 MB 9:32/19.86 MB 6:25/16.01 MB 7:41/24.77 MB 1:42/25.01 MB 0:55/32.51 MB 4:40/24.29 MB 11:33/20.77 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 461 out of 461
Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/Unknown(Skipped):461
- FooterProcessingTime: (Avg: 10.793ms ; Min: 3.087ms ; Max: 20.253ms ; Number of samples: 461)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 44.73 MB (46906407)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 44.73 MB (46906407)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 44.73 MB (46906407)
- CachedFileHandle**bleep**Count: 0 (0)
- CachedFileHandlesMissCount: 461 (461)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 0 (0)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 461 (461)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.52 MB (2641928)
- PerReadThreadRawHdfsThroughput: 1.31 GB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 246.691ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 0 (0)
- RowsReturned: 461 (461)
- RowsReturnedRate: 1.77 K/sec
- ScanRangesComplete: 461 (461)
- ScannerThreadsInvoluntaryContextSwitches: 109 (109)
- ScannerThreadsTotalWallClockTime: 6s030ms
- MaterializeTupleTime(*): 68.735us
- ScannerThreadsSysTime: 33.983ms
- ScannerThreadsUserTime: 647.889ms
- ScannerThreadsVoluntaryContextSwitches: 1.70K (1696)
- TotalRawHdfsReadTime(*): 33.405ms
- TotalReadThroughput: 0.00 /sec
Instance c843dae48ce9b289:6ad3ffc000000003 (host=aopr-dhc002.lpdomain.com:22000):(Total: 302.116ms, non-child: 1.604ms, % non-child: 0.53%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13 MB 9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB 4:41/25.99 MB 8:38/23.79 MB
MemoryUsage(500.000ms): 700.35 KB
ThreadUsage(500.000ms): 25
- AverageThreadTokens: 25.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 2.54 MB (2662753)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.54 MB (2662753)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 300.675us
- TotalStorageWaitTime: 4s661ms
- TotalThreadsInvoluntaryContextSwitches: 90 (90)
- TotalThreadsTotalWallClockTime: 6s256ms
- TotalThreadsSysTime: 30.983ms
- TotalThreadsUserTime: 709.880ms
- TotalThreadsVoluntaryContextSwitches: 2.17K (2175)
Fragment Instance Lifecycle Timings:
- ExecTime: 492.627us
- ExecTreeExecTime: 8.191us
- OpenTime: 277.922ms
- ExecTreeOpenTime: 255.395ms
- PrepareTime: 23.647ms
- ExecTreePrepareTime: 2.243ms
DataStreamSender (dst_id=2):(Total: 194.517us, non-child: 194.517us, % non-child: 100.00%)
- BytesSent: 16.00 B (16)
- NetworkThroughput(*): 371.86 KB/sec
- OverallThroughput: 80.33 KB/sec
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- SerializeBatchTime: 14.035us
- TransmitDataRPCTime: 42.019us
- UncompressedRowBatchSize: 16.00 B (16)
CodeGen:(Total: 42.676ms, non-child: 42.676ms, % non-child: 100.00%)
- CodegenTime: 1.236ms
- CompileTime: 7.717ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 15 (15)
- NumInstructions: 214 (214)
- OptimizationTime: 14.130ms
- PeakMemoryUsage: 107.00 KB (109568)
- PrepareTime: 20.109ms
AGGREGATION_NODE (id=1):(Total: 257.640ms, non-child: 1.541ms, % non-child: 0.60%)
ExecOption: Codegen Enabled
- BuildTime: 1.252ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 20.00 KB (20480)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 3.00 /sec
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 256.099ms, non-child: 256.099ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:27/16.21 MB 11:33/21.23 MB 6:36/23.15 MB 10:41/25.09 MB 1:41/24.13 MB 9:32/18.79 MB 5:41/25.31 MB 2:42/26.49 MB 7:36/22.87 MB 3:51/31.00 MB 4:41/25.99 MB 8:38/23.79 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 459 out of 459
Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/Unknown(Skipped):459
BytesRead(500.000ms): 43.15 MB
- FooterProcessingTime: (Avg: 11.257ms ; Min: 3.989ms ; Max: 23.004ms ; Number of samples: 459)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 24.00
- BytesRead: 44.52 MB (46678031)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 44.52 MB (46678031)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 44.52 MB (46678031)
- CachedFileHandle**bleep**Count: 0 (0)
- CachedFileHandlesMissCount: 459 (459)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 0 (0)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 459 (459)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.50 MB (2621440)
- PerReadThreadRawHdfsThroughput: 1.33 GB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 243.426ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 0 (0)
- RowsReturned: 459 (459)
- RowsReturnedRate: 1.79 K/sec
- ScanRangesComplete: 459 (459)
- ScannerThreadsInvoluntaryContextSwitches: 85 (85)
- ScannerThreadsTotalWallClockTime: 5s978ms
- MaterializeTupleTime(*): 67.997us
- ScannerThreadsSysTime: 22.984ms
- ScannerThreadsUserTime: 678.884ms
- ScannerThreadsVoluntaryContextSwitches: 1.72K (1724)
- TotalRawHdfsReadTime(*): 32.626ms
- TotalReadThroughput: 86.30 MB/sec
Instance c843dae48ce9b289:6ad3ffc000000005 (host=aopr-dhc001.lpdomain.com:22000):(Total: 298.093ms, non-child: 1.242ms, % non-child: 0.42%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB 4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB 6:42/27.03 MB 7:47/29.22 MB
- AverageThreadTokens: 0.00
- BloomFilterBytes: 0
- PeakMemoryUsage: 2.66 MB (2793825)
- PeakReservation: 0
- PeakUsedReservation: 0
- PerHostPeakMemUsage: 2.68 MB (2806113)
- RowsProduced: 1 (1)
- TotalNetworkReceiveTime: 0.000ns
- TotalNetworkSendTime: 198.894us
- TotalStorageWaitTime: 4s374ms
- TotalThreadsInvoluntaryContextSwitches: 110 (110)
- TotalThreadsTotalWallClockTime: 6s201ms
- TotalThreadsSysTime: 29.983ms
- TotalThreadsUserTime: 642.892ms
- TotalThreadsVoluntaryContextSwitches: 2.17K (2174)
Fragment Instance Lifecycle Timings:
- ExecTime: 350.873us
- ExecTreeExecTime: 8.173us
- OpenTime: 275.347ms
- ExecTreeOpenTime: 253.425ms
- PrepareTime: 22.341ms
- ExecTreePrepareTime: 2.251ms
DataStreamSender (dst_id=2):(Total: 164.394us, non-child: 164.394us, % non-child: 100.00%)
- BytesSent: 16.00 B (16)
- NetworkThroughput(*): 493.42 KB/sec
- OverallThroughput: 95.05 KB/sec
- PeakMemoryUsage: 16.00 KB (16384)
- RowsReturned: 1 (1)
- SerializeBatchTime: 14.262us
- TransmitDataRPCTime: 31.667us
- UncompressedRowBatchSize: 16.00 B (16)
CodeGen:(Total: 41.008ms, non-child: 41.008ms, % non-child: 100.00%)
- CodegenTime: 1.049ms
- CompileTime: 7.248ms
- LoadTime: 0.000ns
- ModuleBitcodeSize: 1.86 MB (1950880)
- NumFunctions: 15 (15)
- NumInstructions: 214 (214)
- OptimizationTime: 14.018ms
- PeakMemoryUsage: 107.00 KB (109568)
- PrepareTime: 19.080ms
AGGREGATION_NODE (id=1):(Total: 255.678ms, non-child: 1.046ms, % non-child: 0.41%)
ExecOption: Codegen Enabled
- BuildTime: 1.015ms
- GetResultsTime: 0.000ns
- HTResizeTime: 0.000ns
- HashBuckets: 0 (0)
- LargestPartitionPercent: 0 (0)
- MaxPartitionLevel: 0 (0)
- NumRepartitions: 0 (0)
- PartitionsCreated: 0 (0)
- PeakMemoryUsage: 20.00 KB (20480)
- RowsRepartitioned: 0 (0)
- RowsReturned: 1 (1)
- RowsReturnedRate: 3.00 /sec
- SpilledPartitions: 0 (0)
HDFS_SCAN_NODE (id=0):(Total: 254.631ms, non-child: 254.631ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 9:55/35.38 MB 0:37/24.23 MB 5:37/23.31 MB 11:22/13.79 MB 8:30/18.53 MB 4:33/20.45 MB 1:38/24.06 MB 10:29/18.40 MB 3:39/24.35 MB 2:41/25.45 MB 6:42/27.03 MB 7:47/29.22 MB
ExecOption: PARQUET Codegen Enabled, Codegen enabled: 450 out of 450
Hdfs Read Thread Concurrency Bucket: 0:0% 1:0% 2:0% 3:0% 4:0% 5:0% 6:0% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% 16:0%
File Formats: PARQUET/Unknown(Skipped):450
- FooterProcessingTime: (Avg: 10.727ms ; Min: 3.813ms ; Max: 23.755ms ; Number of samples: 450)
- AverageHdfsReadThreadConcurrency: 0.00
- AverageScannerThreadConcurrency: 0.00
- BytesRead: 43.85 MB (45983284)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 43.85 MB (45983284)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 43.85 MB (45983284)
- CachedFileHandle**bleep**Count: 0 (0)
- CachedFileHandlesMissCount: 450 (450)
- CollectionItemsRead: 0 (0)
- DecompressionTime: 0.000ns
- MaxCompressedTextFileLength: 0
- NumColumns: 0 (0)
- NumDictFilteredRowGroups: 0 (0)
- NumDisksAccessed: 12 (12)
- NumRowGroups: 450 (450)
- NumScannerThreadsStarted: 24 (24)
- NumScannersWithNoReads: 0 (0)
- NumStatsFilteredRowGroups: 0 (0)
- PeakMemoryUsage: 2.62 MB (2752512)
- PerReadThreadRawHdfsThroughput: 1.56 GB/sec
- RemoteScanRanges: 0 (0)
- RowBatchQueueGetWaitTime: 242.362ms
- RowBatchQueuePutWaitTime: 0.000ns
- RowsRead: 0 (0)
- RowsReturned: 450 (450)
- RowsReturnedRate: 1.77 K/sec
- ScanRangesComplete: 450 (450)
- ScannerThreadsInvoluntaryContextSwitches: 106 (106)
- ScannerThreadsTotalWallClockTime: 5s926ms
- MaterializeTupleTime(*): 69.447us
- ScannerThreadsSysTime: 26.984ms
- ScannerThreadsUserTime: 607.897ms
- ScannerThreadsVoluntaryContextSwitches: 1.72K (1723)
- TotalRawHdfsReadTime(*): 27.393ms
- TotalReadThroughput: 0.00 /sec
===================================================

Coordinator log:

01: query (string) = "select count(*) from dv_entersiteevent_prq_remote where year=2017 and month=10",
03: configuration (list) = list<string>[1] {
[0] = "REQUEST_POOL=impreports",
},
04: hadoop_user (string) = "fawzea",
}
I0208 01:07:06.321281 7154 impala-beeswax-server.cc:459] TClientRequest.queryOptions: TQueryOptions {
01: abort_on_error (bool) = false,
02: max_errors (i32) = 100,
03: disable_codegen (bool) = false,
04: batch_size (i32) = 0,
05: num_nodes (i32) = 0,
06: max_scan_range_length (i64) = 0,
07: num_scanner_threads (i32) = 0,
08: max_io_buffers (i32) = 0,
09: allow_unsupported_formats (bool) = false,
10: default_order_by_limit (i64) = -1,
11: debug_action (string) = "",
12: mem_limit (i64) = 0,
13: abort_on_default_limit_exceeded (bool) = false,
15: hbase_caching (i32) = 0,
16: hbase_cache_blocks (bool) = false,
17: parquet_file_size (i64) = 0,
18: explain_level (i32) = 1,
19: sync_ddl (bool) = false,
20: request_pool (string) = "impreports",
23: disable_cached_reads (bool) = false,
24: disable_outermost_topn (bool) = false,
25: rm_initial_mem (i64) = 0,
26: query_timeout_s (i32) = 0,
28: appx_count_distinct (bool) = false,
29: disable_unsafe_spills (bool) = false,
31: exec_single_node_rows_threshold (i32) = 100,
32: optimize_partition_key_scans (bool) = false,
33: replica_preference (i32) = 0,
34: schedule_random_replica (bool) = false,
35: scan_node_codegen_threshold (i64) = 1800000,
36: disable_streaming_preaggregations (bool) = false,
37: runtime_filter_mode (i32) = 2,
38: runtime_bloom_filter_size (i32) = 1048576,
39: runtime_filter_wait_time_ms (i32) = 0,
40: disable_row_runtime_filtering (bool) = false,
41: max_num_runtime_filters (i32) = 10,
42: parquet_annotate_strings_utf8 (bool) = false,
43: parquet_fallback_schema_resolution (i32) = 0,
45: s3_skip_insert_staging (bool) = true,
46: runtime_filter_min_size (i32) = 1048576,
47: runtime_filter_max_size (i32) = 16777216,
48: prefetch_mode (i32) = 1,
49: strict_mode (bool) = false,
50: scratch_limit (i64) = -1,
51: enable_expr_rewrites (bool) = true,
52: decimal_v2 (bool) = false,
53: parquet_dictionary_filtering (bool) = true,
54: parquet_array_resolution (i32) = 2,
55: parquet_read_statistics (bool) = true,
56: default_join_distribution_mode (i32) = 0,
57: disable_codegen_rows_threshold (i32) = 50000,
58: default_spillable_buffer_size (i64) = 2097152,
59: min_spillable_buffer_size (i64) = 65536,
60: max_row_size (i64) = 524288,
}
I0208 01:07:06.324410 7154 Frontend.java:909] Compiling query: select count(*) from dv_entersiteevent_prq_remote where year=2017 and month=10
I0208 01:07:06.327203 7154 Frontend.java:948] Compiled query.
I0208 01:07:06.359748 7154 admission-controller.cc:510] Schedule for id=5f49e927591a9042:faf8f28600000000 in pool_name=root.impreports cluster_mem_needed=105.00 MB PoolConfig: max_requests=-1 max_queued=200 max_mem=-1.00 B
I0208 01:07:06.359776 7154 admission-controller.cc:515] Stats: agg_num_running=0, agg_num_queued=0, agg_mem_reserved=0, local_host(local_mem_admitted=0, num_admitted_running=0, num_queued=0, backend_mem_reserved=0)
I0208 01:07:06.359784 7154 admission-controller.cc:531] Admitted query id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.359807 7154 coordinator.cc:99] Exec() query_id=5f49e927591a9042:faf8f28600000000 stmt=select count(*) from dv_entersiteevent_prq_remote where year=2017 and month=10
I0208 01:07:06.360694 7154 coordinator.cc:357] starting execution on 5 backends for query 5f49e927591a9042:faf8f28600000000
I0208 01:07:06.363255 63583 impala-internal-service.cc:44] ExecQueryFInstances(): query_id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.363282 63583 query-exec-mgr.cc:46] StartQueryFInstances() query_id=5f49e927591a9042:faf8f28600000000 coord=aopr-dhc001.lpdomain.com:22000
I0208 01:07:06.363296 63583 query-state.cc:173] Buffer pool limit for 5f49e927591a9042:faf8f28600000000: 10802010521
I0208 01:07:06.363364 63583 initial-reservations.cc:60] Successfully claimed initial reservations (0) for query 5f49e927591a9042:faf8f28600000000
I0208 01:07:06.363544 7155 query-state.cc:286] StartFInstances(): query_id=5f49e927591a9042:faf8f28600000000 #instances=2
I0208 01:07:06.363844 7155 query-state.cc:299] descriptor table for query=5f49e927591a9042:faf8f28600000000
tuples:
Tuple(id=2 size=0 slots=[] tuple_path=[])
Tuple(id=1 size=8 slots=[Slot(id=2 type=BIGINT col_path=[] offset=0 null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
Tuple(id=0 size=8 slots=[Slot(id=3 type=BIGINT col_path=[] offset=0 null=(offset=0 mask=0) slot_idx=0 field_idx=-1)] tuple_path=[])
I0208 01:07:06.364013 7156 query-state.cc:377] Executing instance. instance_id=5f49e927591a9042:faf8f28600000000 fragment_idx=0 per_fragment_instance_idx=0 coord_state_idx=0 #in-flight=1
I0208 01:07:06.364135 7157 query-state.cc:377] Executing instance. instance_id=5f49e927591a9042:faf8f28600000005 fragment_idx=1 per_fragment_instance_idx=4 coord_state_idx=0 #in-flight=2
I0208 01:07:06.364241 7157 hdfs-scan-node.cc:160] Max row batch queue size for scan node '0' in fragment instance '5f49e927591a9042:faf8f28600000005': 160
I0208 01:07:06.366775 7154 coordinator.cc:370] started execution on 5 backends for query 5f49e927591a9042:faf8f28600000000
I0208 01:07:06.385625 7155 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=5f49e927591a9042:faf8f28600000000 refcnt=4
I0208 01:07:06.671193 7559 impala-server.cc:1796] Connection from client ::ffff:10.16.136.3:55598 closed, closing 1 associated session(s)
I0208 01:07:06.676138 544 data-stream-mgr.cc:228] Reduced stream ID cache from 2 items, to 0, eviction took: 0
I0208 01:07:06.677202 544 coordinator.cc:955] Backend completed: host=aopr-dhc002.lpdomain.com:22000 remaining=4
I0208 01:07:06.677214 544 coordinator-backend-state.cc:229] query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend: aopr-dhc001.lpdomain.com:22000
I0208 01:07:06.684221 7157 query-state.cc:384] Instance completed. instance_id=5f49e927591a9042:faf8f28600000005 #in-flight=1 status=OK
I0208 01:07:06.684237 7157 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=5f49e927591a9042:faf8f28600000000 refcnt=3
I0208 01:07:06.700275 520 coordinator.cc:955] Backend completed: host=aopr-dhc005.lpdomain.com:22000 remaining=3
I0208 01:07:06.700289 520 coordinator-backend-state.cc:229] query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend: aopr-dhc001.lpdomain.com:22000
I0208 01:07:06.704620 65258 coordinator.cc:955] Backend completed: host=aopr-dhc003.lpdomain.com:22000 remaining=2
I0208 01:07:06.704666 65258 coordinator-backend-state.cc:229] query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend: aopr-dhc001.lpdomain.com:22000
I0208 01:07:06.706041 7156 data-stream-mgr.cc:238] DeregisterRecvr(): fragment_instance_id=5f49e927591a9042:faf8f28600000000, node=2
I0208 01:07:06.706070 7156 data-stream-recvr.cc:235] cancelled stream: fragment_instance_id_=5f49e927591a9042:faf8f28600000000 node_id=2
I0208 01:07:06.706809 64916 coordinator.cc:955] Backend completed: host=aopr-dhc004.lpdomain.com:22000 remaining=1
I0208 01:07:06.706821 64916 coordinator-backend-state.cc:229] query_id=5f49e927591a9042:faf8f28600000000: first in-progress backend: aopr-dhc001.lpdomain.com:22000
I0208 01:07:06.789568 7154 impala-beeswax-server.cc:193] get_results_metadata(): query_id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.790439 7154 coordinator.cc:789] Coordinator waiting for backends to finish, 1 remaining
I0208 01:07:06.791052 7154 coordinator.cc:794] All backends finished successfully.
I0208 01:07:06.791086 7154 coordinator.cc:1090] Release admssion control resources for query 5f49e927591a9042:faf8f28600000000
I0208 01:07:06.791363 7156 query-state.cc:384] Instance completed. instance_id=5f49e927591a9042:faf8f28600000000 #in-flight=0 status=OK
I0208 01:07:06.791414 7156 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=5f49e927591a9042:faf8f28600000000 refcnt=2
I0208 01:07:06.793586 7154 impala-beeswax-server.cc:235] close(): query_id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.793602 7154 impala-server.cc:992] UnregisterQuery(): query_id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.793614 7154 impala-server.cc:1075] Cancel(): query_id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.793624 7154 coordinator.cc:895] Cancel() query_id=5f49e927591a9042:faf8f28600000000
I0208 01:07:06.793637 7154 coordinator.cc:905] CancelBackends() query_id=5f49e927591a9042:faf8f28600000000, tried to cancel 0 backends
I0208 01:07:06.802776 7154 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=5f49e927591a9042:faf8f28600000000 refcnt=1
I0208 01:07:06.814214 7154 impala-server.cc:1796] Connection from client ::ffff:10.16.144.131:51012 closed, closing 1 associated session(s)
I0208 01:07:07.258891 7567 impala-server.cc:1796] Connection from client ::ffff:10.16.136.2:56375 closed, closing 1 associated session(s)
I0208 01:07:11.600131 7819 impala-server.cc:1796] Connection from client ::ffff:10.16.136.3:27460 closed, closing 1 associated session(s)
I0208 01:07:12.262739 7821 impala-server.cc:1796] Connection from client ::ffff:10.16.136.2:52672 closed, closing 1 associated session(s)
I0208 01:07:16.665451 7822 impala-server.cc:1796] Connection from client ::ffff:10.16.136.3:10693 closed, closing 1 associated session(s)
I0208 01:07:17.235321 7825 impala-server.cc:1796] Connection from client ::ffff:10.16.136.2:34752 closed, 

Announcements