NumScannerThreadsStarted of a specified node is always 1, what it indicates?

[#1]

Executing whatever query, NumScannerThreadsStarted of a specified node is always 1 in profile, but others  are always obvious greater than 1.
Compared with other nodes, no found any difference of the specified node.
The profiles are too long, so filtered some properties from one of profiles as follows:

hadoop1:~ # fgrep "BytesRead:" profile_1.txt
- BytesRead: 1.55 GB (1668901870)
- BytesRead: 3.68 GB (3955651251)
- BytesRead: 4.62 GB (4955876955)
- BytesRead: 1.59 GB (1710640190)
- BytesRead: 1.50 GB (1609316761)
- BytesRead: 2.61 GB (2804563939)
- BytesRead: 1.43 GB (1538913826)
- BytesRead: 835.93 MB (876532952)
- BytesRead: 1.04 GB (1111533925)
- BytesRead: 1.41 GB (1518383151)
- BytesRead: 962.29 MB (1009031071)
- BytesRead: 693.25 MB (726928716)
- BytesRead: 960.42 MB (1007073793)
- BytesRead: 620.05 MB (650173562)
- BytesRead: 768.43 MB (805753576)
- BytesRead: 1.39 GB (1489169920)
- BytesRead: 889.67 MB (932886343)

hadoop1:~ # fgrep "NumScannerThreadsStarted:" profile_1.txt
- NumScannerThreadsStarted: 10 (10)
- NumScannerThreadsStarted: 1 (1)
- NumScannerThreadsStarted: 15 (15)
- NumScannerThreadsStarted: 15 (15)
- NumScannerThreadsStarted: 17 (17)
- NumScannerThreadsStarted: 17 (17)
- NumScannerThreadsStarted: 14 (14)
- NumScannerThreadsStarted: 3 (3)
- NumScannerThreadsStarted: 9 (9)
- NumScannerThreadsStarted: 14 (14)
- NumScannerThreadsStarted: 9 (9)
- NumScannerThreadsStarted: 8 (8)
- NumScannerThreadsStarted: 9 (9)
- NumScannerThreadsStarted: 5 (5)
- NumScannerThreadsStarted: 8 (8)
- NumScannerThreadsStarted: 13 (13)
- NumScannerThreadsStarted: 8 (8)

hadoop1:~ # fgrep "TotalReadThroughput:" profile_1.txt
- TotalReadThroughput: 88.05 MB/sec
- TotalReadThroughput: 27.84 MB/sec
- TotalReadThroughput: 115.28 MB/sec
- TotalReadThroughput: 52.63 MB/sec
- TotalReadThroughput: 69.76 MB/sec
- TotalReadThroughput: 144.58 MB/sec
- TotalReadThroughput: 94.69 MB/sec
- TotalReadThroughput: 59.71 MB/sec
- TotalReadThroughput: 84.80 MB/sec
- TotalReadThroughput: 115.84 MB/sec
- TotalReadThroughput: 80.19 MB/sec
- TotalReadThroughput: 57.77 MB/sec
- TotalReadThroughput: 83.51 MB/sec
- TotalReadThroughput: 56.37 MB/sec
- TotalReadThroughput: 80.89 MB/sec
- TotalReadThroughput: 157.80 MB/sec
- TotalReadThroughput: 127.10 MB/sec

The  BytesRead, NumScannerThreadsStarted and TotalReadThroughput property is 3.68 GB, 1 and  27.84 MB/sec respectively for the specified node,  looks like both TotalReadThroughput and NumScannerThreadsStarted are relative.
What the symptom indicates?
The complete profile can be providee if neccessary.

[#2]

Hi David,

without the full query profile it is difficult to tell what it happening.
It could be possible that the particular node only has been assigned a
single scan range to process and therefore ended up with only one scanner
thread. Can you send the full profile?

Thanks, Lars

[#3]

Thanks for your promt response.

The full profile is too long to post once, so will post it in multi-threads.

The first fragment is as follows:

Query Runtime Profile:
Query (id=704b7e71011e9573:13c1490afe0019aa):
Summary:
Session ID: a64bfccabfc44c7a:4f426c2ace530499
Session Type: BEESWAX
Start Time: 2016-04-25 20:01:42.047887000
End Time: 2016-04-25 20:03:58.618135000
Query Type: QUERY
Query State: FINISHED
Query Status: OK
Impala Version: impalad version 2.2.0-cdh5.4.8 RELEASE (build 137d99e9f751c454a0c79f3b00302938e4984f9c)
User: testuser@HADOOP.COM
Connected User: testuser@HADOOP.COM
Delegated User:
Network Address: 192.168.1.45:57184
Default Db: default
Sql Statement: select * from ps.test_tab where col_a='538938396' and no='6'
Coordinator: hadoop13:22000
Plan:
----------------
Estimated Per-Host Requirements: Memory=4.73GB VCores=1
WARNING: The following tables are missing relevant table and/or column statistics.
ps.test_tab

F01:PLAN FRAGMENT [UNPARTITIONED]
01:EXCHANGE [UNPARTITIONED]
hosts=29 per-host-mem=unavailable
tuple-ids=0 row-size=622B cardinality=unavailable

F00:PLAN FRAGMENT [RANDOM]
DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=01, UNPARTITIONED]
00:SCAN HDFS [ps.test_tab, RANDOM]
partitions=1/13 files=243 size=24.85GB
predicates: col_a = '538938396'
table stats: unavailable
columns missing stats: *
hosts=29 per-host-mem=4.73GB
tuple-ids=0 row-size=622B cardinality=unavailable
----------------
Estimated Per-Host Mem: 5075107840
Estimated Per-Host VCores: 1
Tables Missing Stats: ps.test_tab
Request Pool: default-pool
ExecSummary:
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
----------------------------------------------------------------------------------------------------------------------
01:EXCHANGE 1 9.925us 9.925us 0 -1 0 -1.00 B UNPARTITIONED
00:SCAN HDFS 16 23s428ms 2m15s 0 -1 1.36 GB 4.73 GB ps.test_tab
Planner Timeline: 3.794ms
- Analysis finished: 1.114ms (1.114ms)
- Equivalence classes computed: 1.306ms (191.564us)
- Single node plan created: 2.322ms (1.16ms)
- Distributed plan created: 2.807ms (484.899us)
- Planning finished: 3.794ms (987.364us)
Query Timeline: 2m16s
- Start execution: 109.651us (109.651us)
- Planning finished: 5.947ms (5.837ms)
- Ready to start remote fragments: 7.248ms (1.301ms)
- Remote fragments started: 329.94ms (321.845ms)
- Rows available: 2m15s (2m15s)
- First row fetched: 2m16s (723.75ms)
- Unregister query: 2m16s (2.442ms)
ImpalaServer:
- ClientFetchWaitTimer: 724.375ms
- RowMaterializationTimer: 0ns
Execution Profile 704b7e71011e9573:13c1490afe0019aa:(Total: 2m15s, non-child: 0ns, % non-child: 0.00%)
Fragment start latencies: count: 16, last: 0.319323ns, min: 0.004591ns, max: 0.319323ns, mean: 0.036861ns, stddev: 0.080221ns
Per Node Peak Memory Usage: hadoop32:22000(557.23 MB) hadoop6:22000(1.07 GB) hadoop16:22000(824.00 MB) hadoop15:22000(1.15 GB) hadoop14:22000(774.47 MB) hadoop13:22000(488.44 MB) hadoop12:22000(175.64 MB) hadoop7:22000(1.36 GB) hadoop30:22000(968.32 MB) hadoop11:22000(480.09 MB) hadoop8:22000(556.99 MB) hadoop31:22000(678.14 MB) hadoop10:22000(931.50 MB) hadoop4:22000(652.39 MB) hadoop9:22000(700.22 MB) hadoop5:22000(459.63 MB)
- FinalizationTimer: 0ns
Coordinator Fragment F01:(Total: 2m15s, non-child: 250.809us, % non-child: 0.00%)
MemoryUsage(4s000ms): 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB
ThreadUsage(500.0ms): 1
- AverageThreadTokens: 1.00
- PeakMemoryUsage: 8.00 KB (8192)
- PerHostPeakMemUsage: 0
- PrepareTime: 34.243us
- RowsProduced: 0 (0)
- TotalCpuTime: 1s046ms
- TotalNetworkReceiveTime: 2m15s
- TotalNetworkSendTime: 0ns
- TotalStorageWaitTime: 0ns
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
EXCHANGE_NODE (id=1):(Total: 2m15s, non-child: 9.925us, % non-child: 0.00%)
BytesReceived(4s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
- BytesReceived: 0
- ConvertRowBatchTime: 1.906us
- DeserializeRowBatchTimer: 0ns
- FirstBatchArrivalWaitTime: 2m15s
- PeakMemoryUsage: 0
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- SendersBlockedTimer: 0ns
- SendersBlockedTotalTimer(*): 0ns
Averaged Fragment F00:(Total: 23s460ms, non-child: 31.950ms, % non-child: 0.14%)
split sizes: min: 619.60 MB, max: 4.61 GB, avg: 1.55 GB, stddev: 1.10 GB
completion times: min:7s229ms max:2m15s mean: 23s454ms stddev:30s248ms
execution rates: min:27.75 MB/sec max:156.48 MB/sec mean:87.51 MB/sec stddev:34.03 MB/sec
num instances: 16
- AverageThreadTokens: 7.04
- PeakMemoryUsage: 745.07 MB (781258467)
- PerHostPeakMemUsage: 745.07 MB (781258979)
- PrepareTime: 351.778us
- RowsProduced: 0 (0)
- TotalCpuTime: 1m57s
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 123ns
- TotalStorageWaitTime: 1m12s
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 22.240us, non-child: 22.240us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 23s428ms, non-child: 23s428ms, % non-child: 100.00%)
- AverageHdfsReadThreadConcurrency: 6.21
- AverageScannerThreadConcurrency: 6.04
- BytesRead: 1.55 GB (1668901870)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.46 GB (1565486679)
- BytesReadRemoteUnexpected: 98.62 MB (103415191)
- BytesReadShortCircuit: 1.46 GB (1565486679)
- DecompressionTime: 267.18ms
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 7 (7)
- NumScannerThreadsStarted: 10 (10)
- PeakMemoryUsage: 745.05 MB (781246179)
- PerReadThreadRawHdfsThroughput: 14.41 MB/sec
- RemoteScanRanges: 56 (56)
- RowsRead: 22.57M (22567007)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 15 (15)
- ScannerThreadsInvoluntaryContextSwitches: 1.17K (1166)
- ScannerThreadsTotalWallClockTime: 1m52s
- MaterializeTupleTime(*): 38s478ms
- ScannerThreadsSysTime: 151.6ms
- ScannerThreadsUserTime: 38s641ms
- ScannerThreadsVoluntaryContextSwitches: 800 (800)
- TotalRawHdfsReadTime(*): 1m49s
- TotalReadThroughput: 88.05 MB/sec
Fragment F00:
Instance 704b7e71011e9573:13c1490afe0019b2 (host=hadoop12:22000):(Total: 2m15s, non-child: 1.479ms, % non-child: 0.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:7/519.57 MB 1:1/146.98 MB 2:3/278.56 MB 3:6/558.73 MB 4:3/324.40 MB 5:1/121.36 MB 6:4/381.73 MB 7:3/391.78 MB 8:3/249.89 MB 10:4/461.08 MB 11:3/334.88 MB
MemoryUsage(4s000ms): 79.95 MB, 86.50 MB, 93.83 MB, 98.06 MB, 99.87 MB, 63.14 MB, 65.26 MB, 85.37 MB, 61.75 MB, 109.14 MB, 85.33 MB, 84.14 MB, 61.62 MB, 61.50 MB, 107.47 MB, 39.42 MB, 66.96 MB, 104.71 MB, 102.27 MB, 100.25 MB, 50.51 MB, 92.83 MB, 93.34 MB, 51.29 MB, 102.11 MB, 79.02 MB, 100.86 MB, 94.76 MB, 52.75 MB, 102.00 MB, 61.42 MB, 105.48 MB, 54.05 MB
ThreadUsage(4s000ms): 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2
- AverageThreadTokens: 2.00
- PeakMemoryUsage: 175.64 MB (184170048)
- PerHostPeakMemUsage: 175.64 MB (184170048)
- PrepareTime: 280.49us
- RowsProduced: 0 (0)
- TotalCpuTime: 3m27s
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 229ns
- TotalStorageWaitTime: 1m3s
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 14.736us, non-child: 14.736us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 2m15s, non-child: 2m15s, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:7/519.57 MB 1:1/146.98 MB 2:3/278.56 MB 3:6/558.73 MB 4:3/324.40 MB 5:1/121.36 MB 6:4/381.73 MB 7:3/391.78 MB 8:3/249.89 MB 10:4/461.08 MB 11:3/334.88 MB
Hdfs Read Thread Concurrency Bucket: 0:64.94% 1:3.69% 2:1.476% 3:1.107% 4:2.214% 5:6.642% 6:19.93% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0%
File Formats: PARQUET/NONE:2090
ExecOption: Codegen enabled: 0 out of 38
BytesRead(4s000ms): 64.87 MB, 181.39 MB, 313.01 MB, 441.49 MB, 589.11 MB, 703.53 MB, 759.85 MB, 858.54 MB, 953.43 MB, 1.07 GB, 1.20 GB, 1.29 GB, 1.37 GB, 1.48 GB, 1.62 GB, 1.67 GB, 1.71 GB, 1.80 GB, 1.93 GB, 2.05 GB, 2.13 GB, 2.25 GB, 2.38 GB, 2.47 GB, 2.62 GB, 2.74 GB, 2.85 GB, 2.97 GB, 3.08 GB, 3.22 GB, 3.32 GB, 3.45 GB, 3.54 GB
- AverageHdfsReadThreadConcurrency: 1.72
- AverageScannerThreadConcurrency: 1.00
- BytesRead: 3.68 GB (3955651251)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 3.32 GB (3566509512)
- BytesReadRemoteUnexpected: 371.11 MB (389141739)
- BytesReadShortCircuit: 3.32 GB (3566509512)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 11 (11)
- NumScannerThreadsStarted: 1 (1)
- PeakMemoryUsage: 175.63 MB (184157760)
- PerReadThreadRawHdfsThroughput: 16.33 MB/sec
- RemoteScanRanges: 168 (168)
- RowsRead: 51.40M (51395362)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 38 (38)
- ScannerThreadsInvoluntaryContextSwitches: 767 (767)
- ScannerThreadsTotalWallClockTime: 2m15s
- MaterializeTupleTime(*): 1m11s
- ScannerThreadsSysTime: 152.9ms
- ScannerThreadsUserTime: 1m11s
- ScannerThreadsVoluntaryContextSwitches: 1.38K (1379)
- TotalRawHdfsReadTime(*): 3m51s
- TotalReadThroughput: 27.84 MB/sec
Instance 704b7e71011e9573:13c1490afe0019af (host=hadoop15:22000):(Total: 41s242ms, non-child: 49.114ms, % non-child: 0.12%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:3/364.32 MB 1:4/372.05 MB 2:3/289.97 MB 3:4/639.33 MB 4:4/493.65 MB 5:3/143.62 MB 6:2/154.68 MB 7:6/589.98 MB 8:4/516.45 MB 9:6/712.48 MB 10:6/445.67 MB
MemoryUsage(1s000ms): 230.80 MB, 474.91 MB, 635.97 MB, 912.24 MB, 1.11 GB, 982.84 MB, 762.34 MB, 576.20 MB, 477.20 MB, 637.89 MB, 900.22 MB, 978.83 MB, 1.04 GB, 1.07 GB, 1.05 GB, 905.77 MB, 825.52 MB, 867.11 MB, 841.12 MB, 954.98 MB, 860.56 MB, 830.51 MB, 881.14 MB, 821.94 MB, 703.41 MB, 637.89 MB, 672.36 MB, 574.61 MB, 574.55 MB, 446.31 MB, 507.56 MB, 527.58 MB, 424.05 MB, 447.66 MB, 342.97 MB, 263.54 MB, 273.37 MB, 241.94 MB, 248.92 MB, 205.48 MB, 136.53 MB
ThreadUsage(1s000ms): 15, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 15, 14, 13, 12, 12, 11, 10, 9, 8, 8, 6, 5, 5, 5, 5, 4, 3, 3, 3, 3, 2, 2
- AverageThreadTokens: 11.13
- PeakMemoryUsage: 1.15 GB (1235100464)
- PerHostPeakMemUsage: 1.15 GB (1235100464)
- PrepareTime: 450.3us
- RowsProduced: 0 (0)
- TotalCpuTime: 6m58s
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 97ns
- TotalStorageWaitTime: 4m40s
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 23.286us, non-child: 23.286us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 41s193ms, non-child: 41s193ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:3/364.32 MB 1:4/372.05 MB 2:3/289.97 MB 3:4/639.33 MB 4:4/493.65 MB 5:3/143.62 MB 6:2/154.68 MB 7:6/589.98 MB 8:4/516.45 MB 9:6/712.48 MB 10:6/445.67 MB
Hdfs Read Thread Concurrency Bucket: 0:24.39% 1:9.756% 2:3.659% 3:2.439% 4:6.098% 5:2.439% 6:8.537% 7:2.439% 8:3.659% 9:1.22% 10:0% 11:2.439% 12:1.22% 13:1.22% 14:3.659% 15:26.83%
File Formats: PARQUET/NONE:2310 PARQUET/SNAPPY:165
ExecOption: Codegen enabled: 0 out of 45
BytesRead(1s000ms): 103.87 MB, 321.67 MB, 498.15 MB, 838.14 MB, 1.10 GB, 1.16 GB, 1.27 GB, 1.41 GB, 1.51 GB, 1.76 GB, 2.12 GB, 2.30 GB, 2.42 GB, 2.60 GB, 2.80 GB, 2.93 GB, 3.09 GB, 3.31 GB, 3.49 GB, 3.68 GB, 3.82 GB, 3.91 GB, 4.04 GB, 4.10 GB, 4.15 GB, 4.18 GB, 4.24 GB, 4.33 GB, 4.36 GB, 4.47 GB, 4.55 GB, 4.55 GB, 4.55 GB, 4.56 GB, 4.57 GB, 4.59 GB, 4.61 GB, 4.61 GB, 4.62 GB, 4.62 GB, 4.62 GB
- AverageHdfsReadThreadConcurrency: 7.91
- AverageScannerThreadConcurrency: 10.13
- BytesRead: 4.62 GB (4955876955)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 4.30 GB (4617848579)
- BytesReadRemoteUnexpected: 322.37 MB (338028376)
- BytesReadShortCircuit: 4.30 GB (4617848579)
- DecompressionTime: 2s446ms
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 11 (11)
- NumScannerThreadsStarted: 15 (15)
- PeakMemoryUsage: 1.15 GB (1235088176)
- PerReadThreadRawHdfsThroughput: 14.59 MB/sec
- RemoteScanRanges: 168 (168)
- RowsRead: 76.26M (76259233)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 45 (45)
- ScannerThreadsInvoluntaryContextSwitches: 6.97K (6970)
- ScannerThreadsTotalWallClockTime: 6m58s
- MaterializeTupleTime(*): 2m14s
- ScannerThreadsSysTime: 668.37ms
- ScannerThreadsUserTime: 2m16s
- ScannerThreadsVoluntaryContextSwitches: 2.85K (2846)
- TotalRawHdfsReadTime(*): 5m24s
- TotalReadThroughput: 115.28 MB/sec
Instance 704b7e71011e9573:13c1490afe0019ba (host=hadoop10:22000):(Total: 30s962ms, non-child: 315.670ms, % non-child: 1.02%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/121.46 MB 2:2/228.69 MB 3:1/98.85 MB 4:1/96.97 MB 5:1/144.57 MB 7:1/133.90 MB 9:4/518.27 MB 10:4/287.31 MB
MemoryUsage(500.0ms): 0, 107.28 MB, 184.44 MB, 325.77 MB, 591.49 MB, 658.19 MB, 696.13 MB, 656.14 MB, 757.04 MB, 795.58 MB, 920.26 MB, 817.65 MB, 502.89 MB, 534.70 MB, 387.47 MB, 391.78 MB, 360.46 MB, 301.41 MB, 243.39 MB, 275.44 MB, 298.12 MB, 322.89 MB, 335.99 MB, 338.51 MB, 338.51 MB, 346.52 MB, 346.52 MB, 362.53 MB, 253.40 MB, 258.90 MB, 275.57 MB, 291.52 MB, 293.15 MB, 294.54 MB, 293.25 MB, 365.11 MB, 343.14 MB, 296.18 MB, 341.48 MB, 291.62 MB, 286.98 MB, 296.68 MB, 296.68 MB, 296.68 MB, 296.69 MB, 296.69 MB, 298.19 MB, 300.23 MB, 292.76 MB, 335.76 MB, 323.65 MB, 352.95 MB, 311.42 MB, 319.02 MB, 190.79 MB, 173.40 MB, 181.94 MB, 179.34 MB, 211.58 MB, 139.69 MB, 155.11 MB, 187.90 MB
ThreadUsage(500.0ms): 1, 15, 15, 15, 15, 16, 16, 14, 13, 13, 13, 13, 9, 9, 7, 7, 7, 6, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 2, 2, 2, 2, 2, 2, 2, 2
- AverageThreadTokens: 6.10
- PeakMemoryUsage: 931.50 MB (976747384)
- PerHostPeakMemUsage: 931.50 MB (976747384)
- PrepareTime: 212.472us
- RowsProduced: 0 (0)
- TotalCpuTime: 2m39s
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 230ns
- TotalStorageWaitTime: 1m49s
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 35.319us, non-child: 35.319us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 30s646ms, non-child: 30s646ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/121.46 MB 2:2/228.69 MB 3:1/98.85 MB 4:1/96.97 MB 5:1/144.57 MB 7:1/133.90 MB 9:4/518.27 MB 10:4/287.31 MB
Hdfs Read Thread Concurrency Bucket: 0:42.62% 1:13.11% 2:4.918% 3:6.557% 4:13.11% 5:3.279% 6:0% 7:0% 8:1.639% 9:0% 10:0% 11:0% 12:1.639% 13:0% 14:1.639% 15:11.48%
File Formats: PARQUET/NONE:770 PARQUET/SNAPPY:55
ExecOption: Codegen enabled: 0 out of 15
BytesRead(500.0ms): 0, 2.10 MB, 76.28 MB, 200.65 MB, 422.32 MB, 447.62 MB, 536.97 MB, 652.68 MB, 805.66 MB, 922.02 MB, 1.05 GB, 1.07 GB, 1.09 GB, 1.16 GB, 1.19 GB, 1.20 GB, 1.22 GB, 1.24 GB, 1.25 GB, 1.26 GB, 1.26 GB, 1.26 GB, 1.26 GB, 1.27 GB, 1.27 GB, 1.27 GB, 1.27 GB, 1.27 GB, 1.28 GB, 1.28 GB, 1.29 GB, 1.31 GB, 1.31 GB, 1.31 GB, 1.34 GB, 1.38 GB, 1.39 GB, 1.41 GB, 1.42 GB, 1.42 GB, 1.42 GB, 1.44 GB, 1.44 GB, 1.44 GB, 1.44 GB, 1.44 GB, 1.44 GB, 1.45 GB, 1.46 GB, 1.53 GB, 1.56 GB, 1.56 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB
- AverageHdfsReadThreadConcurrency: 3.70
- AverageScannerThreadConcurrency: 5.18
- BytesRead: 1.59 GB (1710640190)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.55 GB (1667758557)
- BytesReadRemoteUnexpected: 40.90 MB (42881633)
- BytesReadShortCircuit: 1.55 GB (1667758557)
- DecompressionTime: 1s826ms
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 8 (8)
- NumScannerThreadsStarted: 15 (15)
- PeakMemoryUsage: 931.49 MB (976735096)
- PerReadThreadRawHdfsThroughput: 14.56 MB/sec
- RemoteScanRanges: 56 (56)
- RowsRead: 26.93M (26931028)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 15 (15)
- ScannerThreadsInvoluntaryContextSwitches: 979 (979)
- ScannerThreadsTotalWallClockTime: 2m39s
- MaterializeTupleTime(*): 44s978ms
- ScannerThreadsSysTime: 288.13ms
- ScannerThreadsUserTime: 45s830ms
- ScannerThreadsVoluntaryContextSwitches: 1.72K (1723)
- TotalRawHdfsReadTime(*): 1m52s
- TotalReadThroughput: 52.63 MB/sec
Instance 704b7e71011e9573:13c1490afe0019b8 (host=hadoop9:22000):(Total: 21s516ms, non-child: 1.800ms, % non-child: 0.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/132.06 MB 11:1/54.86 MB 2:1/72.90 MB 3:1/45.58 MB 4:3/390.45 MB 5:4/367.45 MB 7:2/177.69 MB 8:1/63.92 MB 9:1/139.10 MB 10:2/89.20 MB
MemoryUsage(500.0ms): 5.24 MB, 97.25 MB, 220.33 MB, 435.04 MB, 488.75 MB, 678.23 MB, 699.41 MB, 661.17 MB, 583.29 MB, 453.71 MB, 374.29 MB, 415.89 MB, 438.55 MB, 423.82 MB, 491.63 MB, 542.10 MB, 549.99 MB, 568.94 MB, 553.48 MB, 509.58 MB, 529.10 MB, 506.61 MB, 459.41 MB, 415.77 MB, 501.36 MB, 547.47 MB, 569.50 MB, 524.37 MB, 424.03 MB, 317.78 MB, 294.50 MB, 312.50 MB, 250.79 MB, 251.13 MB, 267.86 MB, 277.08 MB, 280.64 MB, 297.34 MB, 298.61 MB, 341.80 MB, 294.50 MB, 178.31 MB, 89.47 MB, 65.70 MB
ThreadUsage(500.0ms): 18, 18, 18, 18, 17, 17, 17, 17, 17, 15, 14, 14, 14, 13, 13, 13, 12, 12, 12, 11, 11, 10, 9, 8, 8, 8, 8, 8, 7, 6, 6, 6, 5, 5, 5, 5, 5, 5, 5, 4, 4, 3, 2, 2
- AverageThreadTokens: 10.11
- PeakMemoryUsage: 700.22 MB (734232408)
- PerHostPeakMemUsage: 700.22 MB (734232408)
- PrepareTime: 317.552us
- RowsProduced: 0 (0)
- TotalCpuTime: 3m16s
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 107ns
- TotalStorageWaitTime: 2m37s
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 24.317us, non-child: 24.317us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 21s514ms, non-child: 21s514ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/132.06 MB 11:1/54.86 MB 2:1/72.90 MB 3:1/45.58 MB 4:3/390.45 MB 5:4/367.45 MB 7:2/177.69 MB 8:1/63.92 MB 9:1/139.10 MB 10:2/89.20 MB
Hdfs Read Thread Concurrency Bucket: 0:31.82% 1:9.091% 2:9.091% 3:9.091% 4:6.818% 5:6.818% 6:2.273% 7:6.818% 8:2.273% 9:0% 10:0% 11:0% 12:0% 13:6.818% 14:2.273% 15:6.818%
File Formats: PARQUET/NONE:935
ExecOption: Codegen enabled: 0 out of 17
BytesRead(500.0ms): 0, 18.88 MB, 87.98 MB, 249.89 MB, 378.11 MB, 584.27 MB, 622.83 MB, 655.74 MB, 660.83 MB, 669.58 MB, 680.35 MB, 702.98 MB, 711.95 MB, 737.54 MB, 770.92 MB, 847.56 MB, 886.72 MB, 967.54 MB, 998.15 MB, 1005.62 MB, 1.01 GB, 1.02 GB, 1.04 GB, 1.07 GB, 1.15 GB, 1.23 GB, 1.31 GB, 1.31 GB, 1.31 GB, 1.31 GB, 1.31 GB, 1.32 GB, 1.32 GB, 1.33 GB, 1.33 GB, 1.35 GB, 1.36 GB, 1.38 GB, 1.38 GB, 1.50 GB, 1.50 GB, 1.50 GB, 1.50 GB, 1.50 GB
- AverageHdfsReadThreadConcurrency: 4.52
- AverageScannerThreadConcurrency: 9.11
- BytesRead: 1.50 GB (1609316761)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.32 GB (1418197209)
- BytesReadRemoteUnexpected: 182.27 MB (191119552)
- BytesReadShortCircuit: 1.32 GB (1418197209)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 10 (10)
- NumScannerThreadsStarted: 17 (17)
- PeakMemoryUsage: 700.21 MB (734220120)
- PerReadThreadRawHdfsThroughput: 14.42 MB/sec
- RemoteScanRanges: 112 (112)
- RowsRead: 21.32M (21316737)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 17 (17)
- ScannerThreadsInvoluntaryContextSwitches: 1.95K (1948)
- ScannerThreadsTotalWallClockTime: 3m16s
- MaterializeTupleTime(*): 37s777ms
- ScannerThreadsSysTime: 240.8ms
- ScannerThreadsUserTime: 37s614ms
- ScannerThreadsVoluntaryContextSwitches: 1.19K (1190)
- TotalRawHdfsReadTime(*): 1m46s
- TotalReadThroughput: 69.76 MB/sec
Instance 704b7e71011e9573:13c1490afe0019b3 (host=hadoop7:22000):(Total: 18s713ms, non-child: 1.174ms, % non-child: 0.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/117.90 MB 11:2/186.31 MB 1:3/213.89 MB 3:3/396.93 MB 4:2/229.78 MB 6:3/343.63 MB 7:1/140.95 MB 8:3/347.39 MB 9:1/59.54 MB 10:5/636.14 MB
MemoryUsage(500.0ms): 36.09 MB, 171.64 MB, 279.90 MB, 330.08 MB, 482.92 MB, 557.69 MB, 631.92 MB, 683.42 MB, 813.40 MB, 856.23 MB, 890.94 MB, 991.39 MB, 1.06 GB, 1.13 GB, 1.34 GB, 1.24 GB, 1.11 GB, 1016.00 MB, 1.13 GB, 941.57 MB, 769.10 MB, 502.54 MB, 446.45 MB, 346.56 MB, 245.62 MB, 190.13 MB, 236.46 MB, 247.23 MB, 260.49 MB, 264.49 MB, 270.19 MB, 273.72 MB, 239.61 MB, 243.18 MB, 211.42 MB, 87.91 MB, 80.00 MB
ThreadUsage(500.0ms): 15, 15, 15, 15, 15, 16, 16, 16, 16, 18, 18, 18, 18, 18, 18, 18, 18, 16, 14, 13, 11, 7, 7, 6, 5, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 2, 2
- AverageThreadTokens: 10.89
- PeakMemoryUsage: 1.36 GB (1465641912)
- PerHostPeakMemUsage: 1.36 GB (1465641912)
- PrepareTime: 254.381us
- RowsProduced: 0 (0)
- TotalCpuTime: 3m6s
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 93ns
- TotalStorageWaitTime: 2m6s
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 14.645us, non-child: 14.645us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 18s712ms, non-child: 18s712ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/117.90 MB 11:2/186.31 MB 1:3/213.89 MB 3:3/396.93 MB 4:2/229.78 MB 6:3/343.63 MB 7:1/140.95 MB 8:3/347.39 MB 9:1/59.54 MB 10:5/636.14 MB
Hdfs Read Thread Concurrency Bucket: 0:24.32% 1:2.703% 2:5.405% 3:2.703% 4:8.108% 5:8.108% 6:5.405% 7:2.703% 8:0% 9:0% 10:0% 11:0% 12:2.703% 13:0% 14:0% 15:37.84%
File Formats: PARQUET/NONE:1320
ExecOption: Codegen enabled: 0 out of 24
BytesRead(500.0ms): 100.00 KB, 7.96 MB, 66.44 MB, 146.37 MB, 295.79 MB, 358.55 MB, 421.58 MB, 519.90 MB, 724.48 MB, 835.77 MB, 903.78 MB, 970.56 MB, 1.19 GB, 1.29 GB, 1.53 GB, 1.71 GB, 1.82 GB, 1.87 GB, 2.16 GB, 2.18 GB, 2.23 GB, 2.32 GB, 2.37 GB, 2.38 GB, 2.39 GB, 2.41 GB, 2.44 GB, 2.46 GB, 2.47 GB, 2.49 GB, 2.50 GB, 2.50 GB, 2.56 GB, 2.61 GB, 2.61 GB, 2.61 GB, 2.61 GB
- AverageHdfsReadThreadConcurrency: 10.49
- AverageScannerThreadConcurrency: 9.89
- BytesRead: 2.61 GB (2804563939)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 2.43 GB (2612224351)
- BytesReadRemoteUnexpected: 183.43 MB (192339588)
- BytesReadShortCircuit: 2.43 GB (2612224351)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 10 (10)
- NumScannerThreadsStarted: 17 (17)
- PeakMemoryUsage: 1.36 GB (1465629624)
- PerReadThreadRawHdfsThroughput: 13.32 MB/sec
- RemoteScanRanges: 112 (112)
- RowsRead: 35.75M (35745185)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 24 (24)
- ScannerThreadsInvoluntaryContextSwitches: 1.75K (1752)
- ScannerThreadsTotalWallClockTime: 3m6s
- MaterializeTupleTime(*): 58s922ms
- ScannerThreadsSysTime: 280.12ms
- ScannerThreadsUserTime: 58s839ms
- ScannerThreadsVoluntaryContextSwitches: 1.03K (1035)
- TotalRawHdfsReadTime(*): 3m20s
- TotalReadThroughput: 144.58 MB/sec
Instance 704b7e71011e9573:13c1490afe0019b9 (host=hadoop4:22000):(Total: 15s643ms, non-child: 958.853us, % non-child: 0.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/129.56 MB 0:3/382.82 MB 3:3/252.99 MB 5:2/184.55 MB 7:3/310.95 MB 8:2/205.49 MB
MemoryUsage(500.0ms): 75.78 MB, 192.04 MB, 316.14 MB, 428.84 MB, 503.28 MB, 480.35 MB, 627.36 MB, 530.13 MB, 502.11 MB, 442.27 MB, 354.66 MB, 473.23 MB, 489.97 MB, 535.28 MB, 442.60 MB, 402.28 MB, 395.24 MB, 404.04 MB, 416.96 MB, 442.66 MB, 452.68 MB, 481.41 MB, 471.95 MB, 424.20 MB, 494.05 MB, 471.53 MB, 523.33 MB, 399.42 MB, 272.36 MB, 183.12 MB, 120.26 MB
ThreadUsage(500.0ms): 15, 15, 15, 15, 15, 14, 14, 12, 12, 12, 11, 10, 10, 10, 9, 8, 8, 7, 7, 7, 7, 7, 7, 6, 6, 6, 6, 5, 4, 3, 3
- AverageThreadTokens: 9.23
- PeakMemoryUsage: 652.39 MB (684083360)
- PerHostPeakMemUsage: 652.39 MB (684083360)
- PrepareTime: 450.221us
- RowsProduced: 0 (0)
- TotalCpuTime: 2m11s
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 104ns
- TotalStorageWaitTime: 1m35s
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 22.22us, non-child: 22.22us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0

[#4]

The second fragment is as follows:

HDFS_SCAN_NODE (id=0):(Total: 15s642ms, non-child: 15s642ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/129.56 MB 0:3/382.82 MB 3:3/252.99 MB 5:2/184.55 MB 7:3/310.95 MB 8:2/205.49 MB
Hdfs Read Thread Concurrency Bucket: 0:22.58% 1:3.226% 2:3.226% 3:6.452% 4:3.226% 5:6.452% 6:6.452% 7:3.226% 8:6.452% 9:9.677% 10:9.677% 11:0% 12:3.226% 13:0% 14:0% 15:16.13%
File Formats: PARQUET/NONE:770
ExecOption: Codegen enabled: 0 out of 14
BytesRead(500.0ms): 322.81 KB, 64.60 MB, 193.73 MB, 295.27 MB, 337.90 MB, 404.14 MB, 617.34 MB, 650.05 MB, 666.97 MB, 680.94 MB, 714.84 MB, 903.24 MB, 945.56 MB, 997.88 MB, 1.00 GB, 1.02 GB, 1.05 GB, 1.11 GB, 1.15 GB, 1.18 GB, 1.18 GB, 1.20 GB, 1.21 GB, 1.26 GB, 1.36 GB, 1.36 GB, 1.43 GB, 1.43 GB, 1.43 GB, 1.43 GB, 1.43 GB
- AverageHdfsReadThreadConcurrency: 6.81
- AverageScannerThreadConcurrency: 8.23
- BytesRead: 1.43 GB (1538913826)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.43 GB (1538913826)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.43 GB (1538913826)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 6 (6)
- NumScannerThreadsStarted: 14 (14)
- PeakMemoryUsage: 652.38 MB (684071072)
- PerReadThreadRawHdfsThroughput: 13.67 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 19.70M (19702157)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 14 (14)
- ScannerThreadsInvoluntaryContextSwitches: 1.68K (1681)
- ScannerThreadsTotalWallClockTime: 2m11s
- MaterializeTupleTime(*): 33s178ms
- ScannerThreadsSysTime: 124.4ms
- ScannerThreadsUserTime: 33s098ms
- ScannerThreadsVoluntaryContextSwitches: 577 (577)
- TotalRawHdfsReadTime(*): 1m47s
- TotalReadThroughput: 94.69 MB/sec
Instance 704b7e71011e9573:13c1490afe0019bb (host=hadoop5:22000):(Total: 13s683ms, non-child: 866.247us, % non-child: 0.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/61.51 MB 1:1/136.60 MB 2:1/130.20 MB 5:1/139.10 MB 7:1/123.58 MB 8:1/139.49 MB 10:1/104.81 MB
MemoryUsage(500.0ms): 2.36 MB, 134.74 MB, 189.52 MB, 281.12 MB, 356.36 MB, 324.38 MB, 244.15 MB, 142.80 MB, 181.76 MB, 154.80 MB, 285.80 MB, 381.28 MB, 379.78 MB, 419.75 MB, 403.90 MB, 304.33 MB, 338.74 MB, 314.96 MB, 291.05 MB, 296.32 MB, 290.05 MB, 274.34 MB, 250.56 MB, 242.62 MB, 210.87 MB, 202.93 MB, 86.76 MB, 78.86 MB
ThreadUsage(500.0ms): 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 2, 2
- AverageThreadTokens: 3.64
- PeakMemoryUsage: 459.63 MB (481955896)
- PerHostPeakMemUsage: 459.63 MB (481955896)
- PrepareTime: 212.743us
- RowsProduced: 0 (0)
- TotalCpuTime: 37s570ms
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 89ns
- TotalStorageWaitTime: 12s291ms
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 14.942us, non-child: 14.942us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 13s682ms, non-child: 13s682ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/61.51 MB 1:1/136.60 MB 2:1/130.20 MB 5:1/139.10 MB 7:1/123.58 MB 8:1/139.49 MB 10:1/104.81 MB
Hdfs Read Thread Concurrency Bucket: 0:46.43% 1:3.571% 2:0% 3:0% 4:0% 5:0% 6:32.14% 7:0% 8:0% 9:0% 10:3.571% 11:3.571% 12:3.571% 13:0% 14:0% 15:7.143%
File Formats: PARQUET/NONE:385
ExecOption: Codegen enabled: 0 out of 7
BytesRead(500.0ms): 0, 9.58 MB, 96.03 MB, 218.62 MB, 290.17 MB, 290.17 MB, 290.27 MB, 299.51 MB, 311.65 MB, 365.35 MB, 476.58 MB, 588.33 MB, 607.07 MB, 705.64 MB, 705.64 MB, 705.64 MB, 706.91 MB, 706.91 MB, 714.91 MB, 750.02 MB, 835.93 MB, 835.93 MB, 835.93 MB, 835.93 MB, 835.93 MB, 835.93 MB, 835.93 MB, 835.93 MB
- AverageHdfsReadThreadConcurrency: 4.36
- AverageScannerThreadConcurrency: 2.64
- BytesRead: 835.93 MB (876532952)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 835.93 MB (876532952)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 835.93 MB (876532952)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 7 (7)
- NumScannerThreadsStarted: 3 (3)
- PeakMemoryUsage: 459.62 MB (481943608)
- PerReadThreadRawHdfsThroughput: 13.74 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 11.04M (11040327)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 7 (7)
- ScannerThreadsInvoluntaryContextSwitches: 653 (653)
- ScannerThreadsTotalWallClockTime: 36s174ms
- MaterializeTupleTime(*): 23s772ms
- ScannerThreadsSysTime: 28.1ms
- ScannerThreadsUserTime: 23s797ms
- ScannerThreadsVoluntaryContextSwitches: 151 (151)
- TotalRawHdfsReadTime(*): 1m
- TotalReadThroughput: 59.71 MB/sec
Instance 704b7e71011e9573:13c1490afe0019b6 (host=hadoop8:22000):(Total: 12s579ms, non-child: 132.751ms, % non-child: 1.06%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/139.13 MB 2:2/273.38 MB 3:1/125.03 MB 4:2/127.04 MB 6:2/257.95 MB 7:1/136.70 MB
MemoryUsage(500.0ms): 3.80 MB, 80.67 MB, 106.98 MB, 301.40 MB, 356.72 MB, 351.27 MB, 477.72 MB, 548.32 MB, 391.00 MB, 367.36 MB, 316.51 MB, 269.24 MB, 142.44 MB, 204.52 MB, 235.67 MB, 317.67 MB, 349.00 MB, 355.88 MB, 487.87 MB, 511.48 MB, 440.17 MB, 234.84 MB, 211.11 MB, 83.77 MB, 75.86 MB
ThreadUsage(500.0ms): 10, 10, 10, 10, 10, 9, 9, 9, 7, 7, 7, 7, 5, 5, 5, 5, 5, 5, 5, 5, 5, 3, 3, 2, 2
- AverageThreadTokens: 6.40
- PeakMemoryUsage: 556.99 MB (584048344)
- PerHostPeakMemUsage: 556.99 MB (584048344)
- PrepareTime: 1.455ms
- RowsProduced: 0 (0)
- TotalCpuTime: 1m7s
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 235ns
- TotalStorageWaitTime: 42s377ms
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 19.63us, non-child: 19.63us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 12s446ms, non-child: 12s446ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/139.13 MB 2:2/273.38 MB 3:1/125.03 MB 4:2/127.04 MB 6:2/257.95 MB 7:1/136.70 MB
Hdfs Read Thread Concurrency Bucket: 0:28% 1:0% 2:4% 3:0% 4:4% 5:8% 6:4% 7:12% 8:12% 9:0% 10:8% 11:8% 12:8% 13:0% 14:0% 15:4%
File Formats: PARQUET/NONE:495
ExecOption: Codegen enabled: 0 out of 9
BytesRead(500.0ms): 0, 14.32 MB, 50.98 MB, 171.48 MB, 229.69 MB, 315.23 MB, 449.35 MB, 559.26 MB, 583.26 MB, 586.25 MB, 612.73 MB, 615.09 MB, 644.34 MB, 667.70 MB, 685.74 MB, 746.37 MB, 779.65 MB, 831.11 MB, 999.53 MB, 1.04 GB, 1.04 GB, 1.04 GB, 1.04 GB, 1.04 GB, 1.04 GB
- AverageHdfsReadThreadConcurrency: 5.92
- AverageScannerThreadConcurrency: 5.40
- BytesRead: 1.04 GB (1111533925)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.04 GB (1111533925)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.04 GB (1111533925)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 6 (6)
- NumScannerThreadsStarted: 9 (9)
- PeakMemoryUsage: 556.98 MB (584036056)
- PerReadThreadRawHdfsThroughput: 14.17 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 14.04M (14038683)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 9 (9)
- ScannerThreadsInvoluntaryContextSwitches: 1.00K (1004)
- ScannerThreadsTotalWallClockTime: 1m7s
- MaterializeTupleTime(*): 20s872ms
- ScannerThreadsSysTime: 84.1ms
- ScannerThreadsUserTime: 20s857ms
- ScannerThreadsVoluntaryContextSwitches: 412 (412)
- TotalRawHdfsReadTime(*): 1m14s
- TotalReadThroughput: 84.80 MB/sec
Instance 704b7e71011e9573:13c1490afe0019b4 (host=hadoop30:22000):(Total: 12s350ms, non-child: 981.652us, % non-child: 0.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/111.69 MB 2:2/131.88 MB 3:1/106.78 MB 4:2/227.32 MB 5:2/241.62 MB 7:4/432.52 MB 8:1/50.53 MB 10:1/144.42 MB
MemoryUsage(500.0ms): 62.09 MB, 230.58 MB, 372.08 MB, 736.11 MB, 901.90 MB, 878.44 MB, 878.81 MB, 771.93 MB, 554.02 MB, 460.93 MB, 442.84 MB, 425.77 MB, 505.04 MB, 531.74 MB, 492.57 MB, 493.51 MB, 441.50 MB, 414.66 MB, 407.47 MB, 278.68 MB, 402.63 MB, 339.25 MB, 220.51 MB, 181.05 MB, 68.42 MB
ThreadUsage(500.0ms): 15, 15, 15, 15, 15, 14, 14, 13, 11, 10, 9, 9, 8, 8, 7, 7, 7, 7, 7, 5, 4, 4, 3, 3, 2
- AverageThreadTokens: 9.08
- PeakMemoryUsage: 968.32 MB (1015356592)
- PerHostPeakMemUsage: 968.32 MB (1015356592)
- PrepareTime: 292.223us
- RowsProduced: 0 (0)
- TotalCpuTime: 1m42s
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 120ns
- TotalStorageWaitTime: 1m7s
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 40.134us, non-child: 40.134us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 12s349ms, non-child: 12s349ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/111.69 MB 2:2/131.88 MB 3:1/106.78 MB 4:2/227.32 MB 5:2/241.62 MB 7:4/432.52 MB 8:1/50.53 MB 10:1/144.42 MB
Hdfs Read Thread Concurrency Bucket: 0:24% 1:4% 2:0% 3:8% 4:0% 5:12% 6:4% 7:4% 8:4% 9:12% 10:0% 11:4% 12:0% 13:0% 14:0% 15:24%
File Formats: PARQUET/NONE:770
ExecOption: Codegen enabled: 0 out of 14
BytesRead(500.0ms): 400.00 KB, 65.19 MB, 227.21 MB, 501.10 MB, 660.19 MB, 765.94 MB, 877.98 MB, 877.98 MB, 879.00 MB, 883.91 MB, 942.57 MB, 959.97 MB, 1.08 GB, 1.13 GB, 1.16 GB, 1.19 GB, 1.20 GB, 1.20 GB, 1.20 GB, 1.25 GB, 1.41 GB, 1.41 GB, 1.41 GB, 1.41 GB, 1.41 GB
- AverageHdfsReadThreadConcurrency: 7.96
- AverageScannerThreadConcurrency: 8.08
- BytesRead: 1.41 GB (1518383151)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.26 GB (1348087343)
- BytesReadRemoteUnexpected: 162.41 MB (170295808)
- BytesReadShortCircuit: 1.26 GB (1348087343)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 8 (8)
- NumScannerThreadsStarted: 14 (14)
- PeakMemoryUsage: 968.31 MB (1015344304)
- PerReadThreadRawHdfsThroughput: 15.16 MB/sec
- RemoteScanRanges: 112 (112)
- RowsRead: 19.59M (19590941)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 14 (14)
- ScannerThreadsInvoluntaryContextSwitches: 399 (399)
- ScannerThreadsTotalWallClockTime: 1m42s
- MaterializeTupleTime(*): 33s893ms
- ScannerThreadsSysTime: 72.1ms
- ScannerThreadsUserTime: 33s950ms
- ScannerThreadsVoluntaryContextSwitches: 680 (680)
- TotalRawHdfsReadTime(*): 1m35s
- TotalReadThroughput: 115.84 MB/sec
Instance 704b7e71011e9573:13c1490afe0019ad (host=hadoop16:22000):(Total: 12s192ms, non-child: 669.928us, % non-child: 0.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/60.02 MB 2:1/130.67 MB 5:1/109.50 MB 6:2/139.50 MB 7:2/278.81 MB 8:1/142.73 MB 9:1/100.25 MB
MemoryUsage(500.0ms): 39.40 MB, 225.73 MB, 316.68 MB, 622.29 MB, 818.53 MB, 718.99 MB, 789.95 MB, 659.31 MB, 418.69 MB, 263.30 MB, 216.26 MB, 105.38 MB, 97.49 MB, 824.00 KB, 25.81 MB, 50.85 MB, 54.41 MB, 69.33 MB, 74.11 MB, 104.12 MB, 116.12 MB, 108.07 MB, 108.12 MB, 84.47 MB
ThreadUsage(500.0ms): 10, 10, 10, 10, 10, 9, 8, 8, 5, 4, 4, 3, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2
- AverageThreadTokens: 4.83
- PeakMemoryUsage: 824.00 MB (864023752)
- PerHostPeakMemUsage: 824.00 MB (864023752)
- PrepareTime: 246.900us
- RowsProduced: 0 (0)
- TotalCpuTime: 46s979ms
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 105ns
- TotalStorageWaitTime: 25s927ms
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 22.539us, non-child: 22.539us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 12s191ms, non-child: 12s191ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/60.02 MB 2:1/130.67 MB 5:1/109.50 MB 6:2/139.50 MB 7:2/278.81 MB 8:1/142.73 MB 9:1/100.25 MB
Hdfs Read Thread Concurrency Bucket: 0:16.67% 1:33.33% 2:16.67% 3:0% 4:0% 5:8.333% 6:4.167% 7:0% 8:0% 9:0% 10:4.167% 11:4.167% 12:0% 13:0% 14:0% 15:12.5%
File Formats: PARQUET/NONE:495
ExecOption: Codegen enabled: 0 out of 9
BytesRead(500.0ms): 1.02 MB, 24.16 MB, 140.02 MB, 435.43 MB, 622.26 MB, 669.65 MB, 850.75 MB, 861.95 MB, 861.95 MB, 861.95 MB, 861.95 MB, 861.95 MB, 861.95 MB, 861.95 MB, 862.05 MB, 879.00 MB, 897.47 MB, 921.78 MB, 925.59 MB, 942.84 MB, 945.61 MB, 945.61 MB, 962.29 MB, 962.29 MB
- AverageHdfsReadThreadConcurrency: 6.00
- AverageScannerThreadConcurrency: 3.83
- BytesRead: 962.29 MB (1009031071)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 962.29 MB (1009031071)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 962.29 MB (1009031071)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 7 (7)
- NumScannerThreadsStarted: 9 (9)
- PeakMemoryUsage: 823.99 MB (864011464)
- PerReadThreadRawHdfsThroughput: 12.76 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 12.99M (12991632)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 9 (9)
- ScannerThreadsInvoluntaryContextSwitches: 310 (310)
- ScannerThreadsTotalWallClockTime: 46s979ms
- MaterializeTupleTime(*): 20s641ms
- ScannerThreadsSysTime: 52.1ms
- ScannerThreadsUserTime: 20s641ms
- ScannerThreadsVoluntaryContextSwitches: 364 (364)
- TotalRawHdfsReadTime(*): 1m15s
- TotalReadThroughput: 80.19 MB/sec
Instance 704b7e71011e9573:13c1490afe0019b1 (host=hadoop13:22000):(Total: 12s180ms, non-child: 723.69us, % non-child: 0.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/142.03 MB 3:1/66.55 MB 4:1/40.60 MB 5:1/129.46 MB 6:1/141.91 MB 8:1/51.14 MB 10:2/120.83 MB
MemoryUsage(500.0ms): 5.37 MB, 70.90 MB, 123.44 MB, 242.37 MB, 308.20 MB, 334.88 MB, 398.32 MB, 436.50 MB, 415.32 MB, 435.27 MB, 374.34 MB, 342.00 MB, 281.18 MB, 233.90 MB, 79.53 MB, 81.56 MB, 97.57 MB, 173.57 MB, 173.57 MB, 173.57 MB, 157.57 MB, 133.92 MB, 110.38 MB, 102.44 MB
ThreadUsage(500.0ms): 9, 9, 9, 9, 9, 8, 7, 7, 6, 5, 4, 4, 4, 4, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2
- AverageThreadTokens: 4.75
- PeakMemoryUsage: 488.43 MB (512155152)
- PerHostPeakMemUsage: 488.44 MB (512163344)
- PrepareTime: 226.501us
- RowsProduced: 0 (0)
- TotalCpuTime: 44s694ms
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 93ns
- TotalStorageWaitTime: 25s790ms
DataStreamSender (dst_id=1):(Total: 39.193us, non-child: 39.193us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 12s180ms, non-child: 12s180ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/142.03 MB 3:1/66.55 MB 4:1/40.60 MB 5:1/129.46 MB 6:1/141.91 MB 8:1/51.14 MB 10:2/120.83 MB
Hdfs Read Thread Concurrency Bucket: 0:50% 1:4.167% 2:4.167% 3:4.167% 4:4.167% 5:0% 6:4.167% 7:0% 8:4.167% 9:4.167% 10:0% 11:12.5% 12:4.167% 13:0% 14:0% 15:4.167%
File Formats: PARQUET/NONE:440
ExecOption: Codegen enabled: 0 out of 8
BytesRead(500.0ms): 0, 264.85 KB, 58.54 MB, 98.90 MB, 202.79 MB, 252.33 MB, 398.25 MB, 437.91 MB, 474.12 MB, 567.35 MB, 595.95 MB, 610.21 MB, 618.21 MB, 618.21 MB, 619.59 MB, 620.86 MB, 631.50 MB, 693.25 MB, 693.25 MB, 693.25 MB, 693.25 MB, 693.25 MB, 693.25 MB, 693.25 MB
- AverageHdfsReadThreadConcurrency: 4.08
- AverageScannerThreadConcurrency: 3.75
- BytesRead: 693.25 MB (726928716)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 693.25 MB (726928716)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 693.25 MB (726928716)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 7 (7)
- NumScannerThreadsStarted: 8 (8)
- PeakMemoryUsage: 488.42 MB (512142864)
- PerReadThreadRawHdfsThroughput: 13.61 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 9.59M (9588868)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 8 (8)
- ScannerThreadsInvoluntaryContextSwitches: 400 (400)
- ScannerThreadsTotalWallClockTime: 44s694ms
- MaterializeTupleTime(*): 18s155ms
- ScannerThreadsSysTime: 84.2ms
- ScannerThreadsUserTime: 18s181ms
- ScannerThreadsVoluntaryContextSwitches: 370 (370)
- TotalRawHdfsReadTime(*): 50s935ms
- TotalReadThroughput: 57.77 MB/sec
Instance 704b7e71011e9573:13c1490afe0019b7 (host=hadoop31:22000):(Total: 11s708ms, non-child: 1.129ms, % non-child: 0.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/143.38 MB 2:1/87.61 MB 3:1/136.41 MB 4:2/142.93 MB 5:1/135.42 MB 6:2/270.97 MB 10:1/42.87 MB
MemoryUsage(500.0ms): 59.18 MB, 165.78 MB, 234.78 MB, 496.32 MB, 547.68 MB, 650.17 MB, 661.89 MB, 608.34 MB, 470.59 MB, 508.51 MB, 511.13 MB, 487.35 MB, 386.28 MB, 332.42 MB, 319.51 MB, 208.10 MB, 134.95 MB, 134.95 MB, 136.72 MB, 138.72 MB, 151.40 MB, 83.73 MB, 67.73 MB
ThreadUsage(500.0ms): 10, 10, 10, 10, 10, 10, 10, 10, 8, 7, 6, 6, 5, 5, 5, 4, 3, 3, 3, 3, 3, 2, 2
- AverageThreadTokens: 6.30
- PeakMemoryUsage: 678.14 MB (711084888)
- PerHostPeakMemUsage: 678.14 MB (711084888)
- PrepareTime: 340.659us
- RowsProduced: 0 (0)
- TotalCpuTime: 1m1s
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 89ns
- TotalStorageWaitTime: 38s171ms
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 22.966us, non-child: 22.966us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 11s707ms, non-child: 11s707ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/143.38 MB 2:1/87.61 MB 3:1/136.41 MB 4:2/142.93 MB 5:1/135.42 MB 6:2/270.97 MB 10:1/42.87 MB
Hdfs Read Thread Concurrency Bucket: 0:43.48% 1:8.696% 2:0% 3:8.696% 4:0% 5:0% 6:4.348% 7:0% 8:4.348% 9:0% 10:4.348% 11:8.696% 12:8.696% 13:0% 14:0% 15:8.696%
File Formats: PARQUET/NONE:495
ExecOption: Codegen enabled: 0 out of 9
BytesRead(500.0ms): 200.00 KB, 26.78 MB, 112.46 MB, 394.03 MB, 433.66 MB, 529.39 MB, 531.84 MB, 560.95 MB, 626.03 MB, 786.68 MB, 876.56 MB, 876.56 MB, 883.90 MB, 898.75 MB, 919.85 MB, 928.44 MB, 929.19 MB, 929.19 MB, 930.56 MB, 932.39 MB, 960.42 MB, 960.42 MB, 960.42 MB
- AverageHdfsReadThreadConcurrency: 5.35
- AverageScannerThreadConcurrency: 5.30
- BytesRead: 960.42 MB (1007073793)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 960.42 MB (1007073793)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 960.42 MB (1007073793)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 7 (7)
- NumScannerThreadsStarted: 9 (9)
- PeakMemoryUsage: 678.13 MB (711072600)
- PerReadThreadRawHdfsThroughput: 15.36 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 12.95M (12945014)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 9 (9)
- ScannerThreadsInvoluntaryContextSwitches: 103 (103)
- ScannerThreadsTotalWallClockTime: 1m1s
- MaterializeTupleTime(*): 22s289ms
- ScannerThreadsSysTime: 12.0ms
- ScannerThreadsUserTime: 22s333ms
- ScannerThreadsVoluntaryContextSwitches: 470 (470)
- TotalRawHdfsReadTime(*): 1m2s
- TotalReadThroughput: 83.51 MB/sec
Instance 704b7e71011e9573:13c1490afe0019b5 (host=hadoop11:22000):(Total: 11s065ms, non-child: 1.812ms, % non-child: 0.02%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:1/142.62 MB 5:1/57.63 MB 8:1/140.83 MB 9:1/133.38 MB 10:1/145.14 MB
MemoryUsage(500.0ms): 53.90 MB, 63.50 MB, 103.03 MB, 135.89 MB, 63.65 MB, 132.26 MB, 187.54 MB, 201.00 MB, 212.13 MB, 277.55 MB, 440.50 MB, 464.58 MB, 452.99 MB, 471.39 MB, 432.62 MB, 308.29 MB, 191.71 MB, 211.78 MB, 142.19 MB, 142.19 MB, 94.76 MB, 71.07 MB
ThreadUsage(500.0ms): 6, 6, 6, 6, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 3, 3, 2, 2, 2, 2
- AverageThreadTokens: 4.41
- PeakMemoryUsage: 480.09 MB (503408440)
- PerHostPeakMemUsage: 480.09 MB (503408440)
- PrepareTime: 229.161us
- RowsProduced: 0 (0)
- TotalCpuTime: 38s369ms
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 84ns
- TotalStorageWaitTime: 25s819ms
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 17.220us, non-child: 17.220us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 11s063ms, non-child: 11s063ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:1/142.62 MB 5:1/57.63 MB 8:1/140.83 MB 9:1/133.38 MB 10:1/145.14 MB
Hdfs Read Thread Concurrency Bucket: 0:36.36% 1:0% 2:9.091% 3:4.545% 4:9.091% 5:9.091% 6:22.73% 7:0% 8:0% 9:0% 10:4.545% 11:4.545% 12:0% 13:0% 14:0% 15:0%
File Formats: PARQUET/NONE:275
ExecOption: Codegen enabled: 0 out of 5
BytesRead(500.0ms): 12.93 MB, 40.32 MB, 65.82 MB, 75.22 MB, 91.29 MB, 134.78 MB, 174.10 MB, 200.26 MB, 209.75 MB, 256.35 MB, 405.67 MB, 445.45 MB, 461.35 MB, 520.52 MB, 552.96 MB, 555.47 MB, 555.47 MB, 578.09 MB, 607.11 MB, 620.05 MB, 620.05 MB, 620.05 MB
- AverageHdfsReadThreadConcurrency: 3.45
- AverageScannerThreadConcurrency: 3.41
- BytesRead: 620.05 MB (650173562)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 562.34 MB (589651218)
- BytesReadRemoteUnexpected: 57.72 MB (60522344)
- BytesReadShortCircuit: 562.34 MB (589651218)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 5 (5)
- NumScannerThreadsStarted: 5 (5)
- PeakMemoryUsage: 480.08 MB (503396152)
- PerReadThreadRawHdfsThroughput: 16.21 MB/sec
- RemoteScanRanges: 56 (56)
- RowsRead: 8.11M (8114661)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 5 (5)
- ScannerThreadsInvoluntaryContextSwitches: 388 (388)
- ScannerThreadsTotalWallClockTime: 38s369ms
- MaterializeTupleTime(*): 11s918ms
- ScannerThreadsSysTime: 52.2ms
- ScannerThreadsUserTime: 11s896ms
- ScannerThreadsVoluntaryContextSwitches: 290 (290)
- TotalRawHdfsReadTime(*): 38s248ms
- TotalReadThroughput: 56.37 MB/sec
Instance 704b7e71011e9573:13c1490afe0019ac (host=hadoop32:22000):(Total: 9s419ms, non-child: 656.346us, % non-child: 0.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/94.38 MB 1:1/124.58 MB 2:1/48.36 MB 3:1/109.42 MB 4:1/139.54 MB 7:1/80.43 MB 10:2/170.99 MB
MemoryUsage(500.0ms): 146.18 MB, 244.27 MB, 345.94 MB, 476.02 MB, 486.60 MB, 541.50 MB, 453.52 MB, 404.95 MB, 354.03 MB, 227.85 MB, 218.58 MB, 200.10 MB, 202.36 MB, 186.37 MB, 233.96 MB, 194.50 MB, 129.12 MB, 89.47 MB, 65.69 MB
ThreadUsage(500.0ms): 9, 9, 9, 9, 9, 9, 8, 6, 6, 4, 4, 3, 3, 3, 3, 3, 2, 2, 2
- AverageThreadTokens: 5.42
- PeakMemoryUsage: 557.23 MB (584294024)
- PerHostPeakMemUsage: 557.23 MB (584294024)
- PrepareTime: 220.25us
- RowsProduced: 0 (0)
- TotalCpuTime: 41s764ms
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 113ns
- TotalStorageWaitTime: 22s188ms
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 15.666us, non-child: 15.666us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 9s418ms, non-child: 9s418ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/94.38 MB 1:1/124.58 MB 2:1/48.36 MB 3:1/109.42 MB 4:1/139.54 MB 7:1/80.43 MB 10:2/170.99 MB
Hdfs Read Thread Concurrency Bucket: 0:42.11% 1:5.263% 2:0% 3:0% 4:10.53% 5:0% 6:5.263% 7:5.263% 8:0% 9:5.263% 10:0% 11:5.263% 12:5.263% 13:5.263% 14:0% 15:10.53%
File Formats: PARQUET/NONE:440
ExecOption: Codegen enabled: 0 out of 8
BytesRead(500.0ms): 10.14 MB, 125.25 MB, 269.67 MB, 363.27 MB, 394.25 MB, 443.50 MB, 512.60 MB, 555.10 MB, 589.38 MB, 642.96 MB, 673.56 MB, 717.12 MB, 721.19 MB, 721.19 MB, 768.43 MB, 768.43 MB, 768.43 MB, 768.43 MB, 768.43 MB
- AverageHdfsReadThreadConcurrency: 5.47
- AverageScannerThreadConcurrency: 4.42
- BytesRead: 768.43 MB (805753576)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 768.43 MB (805753576)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 768.43 MB (805753576)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 7 (7)
- NumScannerThreadsStarted: 8 (8)
- PeakMemoryUsage: 557.21 MB (584281736)
- PerReadThreadRawHdfsThroughput: 15.91 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 10.56M (10557343)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 8 (8)
- ScannerThreadsInvoluntaryContextSwitches: 297 (297)
- ScannerThreadsTotalWallClockTime: 41s764ms
- MaterializeTupleTime(*): 18s484ms
- ScannerThreadsSysTime: 92.4ms
- ScannerThreadsUserTime: 18s433ms
- ScannerThreadsVoluntaryContextSwitches: 420 (420)
- TotalRawHdfsReadTime(*): 48s288ms
- TotalReadThroughput: 80.89 MB/sec
Instance 704b7e71011e9573:13c1490afe0019ae (host=hadoop6:22000):(Total: 9s060ms, non-child: 742.383us, % non-child: 0.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/51.63 MB 11:3/265.18 MB 2:1/136.60 MB 4:2/277.44 MB 5:2/206.30 MB 6:1/83.33 MB 8:1/119.78 MB 9:1/137.37 MB 10:1/141.36 MB
MemoryUsage(500.0ms): 183.27 MB, 371.99 MB, 452.27 MB, 701.64 MB, 905.56 MB, 939.74 MB, 897.45 MB, 1.06 GB, 1.04 GB, 1.07 GB, 1.03 GB, 986.96 MB, 1.02 GB, 903.46 MB, 740.03 MB, 387.88 MB, 168.08 MB, 92.30 MB
ThreadUsage(500.0ms): 14, 14, 14, 14, 13, 13, 12, 12, 12, 12, 12, 12, 12, 10, 9, 5, 3, 2
- AverageThreadTokens: 10.83
- PeakMemoryUsage: 1.07 GB (1151744936)
- PerHostPeakMemUsage: 1.07 GB (1151744936)
- PrepareTime: 254.393us
- RowsProduced: 0 (0)
- TotalCpuTime: 1m29s
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 92ns
- TotalStorageWaitTime: 41s997ms
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 16.687us, non-child: 16.687us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 9s059ms, non-child: 9s059ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/51.63 MB 11:3/265.18 MB 2:1/136.60 MB 4:2/277.44 MB 5:2/206.30 MB 6:1/83.33 MB 8:1/119.78 MB 9:1/137.37 MB 10:1/141.36 MB
Hdfs Read Thread Concurrency Bucket: 0:22.22% 1:0% 2:5.556% 3:0% 4:5.556% 5:5.556% 6:0% 7:5.556% 8:5.556% 9:5.556% 10:0% 11:5.556% 12:0% 13:11.11% 14:0% 15:27.78%
File Formats: PARQUET/NONE:715
ExecOption: Codegen enabled: 0 out of 13
BytesRead(500.0ms): 9.83 MB, 128.89 MB, 279.75 MB, 530.25 MB, 767.30 MB, 786.57 MB, 811.06 MB, 1.01 GB, 1.03 GB, 1.07 GB, 1.08 GB, 1.12 GB, 1.25 GB, 1.38 GB, 1.39 GB, 1.39 GB, 1.39 GB, 1.39 GB
- AverageHdfsReadThreadConcurrency: 11.11
- AverageScannerThreadConcurrency: 9.83
- BytesRead: 1.39 GB (1489169920)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 1.39 GB (1489169920)
- BytesReadRemoteUnexpected: 0
- BytesReadShortCircuit: 1.39 GB (1489169920)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 9 (9)
- NumScannerThreadsStarted: 13 (13)
- PeakMemoryUsage: 1.07 GB (1151732648)
- PerReadThreadRawHdfsThroughput: 14.46 MB/sec
- RemoteScanRanges: 0 (0)
- RowsRead: 18.99M (18994692)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 13 (13)
- ScannerThreadsInvoluntaryContextSwitches: 616 (616)
- ScannerThreadsTotalWallClockTime: 1m29s
- MaterializeTupleTime(*): 46s366ms
- ScannerThreadsSysTime: 88.3ms
- ScannerThreadsUserTime: 46s330ms
- ScannerThreadsVoluntaryContextSwitches: 455 (455)
- TotalRawHdfsReadTime(*): 1m38s
- TotalReadThroughput: 157.80 MB/sec
Instance 704b7e71011e9573:13c1490afe0019b0 (host=hadoop14:22000):(Total: 7s224ms, non-child: 683.613us, % non-child: 0.01%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:1/126.79 MB 4:1/135.91 MB 5:1/135.04 MB 6:1/88.99 MB 8:1/47.71 MB 9:2/224.17 MB 10:1/130.33 MB
MemoryUsage(500.0ms): 65.26 MB, 163.02 MB, 289.86 MB, 432.92 MB, 480.47 MB, 655.31 MB, 758.47 MB, 729.98 MB, 725.83 MB, 602.17 MB, 419.64 MB, 420.24 MB, 289.01 MB, 170.49 MB
ThreadUsage(500.0ms): 9, 9, 9, 9, 9, 9, 9, 9, 9, 7, 5, 5, 4, 3
- AverageThreadTokens: 7.50
- PeakMemoryUsage: 774.47 MB (812087880)
- PerHostPeakMemUsage: 774.47 MB (812087880)
- PrepareTime: 186.5us
- RowsProduced: 0 (0)
- TotalCpuTime: 46s197ms
- TotalNetworkReceiveTime: 0ns
- TotalNetworkSendTime: 91ns
- TotalStorageWaitTime: 26s701ms
BlockMgr:
- BlockWritesOutstanding: 0 (0)
- BlocksCreated: 0 (0)
- BlocksRecycled: 0 (0)
- BufferedPins: 0 (0)
- BytesWritten: 0
- MaxBlockSize: 8.00 MB (8388608)
- MemoryLimit: 161.34 GB (173237125120)
- PeakMemoryUsage: 0
- TotalBufferWaitTime: 0ns
- TotalEncryptionTime: 0ns
- TotalIntegrityCheckTime: 0ns
- TotalReadBlockTime: 0ns
DataStreamSender (dst_id=1):(Total: 13.110us, non-child: 13.110us, % non-child: 100.00%)
- BytesSent: 0
- NetworkThroughput(*): 0.00 /sec
- OverallThroughput: 0.00 /sec
- PeakMemoryUsage: 4.00 KB (4096)
- SerializeBatchTime: 0ns
- ThriftTransmitTime(*): 0ns
- UncompressedRowBatchSize: 0
HDFS_SCAN_NODE (id=0):(Total: 7s224ms, non-child: 7s224ms, % non-child: 100.00%)
Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:1/126.79 MB 4:1/135.91 MB 5:1/135.04 MB 6:1/88.99 MB 8:1/47.71 MB 9:2/224.17 MB 10:1/130.33 MB
Hdfs Read Thread Concurrency Bucket: 0:14.29% 1:7.143% 2:0% 3:0% 4:0% 5:7.143% 6:14.29% 7:0% 8:7.143% 9:0% 10:7.143% 11:7.143% 12:7.143% 13:0% 14:0% 15:28.57%
File Formats: PARQUET/NONE:440
ExecOption: Codegen enabled: 0 out of 8
BytesRead(500.0ms): 8.20 MB, 50.79 MB, 96.72 MB, 238.82 MB, 356.45 MB, 529.66 MB, 624.64 MB, 641.76 MB, 720.92 MB, 772.87 MB, 784.49 MB, 880.29 MB, 889.67 MB, 889.67 MB
- AverageHdfsReadThreadConcurrency: 10.43
- AverageScannerThreadConcurrency: 6.50
- BytesRead: 889.67 MB (932886343)
- BytesReadDataNodeCache: 0
- BytesReadLocal: 631.88 MB (662572320)
- BytesReadRemoteUnexpected: 257.79 MB (270314023)
- BytesReadShortCircuit: 631.88 MB (662572320)
- DecompressionTime: 0ns
- MaxCompressedTextFileLength: 0
- NumColumns: 55 (55)
- NumDisksAccessed: 7 (7)
- NumScannerThreadsStarted: 8 (8)
- PeakMemoryUsage: 774.46 MB (812075592)
- PerReadThreadRawHdfsThroughput: 12.35 MB/sec
- RemoteScanRanges: 112 (112)
- RowsRead: 11.86M (11860260)
- RowsReturned: 0 (0)
- RowsReturnedRate: 0
- ScanRangesComplete: 8 (8)
- ScannerThreadsInvoluntaryContextSwitches: 401 (401)
- ScannerThreadsTotalWallClockTime: 46s197ms
- MaterializeTupleTime(*): 18s084ms
- ScannerThreadsSysTime: 100.4ms
- ScannerThreadsUserTime: 18s065ms
- ScannerThreadsVoluntaryContextSwitches: 439 (439)
- TotalRawHdfsReadTime(*): 1m12s
- TotalReadThroughput: 127.10 MB/sec

[#5]

The symptom disappeared after restarted the specified node.
So looks like that the node was too busy to provide extra scanner thread, i.e there is a limit of maximum thread number, which most of them has been occupied by other tasks. is it right? If so, how to adjust the limit?

[#6 · Accepted solution]

Impala limits the number of threads executing the query plan by design. Impala dynamically increases the number of scanner threads provided there are CPU and memory resources available - in this case it seems like there weren't CPU resource available.

If the machine is already busy adding more threads can actually decrease query throughput.

[#7]

Thanks for your reply.

For the word you said "in this case it seems like there weren't CPU resource available", how to find the cause is maybe the insufficient CPU resource?  or how can it find from profile?

[#8]

There's no direct way to find out from the profile unfortunately. If you have a live system you can look at the /threadz page on the impala debug web page (port 25000 on each Impala daemon by default) to see how many hdfs-scan-node threads are running.