Member since
04-25-2014
33
Posts
0
Kudos Received
0
Solutions
11-18-2016
11:19 AM
Another question: what is the specifically certified Cloudera partner appliances.
... View more
11-17-2016
07:41 AM
any ideas?
... View more
11-15-2016
01:33 PM
It looks like that CDH5.9 has support multi-homed from the URL "https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html". But it looks like that CDH does not support multi-homed from the URL "http://www.cloudera.com/documentation/enterprise/5-8-x/topics/cdh_ig_req_supported_versions.html#xd_583c10bfdbd326ba--43d5fd93-1410993f8c2--7e14". So i wonder which is true?
... View more
Labels:
- Labels:
-
HDFS
11-15-2016
06:23 AM
not mean upgrade. If do not configure high availabe for Cloudera manager, once the cloudera manager host crash, maybe can re-install it on the other machine. If so, also need to re-install CDH?
... View more
11-13-2016
06:39 AM
1. if re-install Clouder Manager, whether need to re-install CDH or not? 2. for CDH integrated with Kerberos, once re-install Kerberos, whether need to re-install CDH or not?
... View more
Labels:
- Labels:
-
Cloudera Manager
-
Kerberos
11-13-2016
06:39 AM
if enable Navigator after CDH has run for long time, the old data can not be encrpted, and the new encrypted data also can not be palced into the old diretory. right?
... View more
Labels:
- Labels:
-
Cloudera Navigator
05-02-2016
08:15 PM
Thanks for your reply. For the word you said "in this case it seems like there weren't CPU resource available", how to find the cause is maybe the insufficient CPU resource? or how can it find from profile?
... View more
04-29-2016
12:07 AM
The symptom disappeared after restarted the specified node. So looks like that the node was too busy to provide extra scanner thread, i.e there is a limit of maximum thread number, which most of them has been occupied by other tasks. is it right? If so, how to adjust the limit?
... View more
04-26-2016
07:39 PM
The second fragment is as follows: HDFS_SCAN_NODE (id=0):(Total: 15s642ms, non-child: 15s642ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/129.56 MB 0:3/382.82 MB 3:3/252.99 MB 5:2/184.55 MB 7:3/310.95 MB 8:2/205.49 MB Hdfs Read Thread Concurrency Bucket: 0:22.58% 1:3.226% 2:3.226% 3:6.452% 4:3.226% 5:6.452% 6:6.452% 7:3.226% 8:6.452% 9:9.677% 10:9.677% 11:0% 12:3.226% 13:0% 14:0% 15:16.13% File Formats: PARQUET/NONE:770 ExecOption: Codegen enabled: 0 out of 14 BytesRead(500.0ms): 322.81 KB, 64.60 MB, 193.73 MB, 295.27 MB, 337.90 MB, 404.14 MB, 617.34 MB, 650.05 MB, 666.97 MB, 680.94 MB, 714.84 MB, 903.24 MB, 945.56 MB, 997.88 MB, 1.00 GB, 1.02 GB, 1.05 GB, 1.11 GB, 1.15 GB, 1.18 GB, 1.18 GB, 1.20 GB, 1.21 GB, 1.26 GB, 1.36 GB, 1.36 GB, 1.43 GB, 1.43 GB, 1.43 GB, 1.43 GB, 1.43 GB - AverageHdfsReadThreadConcurrency: 6.81 - AverageScannerThreadConcurrency: 8.23 - BytesRead: 1.43 GB (1538913826) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.43 GB (1538913826) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 1.43 GB (1538913826) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 6 (6) - NumScannerThreadsStarted: 14 (14) - PeakMemoryUsage: 652.38 MB (684071072) - PerReadThreadRawHdfsThroughput: 13.67 MB/sec - RemoteScanRanges: 0 (0) - RowsRead: 19.70M (19702157) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 14 (14) - ScannerThreadsInvoluntaryContextSwitches: 1.68K (1681) - ScannerThreadsTotalWallClockTime: 2m11s - MaterializeTupleTime(*): 33s178ms - ScannerThreadsSysTime: 124.4ms - ScannerThreadsUserTime: 33s098ms - ScannerThreadsVoluntaryContextSwitches: 577 (577) - TotalRawHdfsReadTime(*): 1m47s - TotalReadThroughput: 94.69 MB/sec Instance 704b7e71011e9573:13c1490afe0019bb (host=hadoop5:22000):(Total: 13s683ms, non-child: 866.247us, % non-child: 0.01%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/61.51 MB 1:1/136.60 MB 2:1/130.20 MB 5:1/139.10 MB 7:1/123.58 MB 8:1/139.49 MB 10:1/104.81 MB MemoryUsage(500.0ms): 2.36 MB, 134.74 MB, 189.52 MB, 281.12 MB, 356.36 MB, 324.38 MB, 244.15 MB, 142.80 MB, 181.76 MB, 154.80 MB, 285.80 MB, 381.28 MB, 379.78 MB, 419.75 MB, 403.90 MB, 304.33 MB, 338.74 MB, 314.96 MB, 291.05 MB, 296.32 MB, 290.05 MB, 274.34 MB, 250.56 MB, 242.62 MB, 210.87 MB, 202.93 MB, 86.76 MB, 78.86 MB ThreadUsage(500.0ms): 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 2, 2 - AverageThreadTokens: 3.64 - PeakMemoryUsage: 459.63 MB (481955896) - PerHostPeakMemUsage: 459.63 MB (481955896) - PrepareTime: 212.743us - RowsProduced: 0 (0) - TotalCpuTime: 37s570ms - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 89ns - TotalStorageWaitTime: 12s291ms BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 14.942us, non-child: 14.942us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 13s682ms, non-child: 13s682ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/61.51 MB 1:1/136.60 MB 2:1/130.20 MB 5:1/139.10 MB 7:1/123.58 MB 8:1/139.49 MB 10:1/104.81 MB Hdfs Read Thread Concurrency Bucket: 0:46.43% 1:3.571% 2:0% 3:0% 4:0% 5:0% 6:32.14% 7:0% 8:0% 9:0% 10:3.571% 11:3.571% 12:3.571% 13:0% 14:0% 15:7.143% File Formats: PARQUET/NONE:385 ExecOption: Codegen enabled: 0 out of 7 BytesRead(500.0ms): 0, 9.58 MB, 96.03 MB, 218.62 MB, 290.17 MB, 290.17 MB, 290.27 MB, 299.51 MB, 311.65 MB, 365.35 MB, 476.58 MB, 588.33 MB, 607.07 MB, 705.64 MB, 705.64 MB, 705.64 MB, 706.91 MB, 706.91 MB, 714.91 MB, 750.02 MB, 835.93 MB, 835.93 MB, 835.93 MB, 835.93 MB, 835.93 MB, 835.93 MB, 835.93 MB, 835.93 MB - AverageHdfsReadThreadConcurrency: 4.36 - AverageScannerThreadConcurrency: 2.64 - BytesRead: 835.93 MB (876532952) - BytesReadDataNodeCache: 0 - BytesReadLocal: 835.93 MB (876532952) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 835.93 MB (876532952) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 7 (7) - NumScannerThreadsStarted: 3 (3) - PeakMemoryUsage: 459.62 MB (481943608) - PerReadThreadRawHdfsThroughput: 13.74 MB/sec - RemoteScanRanges: 0 (0) - RowsRead: 11.04M (11040327) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 7 (7) - ScannerThreadsInvoluntaryContextSwitches: 653 (653) - ScannerThreadsTotalWallClockTime: 36s174ms - MaterializeTupleTime(*): 23s772ms - ScannerThreadsSysTime: 28.1ms - ScannerThreadsUserTime: 23s797ms - ScannerThreadsVoluntaryContextSwitches: 151 (151) - TotalRawHdfsReadTime(*): 1m - TotalReadThroughput: 59.71 MB/sec Instance 704b7e71011e9573:13c1490afe0019b6 (host=hadoop8:22000):(Total: 12s579ms, non-child: 132.751ms, % non-child: 1.06%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/139.13 MB 2:2/273.38 MB 3:1/125.03 MB 4:2/127.04 MB 6:2/257.95 MB 7:1/136.70 MB MemoryUsage(500.0ms): 3.80 MB, 80.67 MB, 106.98 MB, 301.40 MB, 356.72 MB, 351.27 MB, 477.72 MB, 548.32 MB, 391.00 MB, 367.36 MB, 316.51 MB, 269.24 MB, 142.44 MB, 204.52 MB, 235.67 MB, 317.67 MB, 349.00 MB, 355.88 MB, 487.87 MB, 511.48 MB, 440.17 MB, 234.84 MB, 211.11 MB, 83.77 MB, 75.86 MB ThreadUsage(500.0ms): 10, 10, 10, 10, 10, 9, 9, 9, 7, 7, 7, 7, 5, 5, 5, 5, 5, 5, 5, 5, 5, 3, 3, 2, 2 - AverageThreadTokens: 6.40 - PeakMemoryUsage: 556.99 MB (584048344) - PerHostPeakMemUsage: 556.99 MB (584048344) - PrepareTime: 1.455ms - RowsProduced: 0 (0) - TotalCpuTime: 1m7s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 235ns - TotalStorageWaitTime: 42s377ms BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 19.63us, non-child: 19.63us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 12s446ms, non-child: 12s446ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/139.13 MB 2:2/273.38 MB 3:1/125.03 MB 4:2/127.04 MB 6:2/257.95 MB 7:1/136.70 MB Hdfs Read Thread Concurrency Bucket: 0:28% 1:0% 2:4% 3:0% 4:4% 5:8% 6:4% 7:12% 8:12% 9:0% 10:8% 11:8% 12:8% 13:0% 14:0% 15:4% File Formats: PARQUET/NONE:495 ExecOption: Codegen enabled: 0 out of 9 BytesRead(500.0ms): 0, 14.32 MB, 50.98 MB, 171.48 MB, 229.69 MB, 315.23 MB, 449.35 MB, 559.26 MB, 583.26 MB, 586.25 MB, 612.73 MB, 615.09 MB, 644.34 MB, 667.70 MB, 685.74 MB, 746.37 MB, 779.65 MB, 831.11 MB, 999.53 MB, 1.04 GB, 1.04 GB, 1.04 GB, 1.04 GB, 1.04 GB, 1.04 GB - AverageHdfsReadThreadConcurrency: 5.92 - AverageScannerThreadConcurrency: 5.40 - BytesRead: 1.04 GB (1111533925) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.04 GB (1111533925) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 1.04 GB (1111533925) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 6 (6) - NumScannerThreadsStarted: 9 (9) - PeakMemoryUsage: 556.98 MB (584036056) - PerReadThreadRawHdfsThroughput: 14.17 MB/sec - RemoteScanRanges: 0 (0) - RowsRead: 14.04M (14038683) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 9 (9) - ScannerThreadsInvoluntaryContextSwitches: 1.00K (1004) - ScannerThreadsTotalWallClockTime: 1m7s - MaterializeTupleTime(*): 20s872ms - ScannerThreadsSysTime: 84.1ms - ScannerThreadsUserTime: 20s857ms - ScannerThreadsVoluntaryContextSwitches: 412 (412) - TotalRawHdfsReadTime(*): 1m14s - TotalReadThroughput: 84.80 MB/sec Instance 704b7e71011e9573:13c1490afe0019b4 (host=hadoop30:22000):(Total: 12s350ms, non-child: 981.652us, % non-child: 0.01%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/111.69 MB 2:2/131.88 MB 3:1/106.78 MB 4:2/227.32 MB 5:2/241.62 MB 7:4/432.52 MB 8:1/50.53 MB 10:1/144.42 MB MemoryUsage(500.0ms): 62.09 MB, 230.58 MB, 372.08 MB, 736.11 MB, 901.90 MB, 878.44 MB, 878.81 MB, 771.93 MB, 554.02 MB, 460.93 MB, 442.84 MB, 425.77 MB, 505.04 MB, 531.74 MB, 492.57 MB, 493.51 MB, 441.50 MB, 414.66 MB, 407.47 MB, 278.68 MB, 402.63 MB, 339.25 MB, 220.51 MB, 181.05 MB, 68.42 MB ThreadUsage(500.0ms): 15, 15, 15, 15, 15, 14, 14, 13, 11, 10, 9, 9, 8, 8, 7, 7, 7, 7, 7, 5, 4, 4, 3, 3, 2 - AverageThreadTokens: 9.08 - PeakMemoryUsage: 968.32 MB (1015356592) - PerHostPeakMemUsage: 968.32 MB (1015356592) - PrepareTime: 292.223us - RowsProduced: 0 (0) - TotalCpuTime: 1m42s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 120ns - TotalStorageWaitTime: 1m7s BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 40.134us, non-child: 40.134us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 12s349ms, non-child: 12s349ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/111.69 MB 2:2/131.88 MB 3:1/106.78 MB 4:2/227.32 MB 5:2/241.62 MB 7:4/432.52 MB 8:1/50.53 MB 10:1/144.42 MB Hdfs Read Thread Concurrency Bucket: 0:24% 1:4% 2:0% 3:8% 4:0% 5:12% 6:4% 7:4% 8:4% 9:12% 10:0% 11:4% 12:0% 13:0% 14:0% 15:24% File Formats: PARQUET/NONE:770 ExecOption: Codegen enabled: 0 out of 14 BytesRead(500.0ms): 400.00 KB, 65.19 MB, 227.21 MB, 501.10 MB, 660.19 MB, 765.94 MB, 877.98 MB, 877.98 MB, 879.00 MB, 883.91 MB, 942.57 MB, 959.97 MB, 1.08 GB, 1.13 GB, 1.16 GB, 1.19 GB, 1.20 GB, 1.20 GB, 1.20 GB, 1.25 GB, 1.41 GB, 1.41 GB, 1.41 GB, 1.41 GB, 1.41 GB - AverageHdfsReadThreadConcurrency: 7.96 - AverageScannerThreadConcurrency: 8.08 - BytesRead: 1.41 GB (1518383151) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.26 GB (1348087343) - BytesReadRemoteUnexpected: 162.41 MB (170295808) - BytesReadShortCircuit: 1.26 GB (1348087343) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 8 (8) - NumScannerThreadsStarted: 14 (14) - PeakMemoryUsage: 968.31 MB (1015344304) - PerReadThreadRawHdfsThroughput: 15.16 MB/sec - RemoteScanRanges: 112 (112) - RowsRead: 19.59M (19590941) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 14 (14) - ScannerThreadsInvoluntaryContextSwitches: 399 (399) - ScannerThreadsTotalWallClockTime: 1m42s - MaterializeTupleTime(*): 33s893ms - ScannerThreadsSysTime: 72.1ms - ScannerThreadsUserTime: 33s950ms - ScannerThreadsVoluntaryContextSwitches: 680 (680) - TotalRawHdfsReadTime(*): 1m35s - TotalReadThroughput: 115.84 MB/sec Instance 704b7e71011e9573:13c1490afe0019ad (host=hadoop16:22000):(Total: 12s192ms, non-child: 669.928us, % non-child: 0.01%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/60.02 MB 2:1/130.67 MB 5:1/109.50 MB 6:2/139.50 MB 7:2/278.81 MB 8:1/142.73 MB 9:1/100.25 MB MemoryUsage(500.0ms): 39.40 MB, 225.73 MB, 316.68 MB, 622.29 MB, 818.53 MB, 718.99 MB, 789.95 MB, 659.31 MB, 418.69 MB, 263.30 MB, 216.26 MB, 105.38 MB, 97.49 MB, 824.00 KB, 25.81 MB, 50.85 MB, 54.41 MB, 69.33 MB, 74.11 MB, 104.12 MB, 116.12 MB, 108.07 MB, 108.12 MB, 84.47 MB ThreadUsage(500.0ms): 10, 10, 10, 10, 10, 9, 8, 8, 5, 4, 4, 3, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 - AverageThreadTokens: 4.83 - PeakMemoryUsage: 824.00 MB (864023752) - PerHostPeakMemUsage: 824.00 MB (864023752) - PrepareTime: 246.900us - RowsProduced: 0 (0) - TotalCpuTime: 46s979ms - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 105ns - TotalStorageWaitTime: 25s927ms BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 22.539us, non-child: 22.539us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 12s191ms, non-child: 12s191ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/60.02 MB 2:1/130.67 MB 5:1/109.50 MB 6:2/139.50 MB 7:2/278.81 MB 8:1/142.73 MB 9:1/100.25 MB Hdfs Read Thread Concurrency Bucket: 0:16.67% 1:33.33% 2:16.67% 3:0% 4:0% 5:8.333% 6:4.167% 7:0% 8:0% 9:0% 10:4.167% 11:4.167% 12:0% 13:0% 14:0% 15:12.5% File Formats: PARQUET/NONE:495 ExecOption: Codegen enabled: 0 out of 9 BytesRead(500.0ms): 1.02 MB, 24.16 MB, 140.02 MB, 435.43 MB, 622.26 MB, 669.65 MB, 850.75 MB, 861.95 MB, 861.95 MB, 861.95 MB, 861.95 MB, 861.95 MB, 861.95 MB, 861.95 MB, 862.05 MB, 879.00 MB, 897.47 MB, 921.78 MB, 925.59 MB, 942.84 MB, 945.61 MB, 945.61 MB, 962.29 MB, 962.29 MB - AverageHdfsReadThreadConcurrency: 6.00 - AverageScannerThreadConcurrency: 3.83 - BytesRead: 962.29 MB (1009031071) - BytesReadDataNodeCache: 0 - BytesReadLocal: 962.29 MB (1009031071) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 962.29 MB (1009031071) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 7 (7) - NumScannerThreadsStarted: 9 (9) - PeakMemoryUsage: 823.99 MB (864011464) - PerReadThreadRawHdfsThroughput: 12.76 MB/sec - RemoteScanRanges: 0 (0) - RowsRead: 12.99M (12991632) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 9 (9) - ScannerThreadsInvoluntaryContextSwitches: 310 (310) - ScannerThreadsTotalWallClockTime: 46s979ms - MaterializeTupleTime(*): 20s641ms - ScannerThreadsSysTime: 52.1ms - ScannerThreadsUserTime: 20s641ms - ScannerThreadsVoluntaryContextSwitches: 364 (364) - TotalRawHdfsReadTime(*): 1m15s - TotalReadThroughput: 80.19 MB/sec Instance 704b7e71011e9573:13c1490afe0019b1 (host=hadoop13:22000):(Total: 12s180ms, non-child: 723.69us, % non-child: 0.01%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/142.03 MB 3:1/66.55 MB 4:1/40.60 MB 5:1/129.46 MB 6:1/141.91 MB 8:1/51.14 MB 10:2/120.83 MB MemoryUsage(500.0ms): 5.37 MB, 70.90 MB, 123.44 MB, 242.37 MB, 308.20 MB, 334.88 MB, 398.32 MB, 436.50 MB, 415.32 MB, 435.27 MB, 374.34 MB, 342.00 MB, 281.18 MB, 233.90 MB, 79.53 MB, 81.56 MB, 97.57 MB, 173.57 MB, 173.57 MB, 173.57 MB, 157.57 MB, 133.92 MB, 110.38 MB, 102.44 MB ThreadUsage(500.0ms): 9, 9, 9, 9, 9, 8, 7, 7, 6, 5, 4, 4, 4, 4, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 - AverageThreadTokens: 4.75 - PeakMemoryUsage: 488.43 MB (512155152) - PerHostPeakMemUsage: 488.44 MB (512163344) - PrepareTime: 226.501us - RowsProduced: 0 (0) - TotalCpuTime: 44s694ms - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 93ns - TotalStorageWaitTime: 25s790ms DataStreamSender (dst_id=1):(Total: 39.193us, non-child: 39.193us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 12s180ms, non-child: 12s180ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/142.03 MB 3:1/66.55 MB 4:1/40.60 MB 5:1/129.46 MB 6:1/141.91 MB 8:1/51.14 MB 10:2/120.83 MB Hdfs Read Thread Concurrency Bucket: 0:50% 1:4.167% 2:4.167% 3:4.167% 4:4.167% 5:0% 6:4.167% 7:0% 8:4.167% 9:4.167% 10:0% 11:12.5% 12:4.167% 13:0% 14:0% 15:4.167% File Formats: PARQUET/NONE:440 ExecOption: Codegen enabled: 0 out of 8 BytesRead(500.0ms): 0, 264.85 KB, 58.54 MB, 98.90 MB, 202.79 MB, 252.33 MB, 398.25 MB, 437.91 MB, 474.12 MB, 567.35 MB, 595.95 MB, 610.21 MB, 618.21 MB, 618.21 MB, 619.59 MB, 620.86 MB, 631.50 MB, 693.25 MB, 693.25 MB, 693.25 MB, 693.25 MB, 693.25 MB, 693.25 MB, 693.25 MB - AverageHdfsReadThreadConcurrency: 4.08 - AverageScannerThreadConcurrency: 3.75 - BytesRead: 693.25 MB (726928716) - BytesReadDataNodeCache: 0 - BytesReadLocal: 693.25 MB (726928716) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 693.25 MB (726928716) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 7 (7) - NumScannerThreadsStarted: 8 (8) - PeakMemoryUsage: 488.42 MB (512142864) - PerReadThreadRawHdfsThroughput: 13.61 MB/sec - RemoteScanRanges: 0 (0) - RowsRead: 9.59M (9588868) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 8 (8) - ScannerThreadsInvoluntaryContextSwitches: 400 (400) - ScannerThreadsTotalWallClockTime: 44s694ms - MaterializeTupleTime(*): 18s155ms - ScannerThreadsSysTime: 84.2ms - ScannerThreadsUserTime: 18s181ms - ScannerThreadsVoluntaryContextSwitches: 370 (370) - TotalRawHdfsReadTime(*): 50s935ms - TotalReadThroughput: 57.77 MB/sec Instance 704b7e71011e9573:13c1490afe0019b7 (host=hadoop31:22000):(Total: 11s708ms, non-child: 1.129ms, % non-child: 0.01%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/143.38 MB 2:1/87.61 MB 3:1/136.41 MB 4:2/142.93 MB 5:1/135.42 MB 6:2/270.97 MB 10:1/42.87 MB MemoryUsage(500.0ms): 59.18 MB, 165.78 MB, 234.78 MB, 496.32 MB, 547.68 MB, 650.17 MB, 661.89 MB, 608.34 MB, 470.59 MB, 508.51 MB, 511.13 MB, 487.35 MB, 386.28 MB, 332.42 MB, 319.51 MB, 208.10 MB, 134.95 MB, 134.95 MB, 136.72 MB, 138.72 MB, 151.40 MB, 83.73 MB, 67.73 MB ThreadUsage(500.0ms): 10, 10, 10, 10, 10, 10, 10, 10, 8, 7, 6, 6, 5, 5, 5, 4, 3, 3, 3, 3, 3, 2, 2 - AverageThreadTokens: 6.30 - PeakMemoryUsage: 678.14 MB (711084888) - PerHostPeakMemUsage: 678.14 MB (711084888) - PrepareTime: 340.659us - RowsProduced: 0 (0) - TotalCpuTime: 1m1s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 89ns - TotalStorageWaitTime: 38s171ms BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 22.966us, non-child: 22.966us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 11s707ms, non-child: 11s707ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/143.38 MB 2:1/87.61 MB 3:1/136.41 MB 4:2/142.93 MB 5:1/135.42 MB 6:2/270.97 MB 10:1/42.87 MB Hdfs Read Thread Concurrency Bucket: 0:43.48% 1:8.696% 2:0% 3:8.696% 4:0% 5:0% 6:4.348% 7:0% 8:4.348% 9:0% 10:4.348% 11:8.696% 12:8.696% 13:0% 14:0% 15:8.696% File Formats: PARQUET/NONE:495 ExecOption: Codegen enabled: 0 out of 9 BytesRead(500.0ms): 200.00 KB, 26.78 MB, 112.46 MB, 394.03 MB, 433.66 MB, 529.39 MB, 531.84 MB, 560.95 MB, 626.03 MB, 786.68 MB, 876.56 MB, 876.56 MB, 883.90 MB, 898.75 MB, 919.85 MB, 928.44 MB, 929.19 MB, 929.19 MB, 930.56 MB, 932.39 MB, 960.42 MB, 960.42 MB, 960.42 MB - AverageHdfsReadThreadConcurrency: 5.35 - AverageScannerThreadConcurrency: 5.30 - BytesRead: 960.42 MB (1007073793) - BytesReadDataNodeCache: 0 - BytesReadLocal: 960.42 MB (1007073793) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 960.42 MB (1007073793) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 7 (7) - NumScannerThreadsStarted: 9 (9) - PeakMemoryUsage: 678.13 MB (711072600) - PerReadThreadRawHdfsThroughput: 15.36 MB/sec - RemoteScanRanges: 0 (0) - RowsRead: 12.95M (12945014) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 9 (9) - ScannerThreadsInvoluntaryContextSwitches: 103 (103) - ScannerThreadsTotalWallClockTime: 1m1s - MaterializeTupleTime(*): 22s289ms - ScannerThreadsSysTime: 12.0ms - ScannerThreadsUserTime: 22s333ms - ScannerThreadsVoluntaryContextSwitches: 470 (470) - TotalRawHdfsReadTime(*): 1m2s - TotalReadThroughput: 83.51 MB/sec Instance 704b7e71011e9573:13c1490afe0019b5 (host=hadoop11:22000):(Total: 11s065ms, non-child: 1.812ms, % non-child: 0.02%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:1/142.62 MB 5:1/57.63 MB 8:1/140.83 MB 9:1/133.38 MB 10:1/145.14 MB MemoryUsage(500.0ms): 53.90 MB, 63.50 MB, 103.03 MB, 135.89 MB, 63.65 MB, 132.26 MB, 187.54 MB, 201.00 MB, 212.13 MB, 277.55 MB, 440.50 MB, 464.58 MB, 452.99 MB, 471.39 MB, 432.62 MB, 308.29 MB, 191.71 MB, 211.78 MB, 142.19 MB, 142.19 MB, 94.76 MB, 71.07 MB ThreadUsage(500.0ms): 6, 6, 6, 6, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 3, 3, 2, 2, 2, 2 - AverageThreadTokens: 4.41 - PeakMemoryUsage: 480.09 MB (503408440) - PerHostPeakMemUsage: 480.09 MB (503408440) - PrepareTime: 229.161us - RowsProduced: 0 (0) - TotalCpuTime: 38s369ms - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 84ns - TotalStorageWaitTime: 25s819ms BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 17.220us, non-child: 17.220us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 11s063ms, non-child: 11s063ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:1/142.62 MB 5:1/57.63 MB 8:1/140.83 MB 9:1/133.38 MB 10:1/145.14 MB Hdfs Read Thread Concurrency Bucket: 0:36.36% 1:0% 2:9.091% 3:4.545% 4:9.091% 5:9.091% 6:22.73% 7:0% 8:0% 9:0% 10:4.545% 11:4.545% 12:0% 13:0% 14:0% 15:0% File Formats: PARQUET/NONE:275 ExecOption: Codegen enabled: 0 out of 5 BytesRead(500.0ms): 12.93 MB, 40.32 MB, 65.82 MB, 75.22 MB, 91.29 MB, 134.78 MB, 174.10 MB, 200.26 MB, 209.75 MB, 256.35 MB, 405.67 MB, 445.45 MB, 461.35 MB, 520.52 MB, 552.96 MB, 555.47 MB, 555.47 MB, 578.09 MB, 607.11 MB, 620.05 MB, 620.05 MB, 620.05 MB - AverageHdfsReadThreadConcurrency: 3.45 - AverageScannerThreadConcurrency: 3.41 - BytesRead: 620.05 MB (650173562) - BytesReadDataNodeCache: 0 - BytesReadLocal: 562.34 MB (589651218) - BytesReadRemoteUnexpected: 57.72 MB (60522344) - BytesReadShortCircuit: 562.34 MB (589651218) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 5 (5) - NumScannerThreadsStarted: 5 (5) - PeakMemoryUsage: 480.08 MB (503396152) - PerReadThreadRawHdfsThroughput: 16.21 MB/sec - RemoteScanRanges: 56 (56) - RowsRead: 8.11M (8114661) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 5 (5) - ScannerThreadsInvoluntaryContextSwitches: 388 (388) - ScannerThreadsTotalWallClockTime: 38s369ms - MaterializeTupleTime(*): 11s918ms - ScannerThreadsSysTime: 52.2ms - ScannerThreadsUserTime: 11s896ms - ScannerThreadsVoluntaryContextSwitches: 290 (290) - TotalRawHdfsReadTime(*): 38s248ms - TotalReadThroughput: 56.37 MB/sec Instance 704b7e71011e9573:13c1490afe0019ac (host=hadoop32:22000):(Total: 9s419ms, non-child: 656.346us, % non-child: 0.01%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/94.38 MB 1:1/124.58 MB 2:1/48.36 MB 3:1/109.42 MB 4:1/139.54 MB 7:1/80.43 MB 10:2/170.99 MB MemoryUsage(500.0ms): 146.18 MB, 244.27 MB, 345.94 MB, 476.02 MB, 486.60 MB, 541.50 MB, 453.52 MB, 404.95 MB, 354.03 MB, 227.85 MB, 218.58 MB, 200.10 MB, 202.36 MB, 186.37 MB, 233.96 MB, 194.50 MB, 129.12 MB, 89.47 MB, 65.69 MB ThreadUsage(500.0ms): 9, 9, 9, 9, 9, 9, 8, 6, 6, 4, 4, 3, 3, 3, 3, 3, 2, 2, 2 - AverageThreadTokens: 5.42 - PeakMemoryUsage: 557.23 MB (584294024) - PerHostPeakMemUsage: 557.23 MB (584294024) - PrepareTime: 220.25us - RowsProduced: 0 (0) - TotalCpuTime: 41s764ms - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 113ns - TotalStorageWaitTime: 22s188ms BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 15.666us, non-child: 15.666us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 9s418ms, non-child: 9s418ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/94.38 MB 1:1/124.58 MB 2:1/48.36 MB 3:1/109.42 MB 4:1/139.54 MB 7:1/80.43 MB 10:2/170.99 MB Hdfs Read Thread Concurrency Bucket: 0:42.11% 1:5.263% 2:0% 3:0% 4:10.53% 5:0% 6:5.263% 7:5.263% 8:0% 9:5.263% 10:0% 11:5.263% 12:5.263% 13:5.263% 14:0% 15:10.53% File Formats: PARQUET/NONE:440 ExecOption: Codegen enabled: 0 out of 8 BytesRead(500.0ms): 10.14 MB, 125.25 MB, 269.67 MB, 363.27 MB, 394.25 MB, 443.50 MB, 512.60 MB, 555.10 MB, 589.38 MB, 642.96 MB, 673.56 MB, 717.12 MB, 721.19 MB, 721.19 MB, 768.43 MB, 768.43 MB, 768.43 MB, 768.43 MB, 768.43 MB - AverageHdfsReadThreadConcurrency: 5.47 - AverageScannerThreadConcurrency: 4.42 - BytesRead: 768.43 MB (805753576) - BytesReadDataNodeCache: 0 - BytesReadLocal: 768.43 MB (805753576) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 768.43 MB (805753576) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 7 (7) - NumScannerThreadsStarted: 8 (8) - PeakMemoryUsage: 557.21 MB (584281736) - PerReadThreadRawHdfsThroughput: 15.91 MB/sec - RemoteScanRanges: 0 (0) - RowsRead: 10.56M (10557343) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 8 (8) - ScannerThreadsInvoluntaryContextSwitches: 297 (297) - ScannerThreadsTotalWallClockTime: 41s764ms - MaterializeTupleTime(*): 18s484ms - ScannerThreadsSysTime: 92.4ms - ScannerThreadsUserTime: 18s433ms - ScannerThreadsVoluntaryContextSwitches: 420 (420) - TotalRawHdfsReadTime(*): 48s288ms - TotalReadThroughput: 80.89 MB/sec Instance 704b7e71011e9573:13c1490afe0019ae (host=hadoop6:22000):(Total: 9s060ms, non-child: 742.383us, % non-child: 0.01%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/51.63 MB 11:3/265.18 MB 2:1/136.60 MB 4:2/277.44 MB 5:2/206.30 MB 6:1/83.33 MB 8:1/119.78 MB 9:1/137.37 MB 10:1/141.36 MB MemoryUsage(500.0ms): 183.27 MB, 371.99 MB, 452.27 MB, 701.64 MB, 905.56 MB, 939.74 MB, 897.45 MB, 1.06 GB, 1.04 GB, 1.07 GB, 1.03 GB, 986.96 MB, 1.02 GB, 903.46 MB, 740.03 MB, 387.88 MB, 168.08 MB, 92.30 MB ThreadUsage(500.0ms): 14, 14, 14, 14, 13, 13, 12, 12, 12, 12, 12, 12, 12, 10, 9, 5, 3, 2 - AverageThreadTokens: 10.83 - PeakMemoryUsage: 1.07 GB (1151744936) - PerHostPeakMemUsage: 1.07 GB (1151744936) - PrepareTime: 254.393us - RowsProduced: 0 (0) - TotalCpuTime: 1m29s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 92ns - TotalStorageWaitTime: 41s997ms BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 16.687us, non-child: 16.687us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 9s059ms, non-child: 9s059ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/51.63 MB 11:3/265.18 MB 2:1/136.60 MB 4:2/277.44 MB 5:2/206.30 MB 6:1/83.33 MB 8:1/119.78 MB 9:1/137.37 MB 10:1/141.36 MB Hdfs Read Thread Concurrency Bucket: 0:22.22% 1:0% 2:5.556% 3:0% 4:5.556% 5:5.556% 6:0% 7:5.556% 8:5.556% 9:5.556% 10:0% 11:5.556% 12:0% 13:11.11% 14:0% 15:27.78% File Formats: PARQUET/NONE:715 ExecOption: Codegen enabled: 0 out of 13 BytesRead(500.0ms): 9.83 MB, 128.89 MB, 279.75 MB, 530.25 MB, 767.30 MB, 786.57 MB, 811.06 MB, 1.01 GB, 1.03 GB, 1.07 GB, 1.08 GB, 1.12 GB, 1.25 GB, 1.38 GB, 1.39 GB, 1.39 GB, 1.39 GB, 1.39 GB - AverageHdfsReadThreadConcurrency: 11.11 - AverageScannerThreadConcurrency: 9.83 - BytesRead: 1.39 GB (1489169920) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.39 GB (1489169920) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 1.39 GB (1489169920) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 9 (9) - NumScannerThreadsStarted: 13 (13) - PeakMemoryUsage: 1.07 GB (1151732648) - PerReadThreadRawHdfsThroughput: 14.46 MB/sec - RemoteScanRanges: 0 (0) - RowsRead: 18.99M (18994692) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 13 (13) - ScannerThreadsInvoluntaryContextSwitches: 616 (616) - ScannerThreadsTotalWallClockTime: 1m29s - MaterializeTupleTime(*): 46s366ms - ScannerThreadsSysTime: 88.3ms - ScannerThreadsUserTime: 46s330ms - ScannerThreadsVoluntaryContextSwitches: 455 (455) - TotalRawHdfsReadTime(*): 1m38s - TotalReadThroughput: 157.80 MB/sec Instance 704b7e71011e9573:13c1490afe0019b0 (host=hadoop14:22000):(Total: 7s224ms, non-child: 683.613us, % non-child: 0.01%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:1/126.79 MB 4:1/135.91 MB 5:1/135.04 MB 6:1/88.99 MB 8:1/47.71 MB 9:2/224.17 MB 10:1/130.33 MB MemoryUsage(500.0ms): 65.26 MB, 163.02 MB, 289.86 MB, 432.92 MB, 480.47 MB, 655.31 MB, 758.47 MB, 729.98 MB, 725.83 MB, 602.17 MB, 419.64 MB, 420.24 MB, 289.01 MB, 170.49 MB ThreadUsage(500.0ms): 9, 9, 9, 9, 9, 9, 9, 9, 9, 7, 5, 5, 4, 3 - AverageThreadTokens: 7.50 - PeakMemoryUsage: 774.47 MB (812087880) - PerHostPeakMemUsage: 774.47 MB (812087880) - PrepareTime: 186.5us - RowsProduced: 0 (0) - TotalCpuTime: 46s197ms - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 91ns - TotalStorageWaitTime: 26s701ms BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 13.110us, non-child: 13.110us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 7s224ms, non-child: 7s224ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 2:1/126.79 MB 4:1/135.91 MB 5:1/135.04 MB 6:1/88.99 MB 8:1/47.71 MB 9:2/224.17 MB 10:1/130.33 MB Hdfs Read Thread Concurrency Bucket: 0:14.29% 1:7.143% 2:0% 3:0% 4:0% 5:7.143% 6:14.29% 7:0% 8:7.143% 9:0% 10:7.143% 11:7.143% 12:7.143% 13:0% 14:0% 15:28.57% File Formats: PARQUET/NONE:440 ExecOption: Codegen enabled: 0 out of 8 BytesRead(500.0ms): 8.20 MB, 50.79 MB, 96.72 MB, 238.82 MB, 356.45 MB, 529.66 MB, 624.64 MB, 641.76 MB, 720.92 MB, 772.87 MB, 784.49 MB, 880.29 MB, 889.67 MB, 889.67 MB - AverageHdfsReadThreadConcurrency: 10.43 - AverageScannerThreadConcurrency: 6.50 - BytesRead: 889.67 MB (932886343) - BytesReadDataNodeCache: 0 - BytesReadLocal: 631.88 MB (662572320) - BytesReadRemoteUnexpected: 257.79 MB (270314023) - BytesReadShortCircuit: 631.88 MB (662572320) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 7 (7) - NumScannerThreadsStarted: 8 (8) - PeakMemoryUsage: 774.46 MB (812075592) - PerReadThreadRawHdfsThroughput: 12.35 MB/sec - RemoteScanRanges: 112 (112) - RowsRead: 11.86M (11860260) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 8 (8) - ScannerThreadsInvoluntaryContextSwitches: 401 (401) - ScannerThreadsTotalWallClockTime: 46s197ms - MaterializeTupleTime(*): 18s084ms - ScannerThreadsSysTime: 100.4ms - ScannerThreadsUserTime: 18s065ms - ScannerThreadsVoluntaryContextSwitches: 439 (439) - TotalRawHdfsReadTime(*): 1m12s - TotalReadThroughput: 127.10 MB/sec
... View more