About ponypony

ponypony · ‎09-10-2017

Sometimes see the similar error during step 2) as well. Do i need to run invalidate after step 1?

ponypony · ‎09-10-2017

Hi, For my table, I am regenerating for a partition, what i do is 1) remove from hdfs 2) insert to the partition from my staging table 3) invalidate metadata on the table 4) compute incremental stat on the partition from time to time, i see this on step 4 ERROR: AnalysisException: Failed to load metadata for table: 'ABC' CAUSED BY: TableLoadingException: Failed to load metadata for table: default.ABC. Running 'invalidate metadata default.ABC' may resolve this problem. CAUSED BY: MetaException: Object with id "" is managed by a different persistence manager Thanks Shannon

ponypony · ‎09-07-2017

Hi, A quick question, if my insert overwrite only goes to one partition, then only that partition (hdfs partition directory) will be overwritten, right? Thanks Shannon

ponypony · ‎08-15-2017

Thanks Lars. Shannon

ponypony · ‎08-14-2017

Hi Lars, Have you had time to look at the profile? Thanks Shannon

ponypony · ‎08-11-2017

Thanks, i will give it a try, i can create my final tables to have it converted to timestamp. Shannon

ponypony · ‎08-11-2017

Too long, had to cut in two parts. Thanks Shannon

ponypony · ‎08-11-2017

Fragment F00: Instance 1344bea35ad1b2d9:3 (host=host3.local:22000):(Total: 1m58s, non-child: 1m43s, % non-child: 87.81%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:8/2.10 GB MemoryUsage(2s000ms): 65.61 MB, 70.61 MB, 70.61 MB, 70.61 MB, 70.61 MB, 70.61 MB, 70.61 MB, 70.61 MB, 70.61 MB, 70.61 MB, 70.61 MB, 70.61 MB, 70.60 MB, 60.08 MB, 52.57 MB, 40.60 MB, 28.57 MB, 20.53 MB, 64.83 MB, 65.82 MB, 58.81 MB, 42.57 MB, 34.58 MB, 26.84 MB, 34.83 MB, 73.09 MB, 60.54 MB, 50.58 MB, 42.58 MB, 28.59 MB, 23.05 MB, 51.08 MB, 69.09 MB, 58.59 MB, 47.08 MB, 32.79 MB, 22.55 MB, 17.81 MB, 50.56 MB, 62.57 MB, 51.55 MB, 41.54 MB, 33.53 MB, 23.79 MB, 36.79 MB, 69.05 MB, 62.29 MB, 50.55 MB, 36.56 MB, 28.52 MB, 43.81 MB, 57.05 MB, 31.55 MB, 44.79 MB, 43.05 MB, 31.68 MB, 63.60 MB, 59.57 MB, 26.96 MB ThreadUsage(2s000ms): 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 2 - AverageThreadTokens: 3.03 - BloomFilterBytes: 0 - PeakMemoryUsage: 85.41 MB (89556266) - PerHostPeakMemUsage: 1.98 GB (2129589388) - RowsProduced: 50.91M (50911383) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 1m34s - TotalStorageWaitTime: 337.999ms - TotalThreadsInvoluntaryContextSwitches: 30 (30) - TotalThreadsTotalWallClockTime: 5m57s - TotalThreadsSysTime: 1s827ms - TotalThreadsUserTime: 25s857ms - TotalThreadsVoluntaryContextSwitches: 79.32K (79316) Fragment Instance Lifecycle Timings: - ExecTime: 1m57s - ExecTreeExecTime: 505.999ms - OpenTime: 23.000ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 23.000ms - ExecTreePrepareTime: 0.000ns DataStreamSender (dst_id=4):(Total: 13s902ms, non-child: 13s902ms, % non-child: 100.00%) - BytesSent: 1.36 GB (1462632568) - NetworkThroughput(*): 377.40 MB/sec - OverallThroughput: 100.33 MB/sec - PeakMemoryUsage: 281.05 KB (287792) - RowsReturned: 50.91M (50911383) - SerializeBatchTime: 10s194ms - TransmitDataRPCTime: 3s695ms - UncompressedRowBatchSize: 3.21 GB (3448072977) CodeGen:(Total: 42.000ms, non-child: 42.000ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 6.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 5 (5) - NumInstructions: 98 (98) - OptimizationTime: 12.000ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 23.000ms HDFS_SCAN_NODE (id=0):(Total: 448.999ms, non-child: 448.999ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:8/2.10 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 8 out of 8 Hdfs Read Thread Concurrency Bucket: 0:99.58% 1:0.4237% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:8 BytesRead(2s000ms): 62.97 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 67.89 MB, 128.91 MB, 137.38 MB, 137.38 MB, 137.38 MB, 137.38 MB, 137.38 MB, 164.03 MB, 208.40 MB, 208.40 MB, 208.40 MB, 208.40 MB, 208.40 MB, 208.40 MB, 251.55 MB, 277.74 MB, 277.74 MB, 277.74 MB, 277.74 MB, 277.74 MB, 287.19 MB, 330.79 MB, 355.60 MB, 355.60 MB, 355.60 MB, 355.60 MB, 355.60 MB, 380.52 MB, 423.62 MB, 423.62 MB, 423.62 MB, 423.62 MB, 423.62 MB, 456.85 MB, 489.69 MB, 489.69 MB, 522.92 MB, 541.33 MB, 547.67 MB, 598.16 MB, 617.44 MB, 618.17 MB - FooterProcessingTime: (Avg: 13.749ms ; Min: 1.999ms ; Max: 43.000ms ; Number of samples: 😎 - AverageHdfsReadThreadConcurrency: 0.00 - AverageScannerThreadConcurrency: 2.03 - BytesRead: 618.17 MB (648193052) - BytesReadDataNodeCache: 0 - BytesReadLocal: 617.97 MB (647988252) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 617.97 MB (647988252) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 1 (1) - NumDisksAccessed: 2 (2) - NumRowGroups: 23 (23) - NumScannerThreadsStarted: 3 (3) - PeakMemoryUsage: 83.37 MB (87422988) - PerReadThreadRawHdfsThroughput: 1.80 GB/sec - RemoteScanRanges: 2 (2) - RowBatchQueueGetWaitTime: 64.000ms - RowBatchQueuePutWaitTime: 3m56s - RowsRead: 50.91M (50911383) - RowsReturned: 50.91M (50911383) - RowsReturnedRate: 113.39 M/sec - ScanRangesComplete: 8 (8) - ScannerThreadsInvoluntaryContextSwitches: 0 (0) - ScannerThreadsTotalWallClockTime: 3m59s - MaterializeTupleTime(*): 1s729ms - ScannerThreadsSysTime: 394.562ms - ScannerThreadsUserTime: 2s376ms - ScannerThreadsVoluntaryContextSwitches: 49.46K (49459) - TotalRawHdfsReadTime(*): 335.999ms - TotalReadThroughput: 5.24 MB/sec Instance 1344bea35ad1b2d9:5 (host=host5.local:22000):(Total: 1m55s, non-child: 1m44s, % non-child: 90.50%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:10/1.70 GB MemoryUsage(2s000ms): 36.44 MB, 48.59 MB, 48.59 MB, 48.59 MB, 48.59 MB, 48.59 MB, 48.59 MB, 48.59 MB, 48.59 MB, 48.59 MB, 48.59 MB, 48.59 MB, 48.57 MB, 38.32 MB, 44.56 MB, 39.59 MB, 30.83 MB, 24.56 MB, 20.60 MB, 40.82 MB, 48.85 MB, 47.58 MB, 38.07 MB, 37.10 MB, 37.83 MB, 26.60 MB, 40.07 MB, 72.59 MB, 63.80 MB, 58.59 MB, 42.57 MB, 34.54 MB, 26.81 MB, 42.57 MB, 74.58 MB, 60.55 MB, 56.55 MB, 44.56 MB, 32.58 MB, 26.59 MB, 21.83 MB, 53.56 MB, 69.59 MB, 60.84 MB, 52.82 MB, 40.57 MB, 32.59 MB, 22.59 MB, 37.83 MB, 72.32 MB, 62.31 MB, 42.83 MB, 28.80 MB, 27.26 MB, 23.50 MB, 25.00 MB, 27.50 MB ThreadUsage(2s000ms): 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2 - AverageThreadTokens: 2.91 - BloomFilterBytes: 0 - PeakMemoryUsage: 84.59 MB (88703298) - PerHostPeakMemUsage: 1.97 GB (2113493539) - RowsProduced: 40.70M (40701211) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 1m36s - TotalStorageWaitTime: 266.000ms - TotalThreadsInvoluntaryContextSwitches: 16 (16) - TotalThreadsTotalWallClockTime: 5m36s - TotalThreadsSysTime: 1s617ms - TotalThreadsUserTime: 20s078ms - TotalThreadsVoluntaryContextSwitches: 62.46K (62456) Fragment Instance Lifecycle Timings: - ExecTime: 1m55s - ExecTreeExecTime: 403.000ms - OpenTime: 18.000ms - ExecTreeOpenTime: 1.000ms - PrepareTime: 19.000ms - ExecTreePrepareTime: 0.000ns DataStreamSender (dst_id=4):(Total: 10s579ms, non-child: 10s579ms, % non-child: 100.00%) - BytesSent: 1.08 GB (1163676328) - NetworkThroughput(*): 407.25 MB/sec - OverallThroughput: 104.90 MB/sec - PeakMemoryUsage: 281.05 KB (287792) - RowsReturned: 40.70M (40701211) - SerializeBatchTime: 7s846ms - TransmitDataRPCTime: 2s725ms - UncompressedRowBatchSize: 2.58 GB (2768169800) CodeGen:(Total: 34.000ms, non-child: 34.000ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 6.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 5 (5) - NumInstructions: 98 (98) - OptimizationTime: 10.000ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 18.000ms HDFS_SCAN_NODE (id=0):(Total: 365.000ms, non-child: 365.000ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:10/1.70 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 10 out of 10 Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:10 BytesRead(2s000ms): 32.51 MB, 43.35 MB, 43.35 MB, 43.35 MB, 43.35 MB, 43.35 MB, 43.35 MB, 43.35 MB, 43.35 MB, 43.35 MB, 43.35 MB, 43.35 MB, 43.35 MB, 44.11 MB, 57.19 MB, 60.79 MB, 60.79 MB, 65.21 MB, 69.62 MB, 100.67 MB, 113.44 MB, 125.47 MB, 125.47 MB, 134.19 MB, 142.91 MB, 142.91 MB, 168.29 MB, 210.33 MB, 210.33 MB, 210.33 MB, 210.33 MB, 210.33 MB, 210.33 MB, 245.18 MB, 280.02 MB, 280.02 MB, 280.02 MB, 280.02 MB, 280.02 MB, 280.02 MB, 291.98 MB, 330.62 MB, 357.44 MB, 357.44 MB, 357.44 MB, 357.44 MB, 357.44 MB, 357.44 MB, 383.16 MB, 426.92 MB, 426.92 MB, 426.92 MB, 426.92 MB, 453.33 MB, 462.13 MB, 479.74 MB, 497.34 MB - FooterProcessingTime: (Avg: 8.500ms ; Min: 2.000ms ; Max: 30.000ms ; Number of samples: 10) - AverageHdfsReadThreadConcurrency: 0.00 - AverageScannerThreadConcurrency: 1.92 - BytesRead: 505.61 MB (530170312) - BytesReadDataNodeCache: 0 - BytesReadLocal: 505.61 MB (530170312) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 505.61 MB (530170312) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 1 (1) - NumDisksAccessed: 2 (2) - NumRowGroups: 22 (22) - NumScannerThreadsStarted: 2 (2) - PeakMemoryUsage: 84.37 MB (88471576) - PerReadThreadRawHdfsThroughput: 1.91 GB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 38.000ms - RowBatchQueuePutWaitTime: 3m39s - RowsRead: 40.70M (40701211) - RowsReturned: 40.70M (40701211) - RowsReturnedRate: 111.51 M/sec - ScanRangesComplete: 10 (10) - ScannerThreadsInvoluntaryContextSwitches: 1 (1) - ScannerThreadsTotalWallClockTime: 3m41s - MaterializeTupleTime(*): 1s374ms - ScannerThreadsSysTime: 324.779ms - ScannerThreadsUserTime: 1s873ms - ScannerThreadsVoluntaryContextSwitches: 39.23K (39235) - TotalRawHdfsReadTime(*): 259.000ms - TotalReadThroughput: 4.38 MB/sec Instance 1344bea35ad1b2d9:1 (host=host2.local:22000):(Total: 1m53s, non-child: 1m39s, % non-child: 87.74%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:7/2.09 GB MemoryUsage(2s000ms): 58.30 MB, 71.54 MB, 71.54 MB, 71.54 MB, 71.54 MB, 71.54 MB, 71.54 MB, 71.54 MB, 71.54 MB, 71.54 MB, 71.54 MB, 71.54 MB, 71.54 MB, 61.04 MB, 53.55 MB, 41.81 MB, 27.55 MB, 26.30 MB, 78.54 MB, 68.28 MB, 56.53 MB, 46.53 MB, 34.80 MB, 22.56 MB, 64.55 MB, 64.55 MB, 55.54 MB, 43.55 MB, 35.78 MB, 23.79 MB, 44.78 MB, 68.03 MB, 55.56 MB, 45.79 MB, 33.78 MB, 20.29 MB, 17.03 MB, 46.56 MB, 39.54 MB, 40.58 MB, 53.83 MB, 42.60 MB, 55.05 MB, 54.08 MB, 42.59 MB, 51.57 MB, 46.04 MB, 43.58 MB, 57.56 MB, 44.55 MB, 43.08 MB, 48.05 MB, 46.30 MB, 44.81 MB, 32.15 MB, 35.09 MB ThreadUsage(2s000ms): 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3 - AverageThreadTokens: 2.99 - BloomFilterBytes: 0 - PeakMemoryUsage: 87.58 MB (91832630) - PerHostPeakMemUsage: 1.99 GB (2131438970) - RowsProduced: 49.80M (49799808) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 1m30s - TotalStorageWaitTime: 226.999ms - TotalThreadsInvoluntaryContextSwitches: 38 (38) - TotalThreadsTotalWallClockTime: 5m38s - TotalThreadsSysTime: 1s958ms - TotalThreadsUserTime: 25s012ms - TotalThreadsVoluntaryContextSwitches: 76.58K (76582) Fragment Instance Lifecycle Timings: - ExecTime: 1m53s - ExecTreeExecTime: 455.999ms - OpenTime: 21.999ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 25.999ms - ExecTreePrepareTime: 0.000ns DataStreamSender (dst_id=4):(Total: 13s474ms, non-child: 13s474ms, % non-child: 100.00%) - BytesSent: 1.33 GB (1431613338) - NetworkThroughput(*): 405.13 MB/sec - OverallThroughput: 101.32 MB/sec - PeakMemoryUsage: 281.05 KB (287792) - RowsReturned: 49.80M (49799808) - SerializeBatchTime: 10s091ms - TransmitDataRPCTime: 3s369ms - UncompressedRowBatchSize: 3.14 GB (3374252371) CodeGen:(Total: 46.999ms, non-child: 46.999ms, % non-child: 100.00%) - CodegenTime: 1.000ms - CompileTime: 4.999ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 5 (5) - NumInstructions: 98 (98) - OptimizationTime: 13.999ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 26.999ms HDFS_SCAN_NODE (id=0):(Total: 398.999ms, non-child: 398.999ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:7/2.09 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7 Hdfs Read Thread Concurrency Bucket: 0:99.56% 1:0.4405% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:7 BytesRead(2s000ms): 54.48 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 67.25 MB, 76.42 MB, 138.17 MB, 138.17 MB, 138.17 MB, 138.17 MB, 138.17 MB, 138.17 MB, 196.75 MB, 204.90 MB, 204.90 MB, 204.90 MB, 204.90 MB, 204.90 MB, 238.84 MB, 271.96 MB, 271.96 MB, 271.96 MB, 271.96 MB, 273.78 MB, 284.92 MB, 324.38 MB, 329.64 MB, 344.24 MB, 369.88 MB, 369.88 MB, 395.64 MB, 404.23 MB, 404.23 MB, 429.62 MB, 438.08 MB, 446.63 MB, 472.30 MB, 472.30 MB, 489.35 MB, 514.95 MB, 540.62 MB, 563.59 MB, 575.33 MB, 599.37 MB - FooterProcessingTime: (Avg: 7.571ms ; Min: 1.999ms ; Max: 16.999ms ; Number of samples: 7) - AverageHdfsReadThreadConcurrency: 0.00 - AverageScannerThreadConcurrency: 1.99 - BytesRead: 605.98 MB (635419430) - BytesReadDataNodeCache: 0 - BytesReadLocal: 605.79 MB (635214630) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 605.79 MB (635214630) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 1 (1) - NumDisksAccessed: 2 (2) - NumRowGroups: 22 (22) - NumScannerThreadsStarted: 2 (2) - PeakMemoryUsage: 87.37 MB (91617292) - PerReadThreadRawHdfsThroughput: 1.65 GB/sec - RemoteScanRanges: 2 (2) - RowBatchQueueGetWaitTime: 26.999ms - RowBatchQueuePutWaitTime: 3m42s - RowsRead: 49.80M (49799808) - RowsReturned: 49.80M (49799808) - RowsReturnedRate: 124.81 M/sec - ScanRangesComplete: 7 (7) - ScannerThreadsInvoluntaryContextSwitches: 0 (0) - ScannerThreadsTotalWallClockTime: 3m45s - MaterializeTupleTime(*): 1s762ms - ScannerThreadsSysTime: 353.880ms - ScannerThreadsUserTime: 2s392ms - ScannerThreadsVoluntaryContextSwitches: 48.34K (48336) - TotalRawHdfsReadTime(*): 358.999ms - TotalReadThroughput: 5.34 MB/sec Instance 1344bea35ad1b2d9:4 (host=host4.local:22000):(Total: 1m40s, non-child: 1m31s, % non-child: 90.45%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:5/1.62 GB MemoryUsage(2s000ms): 63.09 MB, 72.57 MB, 72.57 MB, 72.57 MB, 72.57 MB, 72.57 MB, 72.57 MB, 72.57 MB, 72.57 MB, 72.57 MB, 72.57 MB, 72.57 MB, 72.57 MB, 60.33 MB, 48.58 MB, 36.57 MB, 26.57 MB, 41.55 MB, 69.06 MB, 58.58 MB, 42.59 MB, 34.58 MB, 22.58 MB, 58.78 MB, 69.54 MB, 58.85 MB, 44.58 MB, 30.59 MB, 20.60 MB, 51.34 MB, 69.58 MB, 58.59 MB, 44.58 MB, 34.56 MB, 22.83 MB, 27.30 MB, 59.94 MB, 57.32 MB, 43.55 MB, 31.54 MB, 20.53 MB, 37.53 MB, 26.48 MB, 16.48 MB, 28.24 MB, 27.47 MB, 17.49 MB, 28.99 MB, 27.46 MB, 17.74 MB ThreadUsage(2s000ms): 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2, 2, 2, 2, 2, 2 - AverageThreadTokens: 2.82 - BloomFilterBytes: 0 - PeakMemoryUsage: 83.59 MB (87654710) - PerHostPeakMemUsage: 1.98 GB (2126411237) - RowsProduced: 38.09M (38093430) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 1m24s - TotalStorageWaitTime: 190.000ms - TotalThreadsInvoluntaryContextSwitches: 11 (11) - TotalThreadsTotalWallClockTime: 4m44s - TotalThreadsSysTime: 1s244ms - TotalThreadsUserTime: 17s907ms - TotalThreadsVoluntaryContextSwitches: 60.37K (60365) Fragment Instance Lifecycle Timings: - ExecTime: 1m40s - ExecTreeExecTime: 295.000ms - OpenTime: 13.000ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 19.000ms - ExecTreePrepareTime: 0.000ns DataStreamSender (dst_id=4):(Total: 9s323ms, non-child: 9s323ms, % non-child: 100.00%) - BytesSent: 1.02 GB (1092673052) - NetworkThroughput(*): 447.43 MB/sec - OverallThroughput: 111.77 MB/sec - PeakMemoryUsage: 281.05 KB (287792) - RowsReturned: 38.09M (38093430) - SerializeBatchTime: 6s986ms - TransmitDataRPCTime: 2s329ms - UncompressedRowBatchSize: 2.41 GB (2582603828) CodeGen:(Total: 31.000ms, non-child: 31.000ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 4.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 5 (5) - NumInstructions: 98 (98) - OptimizationTime: 7.000ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 20.000ms HDFS_SCAN_NODE (id=0):(Total: 266.000ms, non-child: 266.000ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:5/1.62 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out of 5 Hdfs Read Thread Concurrency Bucket: 0:99.01% 1:0.9901% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:5 BytesRead(2s000ms): 57.34 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 68.38 MB, 102.17 MB, 135.96 MB, 135.96 MB, 135.96 MB, 135.96 MB, 135.96 MB, 186.33 MB, 206.98 MB, 206.98 MB, 206.98 MB, 206.98 MB, 206.98 MB, 250.72 MB, 276.75 MB, 276.75 MB, 276.75 MB, 276.75 MB, 276.75 MB, 298.48 MB, 346.57 MB, 354.44 MB, 354.44 MB, 354.44 MB, 355.34 MB, 390.55 MB, 390.55 MB, 390.55 MB, 416.50 MB, 425.15 MB, 425.15 MB, 451.18 MB, 459.86 MB, 459.86 MB - FooterProcessingTime: (Avg: 8.800ms ; Min: 1.000ms ; Max: 21.000ms ; Number of samples: 5) - AverageHdfsReadThreadConcurrency: 0.01 - AverageScannerThreadConcurrency: 1.82 - BytesRead: 464.96 MB (487548706) - BytesReadDataNodeCache: 0 - BytesReadLocal: 464.96 MB (487548706) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 464.96 MB (487548706) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 1 (1) - NumDisksAccessed: 2 (2) - NumRowGroups: 17 (17) - NumScannerThreadsStarted: 2 (2) - PeakMemoryUsage: 83.37 MB (87422988) - PerReadThreadRawHdfsThroughput: 1.97 GB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 39.000ms - RowBatchQueuePutWaitTime: 3m2s - RowsRead: 38.09M (38093430) - RowsReturned: 38.09M (38093430) - RowsReturnedRate: 143.21 M/sec - ScanRangesComplete: 5 (5) - ScannerThreadsInvoluntaryContextSwitches: 1 (1) - ScannerThreadsTotalWallClockTime: 3m3s - MaterializeTupleTime(*): 1s127ms - ScannerThreadsSysTime: 236.054ms - ScannerThreadsUserTime: 1s631ms - ScannerThreadsVoluntaryContextSwitches: 36.86K (36861) - TotalRawHdfsReadTime(*): 231.000ms - TotalReadThroughput: 4.60 MB/sec Instance 1344bea35ad1b2d9:2 (host=host1.local:22000):(Total: 1m39s, non-child: 1m27s, % non-child: 87.91%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:8/1.75 GB MemoryUsage(2s000ms): 42.09 MB, 52.59 MB, 52.59 MB, 52.59 MB, 52.59 MB, 52.59 MB, 52.59 MB, 52.59 MB, 52.59 MB, 52.59 MB, 52.59 MB, 52.59 MB, 52.56 MB, 38.57 MB, 57.58 MB, 44.60 MB, 40.82 MB, 53.81 MB, 40.59 MB, 57.53 MB, 44.81 MB, 38.32 MB, 53.57 MB, 38.58 MB, 59.57 MB, 46.57 MB, 30.57 MB, 50.10 MB, 42.82 MB, 56.57 MB, 43.09 MB, 32.71 MB, 39.57 MB, 44.08 MB, 44.07 MB, 53.77 MB, 40.53 MB, 46.80 MB, 34.81 MB, 29.04 MB, 51.04 MB, 42.31 MB, 42.81 MB, 36.06 MB, 22.56 MB, 31.06 MB, 37.74 MB, 23.45 MB, 14.21 MB ThreadUsage(2s000ms): 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2 - AverageThreadTokens: 2.93 - BloomFilterBytes: 0 - PeakMemoryUsage: 82.41 MB (86410562) - PerHostPeakMemUsage: 1.99 GB (2136802350) - RowsProduced: 42.16M (42156762) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 1m19s - TotalStorageWaitTime: 264.000ms - TotalThreadsInvoluntaryContextSwitches: 19 (19) - TotalThreadsTotalWallClockTime: 4m51s - TotalThreadsSysTime: 1s669ms - TotalThreadsUserTime: 21s404ms - TotalThreadsVoluntaryContextSwitches: 67.18K (67177) Fragment Instance Lifecycle Timings: - ExecTime: 1m39s - ExecTreeExecTime: 412.000ms - OpenTime: 18.000ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 23.000ms - ExecTreePrepareTime: 0.000ns DataStreamSender (dst_id=4):(Total: 11s600ms, non-child: 11s600ms, % non-child: 100.00%) - BytesSent: 1.13 GB (1210239627) - NetworkThroughput(*): 368.04 MB/sec - OverallThroughput: 99.50 MB/sec - PeakMemoryUsage: 281.05 KB (287792) - RowsReturned: 42.16M (42156762) - SerializeBatchTime: 8s456ms - TransmitDataRPCTime: 3s136ms - UncompressedRowBatchSize: 2.66 GB (2857012229) CodeGen:(Total: 40.000ms, non-child: 40.000ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 5.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 5 (5) - NumInstructions: 98 (98) - OptimizationTime: 11.000ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 23.000ms HDFS_SCAN_NODE (id=0):(Total: 370.000ms, non-child: 370.000ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:8/1.75 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 8 out of 8 Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:8 BytesRead(2s000ms): 36.49 MB, 45.92 MB, 45.92 MB, 45.92 MB, 45.92 MB, 45.92 MB, 45.92 MB, 45.92 MB, 45.92 MB, 45.92 MB, 45.92 MB, 45.92 MB, 45.92 MB, 45.92 MB, 80.11 MB, 80.11 MB, 88.82 MB, 114.95 MB, 114.95 MB, 148.86 MB, 148.86 MB, 157.57 MB, 183.70 MB, 183.70 MB, 217.80 MB, 217.80 MB, 217.80 MB, 250.28 MB, 259.18 MB, 285.87 MB, 285.87 MB, 288.97 MB, 309.51 MB, 328.78 MB, 342.71 MB, 368.11 MB, 368.11 MB, 390.39 MB, 390.39 MB, 399.10 MB, 435.52 MB, 438.95 MB, 452.64 MB, 457.21 MB, 457.21 MB, 481.41 MB, 501.11 MB, 501.11 MB, 504.28 MB - FooterProcessingTime: (Avg: 6.875ms ; Min: 3.000ms ; Max: 17.000ms ; Number of samples: 😎 - AverageHdfsReadThreadConcurrency: 0.00 - AverageScannerThreadConcurrency: 1.93 - BytesRead: 513.78 MB (538735216) - BytesReadDataNodeCache: 0 - BytesReadLocal: 513.78 MB (538735216) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 513.78 MB (538735216) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 1 (1) - NumDisksAccessed: 2 (2) - NumRowGroups: 21 (21) - NumScannerThreadsStarted: 2 (2) - PeakMemoryUsage: 76.37 MB (80082968) - PerReadThreadRawHdfsThroughput: 1.89 GB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 35.000ms - RowBatchQueuePutWaitTime: 3m10s - RowsRead: 42.16M (42156762) - RowsReturned: 42.16M (42156762) - RowsReturnedRate: 113.94 M/sec - ScanRangesComplete: 8 (8) - ScannerThreadsInvoluntaryContextSwitches: 2 (2) - ScannerThreadsTotalWallClockTime: 3m12s - MaterializeTupleTime(*): 1s535ms - ScannerThreadsSysTime: 296.816ms - ScannerThreadsUserTime: 2s021ms - ScannerThreadsVoluntaryContextSwitches: 40.91K (40915) - TotalRawHdfsReadTime(*): 266.000ms - TotalReadThroughput: 5.16 MB/sec Averaged Fragment F01:(Total: 18s925ms, non-child: 12s840ms, % non-child: 67.85%) split sizes: min: 1.94 GB, max: 2.29 GB, avg: 2.15 GB, stddev: 123.03 MB completion times: min:17s660ms max:22s022ms mean: 18s926ms stddev:1s686ms execution rates: min:90.37 MB/sec max:132.84 MB/sec mean:117.68 MB/sec stddev:15.30 MB/sec num instances: 5 - AverageThreadTokens: 2.93 - BloomFilterBytes: 0 - PeakMemoryUsage: 64.04 MB (67152352) - PerHostPeakMemUsage: 1.41 GB (1514253798) - RowsProduced: 19.26M (19256582) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 9s265ms - TotalStorageWaitTime: 631.399ms - TotalThreadsInvoluntaryContextSwitches: 9 (9) - TotalThreadsTotalWallClockTime: 55s377ms - TotalThreadsSysTime: 785.589ms - TotalThreadsUserTime: 10s291ms - TotalThreadsVoluntaryContextSwitches: 30.07K (30070) Fragment Instance Lifecycle Timings: - ExecTime: 18s883ms - ExecTreeExecTime: 259.399ms - OpenTime: 18.599ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 22.799ms - ExecTreePrepareTime: 0.000ns DataStreamSender (dst_id=5):(Total: 5s805ms, non-child: 5s805ms, % non-child: 100.00%) - BytesSent: 632.46 MB (663179241) - NetworkThroughput(*): 477.86 MB/sec - OverallThroughput: 109.00 MB/sec - PeakMemoryUsage: 151.21 KB (154840) - RowsReturned: 19.26M (19256582) - SerializeBatchTime: 4s465ms - TransmitDataRPCTime: 1s333ms - UncompressedRowBatchSize: 1.36 GB (1459837468) CodeGen:(Total: 37.399ms, non-child: 37.399ms, % non-child: 100.00%) - CodegenTime: 599.999us - CompileTime: 5.399ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 5 (5) - NumInstructions: 98 (98) - OptimizationTime: 9.599ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 21.599ms HDFS_SCAN_NODE (id=1):(Total: 241.599ms, non-child: 241.599ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.02 - AverageScannerThreadConcurrency: 1.94 - BytesRead: 381.78 MB (400327188) - BytesReadDataNodeCache: 0 - BytesReadLocal: 381.70 MB (400245268) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 381.70 MB (400245268) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDisksAccessed: 2 (2) - NumRowGroups: 31 (31) - NumScannerThreadsStarted: 2 (2) - PeakMemoryUsage: 59.10 MB (61972085) - PerReadThreadRawHdfsThroughput: 1.49 GB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 100.399ms - RowBatchQueuePutWaitTime: 34s881ms - RowsRead: 19.26M (19256582) - RowsReturned: 19.26M (19256582) - RowsReturnedRate: 81.69 M/sec - ScanRangesComplete: 6 (6) - ScannerThreadsInvoluntaryContextSwitches: 1 (1) - ScannerThreadsTotalWallClockTime: 36s475ms - MaterializeTupleTime(*): 849.599ms - ScannerThreadsSysTime: 134.190ms - ScannerThreadsUserTime: 1s065ms - ScannerThreadsVoluntaryContextSwitches: 18.82K (18817) - TotalRawHdfsReadTime(*): 253.599ms - TotalReadThroughput: 20.27 MB/sec BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 192 (192) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 1.25 GB (1342177280) - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns Fragment F01: Instance 1344bea35ad1b2d9:a (host=host5.local:22000):(Total: 22s023ms, non-child: 16s484ms, % non-child: 74.85%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:9/1.94 GB MemoryUsage(500.000ms): 11.21 KB, 42.87 MB, 42.90 MB, 42.89 MB, 43.88 MB, 28.90 MB, 39.88 MB, 46.87 MB, 46.91 MB, 39.88 MB, 32.88 MB, 39.87 MB, 46.87 MB, 46.88 MB, 39.90 MB, 32.89 MB, 51.87 MB, 46.89 MB, 39.90 MB, 32.87 MB, 35.87 MB, 30.89 MB, 30.91 MB, 42.90 MB, 43.40 MB, 43.37 MB, 37.35 MB, 51.87 MB, 42.89 MB, 35.85 MB, 31.82 MB, 43.87 MB, 34.90 MB, 42.90 MB, 42.83 MB, 42.88 MB, 35.88 MB, 37.71 MB, 37.68 MB, 30.69 MB, 23.66 MB, 19.19 MB, 15.81 MB, 14.73 MB ThreadUsage(500.000ms): 1, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2 - AverageThreadTokens: 2.89 - BloomFilterBytes: 0 - PeakMemoryUsage: 59.82 MB (62723550) - PerHostPeakMemUsage: 1.49 GB (1605006452) - RowsProduced: 18.09M (18085110) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 13s342ms - TotalStorageWaitTime: 482.000ms - TotalThreadsInvoluntaryContextSwitches: 10 (10) - TotalThreadsTotalWallClockTime: 1m4s - TotalThreadsSysTime: 708.799ms - TotalThreadsUserTime: 9s290ms - TotalThreadsVoluntaryContextSwitches: 27.00K (27003) Fragment Instance Lifecycle Timings: - ExecTime: 21s975ms - ExecTreeExecTime: 207.000ms - OpenTime: 25.000ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 23.000ms - ExecTreePrepareTime: 0.000ns DataStreamSender (dst_id=5):(Total: 5s305ms, non-child: 5s305ms, % non-child: 100.00%) - BytesSent: 593.94 MB (622794709) - NetworkThroughput(*): 459.35 MB/sec - OverallThroughput: 111.96 MB/sec - PeakMemoryUsage: 151.21 KB (154840) - RowsReturned: 18.09M (18085110) - SerializeBatchTime: 4s004ms - TransmitDataRPCTime: 1s293ms - UncompressedRowBatchSize: 1.28 GB (1369086587) CodeGen:(Total: 39.000ms, non-child: 39.000ms, % non-child: 100.00%) - CodegenTime: 1.000ms - CompileTime: 6.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 5 (5) - NumInstructions: 98 (98) - OptimizationTime: 10.000ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 22.000ms HDFS_SCAN_NODE (id=1):(Total: 195.000ms, non-child: 195.000ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:9/1.94 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 9 out of 9 Hdfs Read Thread Concurrency Bucket: 0:95.35% 1:4.651% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:18 BytesRead(500.000ms): 0, 36.38 MB, 36.38 MB, 36.38 MB, 36.38 MB, 36.38 MB, 57.01 MB, 76.66 MB, 76.66 MB, 76.66 MB, 76.66 MB, 97.29 MB, 116.94 MB, 116.94 MB, 116.94 MB, 116.94 MB, 157.22 MB, 157.22 MB, 157.22 MB, 157.22 MB, 179.86 MB, 179.86 MB, 179.86 MB, 203.08 MB, 217.41 MB, 217.41 MB, 217.41 MB, 244.93 MB, 249.33 MB, 249.33 MB, 256.80 MB, 274.24 MB, 274.24 MB, 292.15 MB, 292.15 MB, 292.15 MB, 292.15 MB, 324.96 MB, 324.96 MB, 324.96 MB, 324.96 MB, 324.96 MB, 343.97 MB, 353.67 MB - FooterProcessingTime: (Avg: 6.333ms ; Min: 2.000ms ; Max: 25.000ms ; Number of samples: 9) - AverageHdfsReadThreadConcurrency: 0.05 - AverageScannerThreadConcurrency: 1.93 - BytesRead: 353.67 MB (370846613) - BytesReadDataNodeCache: 0 - BytesReadLocal: 353.67 MB (370846613) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 353.67 MB (370846613) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDisksAccessed: 2 (2) - NumRowGroups: 28 (28) - NumScannerThreadsStarted: 2 (2) - PeakMemoryUsage: 55.70 MB (58406924) - PerReadThreadRawHdfsThroughput: 1.76 GB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 84.000ms - RowBatchQueuePutWaitTime: 40s651ms - RowsRead: 18.09M (18085110) - RowsReturned: 18.09M (18085110) - RowsReturnedRate: 92.74 M/sec - ScanRangesComplete: 9 (9) - ScannerThreadsInvoluntaryContextSwitches: 0 (0) - ScannerThreadsTotalWallClockTime: 42s023ms - MaterializeTupleTime(*): 778.000ms - ScannerThreadsSysTime: 118.880ms - ScannerThreadsUserTime: 982.527ms - ScannerThreadsVoluntaryContextSwitches: 17.70K (17695) - TotalRawHdfsReadTime(*): 196.000ms - TotalReadThroughput: 16.08 MB/sec Instance 1344bea35ad1b2d9:8 (host=host3.local:22000):(Total: 19s441ms, non-child: 13s346ms, % non-child: 68.65%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:7/2.13 GB MemoryUsage(500.000ms): 38.85 MB, 46.87 MB, 46.90 MB, 32.87 MB, 48.85 MB, 44.88 MB, 44.88 MB, 30.84 MB, 35.86 MB, 42.90 MB, 42.87 MB, 39.85 MB, 39.91 MB, 39.91 MB, 39.87 MB, 39.90 MB, 39.90 MB, 27.89 MB, 24.63 MB, 42.88 MB, 42.89 MB, 28.87 MB, 28.90 MB, 34.89 MB, 34.90 MB, 39.90 MB, 39.90 MB, 31.87 MB, 31.88 MB, 50.86 MB, 50.88 MB, 36.88 MB, 36.88 MB, 47.87 MB, 46.87 MB, 47.88 MB, 32.90 MB, 26.83 MB, 8.80 MB ThreadUsage(500.000ms): 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2 - AverageThreadTokens: 2.95 - BloomFilterBytes: 0 - PeakMemoryUsage: 67.84 MB (71136734) - PerHostPeakMemUsage: 1.41 GB (1519184195) - RowsProduced: 18.68M (18681910) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 10s034ms - TotalStorageWaitTime: 667.999ms - TotalThreadsInvoluntaryContextSwitches: 6 (6) - TotalThreadsTotalWallClockTime: 57s057ms - TotalThreadsSysTime: 733.527ms - TotalThreadsUserTime: 10s101ms - TotalThreadsVoluntaryContextSwitches: 31.84K (31838) Fragment Instance Lifecycle Timings: - ExecTime: 19s399ms - ExecTreeExecTime: 280.999ms - OpenTime: 20.000ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 22.000ms - ExecTreePrepareTime: 0.000ns DataStreamSender (dst_id=5):(Total: 5s801ms, non-child: 5s801ms, % non-child: 100.00%) - BytesSent: 613.83 MB (643647363) - NetworkThroughput(*): 436.89 MB/sec - OverallThroughput: 105.80 MB/sec - PeakMemoryUsage: 151.21 KB (154840) - RowsReturned: 18.68M (18681910) - SerializeBatchTime: 4s387ms - TransmitDataRPCTime: 1s404ms - UncompressedRowBatchSize: 1.32 GB (1419484526) CodeGen:(Total: 39.000ms, non-child: 39.000ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 5.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 5 (5) - NumInstructions: 98 (98) - OptimizationTime: 12.000ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 21.000ms HDFS_SCAN_NODE (id=1):(Total: 253.999ms, non-child: 253.999ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:7/2.13 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7 Hdfs Read Thread Concurrency Bucket: 0:97.44% 1:2.564% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:14 BytesRead(500.000ms): 34.61 MB, 39.25 MB, 39.25 MB, 39.25 MB, 59.51 MB, 74.44 MB, 74.44 MB, 74.44 MB, 86.00 MB, 106.34 MB, 106.34 MB, 125.39 MB, 125.39 MB, 145.38 MB, 145.38 MB, 164.33 MB, 164.33 MB, 171.10 MB, 171.52 MB, 209.75 MB, 209.75 MB, 209.75 MB, 209.75 MB, 238.56 MB, 238.56 MB, 258.98 MB, 258.98 MB, 267.49 MB, 267.49 MB, 309.69 MB, 309.69 MB, 309.69 MB, 309.69 MB, 335.45 MB, 347.89 MB, 347.89 MB, 347.89 MB, 368.32 MB, 372.63 MB - FooterProcessingTime: (Avg: 14.571ms ; Min: 2.999ms ; Max: 41.000ms ; Number of samples: 7) - AverageHdfsReadThreadConcurrency: 0.03 - AverageScannerThreadConcurrency: 1.95 - BytesRead: 374.28 MB (392459331) - BytesReadDataNodeCache: 0 - BytesReadLocal: 374.28 MB (392459331) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 374.28 MB (392459331) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDisksAccessed: 2 (2) - NumRowGroups: 30 (30) - NumScannerThreadsStarted: 2 (2) - PeakMemoryUsage: 59.70 MB (62601228) - PerReadThreadRawHdfsThroughput: 1.41 GB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 124.000ms - RowBatchQueuePutWaitTime: 36s006ms - RowsRead: 18.68M (18681910) - RowsReturned: 18.68M (18681910) - RowsReturnedRate: 73.55 M/sec - ScanRangesComplete: 7 (7) - ScannerThreadsInvoluntaryContextSwitches: 2 (2) - ScannerThreadsTotalWallClockTime: 37s637ms - MaterializeTupleTime(*): 852.999ms - ScannerThreadsSysTime: 125.413ms - ScannerThreadsUserTime: 1s091ms - ScannerThreadsVoluntaryContextSwitches: 18.27K (18265) - TotalRawHdfsReadTime(*): 258.999ms - TotalReadThroughput: 19.10 MB/sec Instance 1344bea35ad1b2d9:6 (host=host1.local:22000):(Total: 17s839ms, non-child: 11s543ms, % non-child: 64.71%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:7/2.25 GB MemoryUsage(500.000ms): 8.17 MB, 42.84 MB, 42.88 MB, 28.87 MB, 39.83 MB, 34.83 MB, 39.86 MB, 39.88 MB, 35.90 MB, 39.86 MB, 39.89 MB, 39.86 MB, 39.88 MB, 39.87 MB, 38.90 MB, 28.63 MB, 46.87 MB, 46.90 MB, 32.88 MB, 32.90 MB, 38.87 MB, 38.89 MB, 39.87 MB, 42.87 MB, 35.88 MB, 35.82 MB, 35.87 MB, 39.87 MB, 51.88 MB, 39.89 MB, 27.83 MB, 27.87 MB, 22.82 MB, 15.82 MB, 22.69 MB, 14.83 MB ThreadUsage(500.000ms): 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2 - AverageThreadTokens: 2.89 - BloomFilterBytes: 0 - PeakMemoryUsage: 64.90 MB (68049386) - PerHostPeakMemUsage: 1.37 GB (1466654423) - RowsProduced: 19.87M (19869844) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 7s630ms - TotalStorageWaitTime: 680.000ms - TotalThreadsInvoluntaryContextSwitches: 15 (15) - TotalThreadsTotalWallClockTime: 51s274ms - TotalThreadsSysTime: 879.717ms - TotalThreadsUserTime: 10s668ms - TotalThreadsVoluntaryContextSwitches: 30.18K (30181) Fragment Instance Lifecycle Timings: - ExecTime: 17s796ms - ExecTreeExecTime: 258.000ms - OpenTime: 18.000ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 25.000ms - ExecTreePrepareTime: 0.000ns DataStreamSender (dst_id=5):(Total: 6s017ms, non-child: 6s017ms, % non-child: 100.00%) - BytesSent: 652.13 MB (683811415) - NetworkThroughput(*): 482.35 MB/sec - OverallThroughput: 108.38 MB/sec - PeakMemoryUsage: 151.21 KB (154840) - RowsReturned: 19.87M (19869844) - SerializeBatchTime: 4s662ms - TransmitDataRPCTime: 1s352ms - UncompressedRowBatchSize: 1.41 GB (1509381742) CodeGen:(Total: 40.000ms, non-child: 40.000ms, % non-child: 100.00%) - CodegenTime: 1.000ms - CompileTime: 6.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 5 (5) - NumInstructions: 98 (98) - OptimizationTime: 10.000ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 23.000ms HDFS_SCAN_NODE (id=1):(Total: 239.000ms, non-child: 239.000ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:7/2.25 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 7 out of 7 Hdfs Read Thread Concurrency Bucket: 0:97.22% 1:2.778% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:14 BytesRead(500.000ms): 200.00 KB, 38.59 MB, 38.59 MB, 38.59 MB, 58.85 MB, 65.40 MB, 84.45 MB, 84.45 MB, 103.10 MB, 123.83 MB, 123.83 MB, 142.48 MB, 142.48 MB, 163.21 MB, 173.09 MB, 173.56 MB, 215.29 MB, 215.29 MB, 215.29 MB, 215.29 MB, 245.58 MB, 245.58 MB, 264.80 MB, 283.42 MB, 283.42 MB, 302.35 MB, 302.35 MB, 322.78 MB, 341.61 MB, 341.61 MB, 350.22 MB, 350.22 MB, 370.45 MB, 370.45 MB, 389.28 MB, 396.24 MB - FooterProcessingTime: (Avg: 6.857ms ; Min: 2.000ms ; Max: 14.000ms ; Number of samples: 7) - AverageHdfsReadThreadConcurrency: 0.03 - AverageScannerThreadConcurrency: 1.89 - BytesRead: 397.69 MB (417005514) - BytesReadDataNodeCache: 0 - BytesReadLocal: 397.59 MB (416903114) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 397.59 MB (416903114) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDisksAccessed: 2 (2) - NumRowGroups: 36 (36) - NumScannerThreadsStarted: 2 (2) - PeakMemoryUsage: 64.70 MB (67844120) - PerReadThreadRawHdfsThroughput: 1.40 GB/sec - RemoteScanRanges: 1 (1) - RowBatchQueueGetWaitTime: 94.000ms - RowBatchQueuePutWaitTime: 31s769ms - RowsRead: 19.87M (19869844) - RowsReturned: 19.87M (19869844) - RowsReturnedRate: 83.14 M/sec - ScanRangesComplete: 7 (7) - ScannerThreadsInvoluntaryContextSwitches: 0 (0) - ScannerThreadsTotalWallClockTime: 33s460ms - MaterializeTupleTime(*): 893.000ms - ScannerThreadsSysTime: 142.512ms - ScannerThreadsUserTime: 1s089ms - ScannerThreadsVoluntaryContextSwitches: 19.39K (19385) - TotalRawHdfsReadTime(*): 277.000ms - TotalReadThroughput: 22.01 MB/sec Instance 1344bea35ad1b2d9:7 (host=host2.local:22000):(Total: 17s661ms, non-child: 11s326ms, % non-child: 64.13%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:5/2.14 GB MemoryUsage(500.000ms): 8.18 MB, 46.86 MB, 46.87 MB, 32.83 MB, 33.90 MB, 42.89 MB, 35.90 MB, 28.87 MB, 50.87 MB, 51.87 MB, 36.91 MB, 39.87 MB, 46.88 MB, 39.84 MB, 32.86 MB, 31.90 MB, 35.61 MB, 46.89 MB, 39.88 MB, 30.90 MB, 17.41 MB, 42.88 MB, 42.88 MB, 28.90 MB, 47.87 MB, 42.79 MB, 35.83 MB, 28.87 MB, 46.90 MB, 46.83 MB, 39.91 MB, 35.83 MB, 38.90 MB, 38.90 MB, 31.85 MB, 3.75 MB ThreadUsage(500.000ms): 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2 - AverageThreadTokens: 2.97 - BloomFilterBytes: 0 - PeakMemoryUsage: 63.83 MB (66934238) - PerHostPeakMemUsage: 1.39 GB (1494954663) - RowsProduced: 19.03M (19033993) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 7s542ms - TotalStorageWaitTime: 731.999ms - TotalThreadsInvoluntaryContextSwitches: 14 (14) - TotalThreadsTotalWallClockTime: 52s430ms - TotalThreadsSysTime: 946.483ms - TotalThreadsUserTime: 10s730ms - TotalThreadsVoluntaryContextSwitches: 29.28K (29284) Fragment Instance Lifecycle Timings: - ExecTime: 17s622ms - ExecTreeExecTime: 325.999ms - OpenTime: 16.999ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 21.999ms - ExecTreePrepareTime: 0.000ns DataStreamSender (dst_id=5):(Total: 5s989ms, non-child: 5s989ms, % non-child: 100.00%) - BytesSent: 625.65 MB (656042942) - NetworkThroughput(*): 433.28 MB/sec - OverallThroughput: 104.45 MB/sec - PeakMemoryUsage: 151.21 KB (154840) - RowsReturned: 19.03M (19033993) - SerializeBatchTime: 4s538ms - TransmitDataRPCTime: 1s443ms - UncompressedRowBatchSize: 1.34 GB (1442057958) CodeGen:(Total: 36.999ms, non-child: 36.999ms, % non-child: 100.00%) - CodegenTime: 999.998us - CompileTime: 5.999ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 5 (5) - NumInstructions: 98 (98) - OptimizationTime: 8.999ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 21.999ms HDFS_SCAN_NODE (id=1):(Total: 307.999ms, non-child: 307.999ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:5/2.14 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 5 out of 5 Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:10 BytesRead(500.000ms): 8.20 MB, 39.31 MB, 39.31 MB, 39.31 MB, 62.17 MB, 80.89 MB, 80.89 MB, 80.89 MB, 121.75 MB, 121.75 MB, 121.75 MB, 140.46 MB, 160.89 MB, 160.89 MB, 160.89 MB, 170.89 MB, 191.87 MB, 212.22 MB, 212.22 MB, 217.43 MB, 218.28 MB, 257.94 MB, 257.94 MB, 257.94 MB, 296.21 MB, 296.21 MB, 296.21 MB, 296.21 MB, 336.11 MB, 336.11 MB, 336.11 MB, 354.74 MB, 366.95 MB, 366.95 MB, 366.95 MB, 377.14 MB - FooterProcessingTime: (Avg: 13.999ms ; Min: 3.000ms ; Max: 27.999ms ; Number of samples: 5) - AverageHdfsReadThreadConcurrency: 0.00 - AverageScannerThreadConcurrency: 1.97 - BytesRead: 377.14 MB (395458842) - BytesReadDataNodeCache: 0 - BytesReadLocal: 376.94 MB (395254042) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 376.94 MB (395254042) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDisksAccessed: 2 (2) - NumRowGroups: 33 (33) - NumScannerThreadsStarted: 2 (2) - PeakMemoryUsage: 59.70 MB (62601228) - PerReadThreadRawHdfsThroughput: 1.27 GB/sec - RemoteScanRanges: 2 (2) - RowBatchQueueGetWaitTime: 124.999ms - RowBatchQueuePutWaitTime: 33s033ms - RowsRead: 19.03M (19033993) - RowsReturned: 19.03M (19033993) - RowsReturnedRate: 61.80 M/sec - ScanRangesComplete: 5 (5) - ScannerThreadsInvoluntaryContextSwitches: 2 (2) - ScannerThreadsTotalWallClockTime: 34s790ms - MaterializeTupleTime(*): 913.999ms - ScannerThreadsSysTime: 167.428ms - ScannerThreadsUserTime: 1s145ms - ScannerThreadsVoluntaryContextSwitches: 18.58K (18584) - TotalRawHdfsReadTime(*): 288.999ms - TotalReadThroughput: 20.95 MB/sec Instance 1344bea35ad1b2d9:9 (host=host4.local:22000):(Total: 17s660ms, non-child: 11s501ms, % non-child: 65.12%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/2.29 GB MemoryUsage(500.000ms): 26.90 MB, 42.86 MB, 42.88 MB, 28.91 MB, 42.86 MB, 42.90 MB, 28.83 MB, 42.87 MB, 42.84 MB, 28.80 MB, 35.87 MB, 42.87 MB, 35.83 MB, 21.91 MB, 34.88 MB, 23.90 MB, 35.87 MB, 46.87 MB, 39.91 MB, 39.90 MB, 39.91 MB, 39.87 MB, 39.88 MB, 39.87 MB, 46.90 MB, 39.83 MB, 39.84 MB, 39.89 MB, 31.86 MB, 31.90 MB, 38.86 MB, 46.90 MB, 39.90 MB, 28.87 MB, 20.78 MB ThreadUsage(500.000ms): 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2 - AverageThreadTokens: 2.97 - BloomFilterBytes: 0 - PeakMemoryUsage: 63.82 MB (66917854) - PerHostPeakMemUsage: 1.38 GB (1485469257) - RowsProduced: 20.61M (20612057) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 7s776ms - TotalStorageWaitTime: 595.000ms - TotalThreadsInvoluntaryContextSwitches: 4 (4) - TotalThreadsTotalWallClockTime: 52s102ms - TotalThreadsSysTime: 659.422ms - TotalThreadsUserTime: 10s668ms - TotalThreadsVoluntaryContextSwitches: 32.05K (32046) Fragment Instance Lifecycle Timings: - ExecTime: 17s625ms - ExecTreeExecTime: 225.000ms - OpenTime: 13.000ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 22.000ms - ExecTreePrepareTime: 0.000ns BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 192 (192) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 1.25 GB (1342177280) - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns DataStreamSender (dst_id=5):(Total: 5s915ms, non-child: 5s915ms, % non-child: 100.00%) - BytesSent: 676.73 MB (709599777) - NetworkThroughput(*): 577.41 MB/sec - OverallThroughput: 114.41 MB/sec - PeakMemoryUsage: 151.21 KB (154840) - RowsReturned: 20.61M (20612057) - SerializeBatchTime: 4s736ms - TransmitDataRPCTime: 1s172ms - UncompressedRowBatchSize: 1.45 GB (1559176528) CodeGen:(Total: 32.000ms, non-child: 32.000ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 4.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 5 (5) - NumInstructions: 98 (98) - OptimizationTime: 7.000ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 20.000ms HDFS_SCAN_NODE (id=1):(Total: 212.000ms, non-child: 212.000ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:6/2.29 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 6 out of 6 Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% File Formats: PARQUET/NONE:12 BytesRead(500.000ms): 23.37 MB, 37.04 MB, 37.04 MB, 37.04 MB, 75.42 MB, 75.42 MB, 75.42 MB, 113.82 MB, 113.82 MB, 113.82 MB, 132.45 MB, 152.22 MB, 152.22 MB, 155.70 MB, 184.73 MB, 185.67 MB, 205.89 MB, 226.52 MB, 226.52 MB, 246.36 MB, 246.36 MB, 266.99 MB, 266.99 MB, 286.91 MB, 307.54 MB, 307.54 MB, 327.46 MB, 327.46 MB, 339.18 MB, 339.18 MB, 367.88 MB, 387.20 MB, 387.20 MB, 390.09 MB, 406.14 MB - FooterProcessingTime: (Avg: 13.000ms ; Min: 2.000ms ; Max: 28.000ms ; Number of samples: 6) - AverageHdfsReadThreadConcurrency: 0.00 - AverageScannerThreadConcurrency: 1.97 - BytesRead: 406.14 MB (425865641) - BytesReadDataNodeCache: 0 - BytesReadLocal: 406.04 MB (425763241) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 406.04 MB (425763241) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDisksAccessed: 2 (2) - NumRowGroups: 32 (32) - NumScannerThreadsStarted: 2 (2) - PeakMemoryUsage: 55.70 MB (58406928) - PerReadThreadRawHdfsThroughput: 1.61 GB/sec - RemoteScanRanges: 1 (1) - RowBatchQueueGetWaitTime: 75.000ms - RowBatchQueuePutWaitTime: 32s945ms - RowsRead: 20.61M (20612057) - RowsReturned: 20.61M (20612057) - RowsReturnedRate: 97.23 M/sec - ScanRangesComplete: 6 (6) - ScannerThreadsInvoluntaryContextSwitches: 1 (1) - ScannerThreadsTotalWallClockTime: 34s464ms - MaterializeTupleTime(*): 810.000ms - ScannerThreadsSysTime: 116.719ms - ScannerThreadsUserTime: 1s019ms - ScannerThreadsVoluntaryContextSwitches: 20.16K (20160) - TotalRawHdfsReadTime(*): 247.000ms - TotalReadThroughput: 23.21 MB/sec

ponypony · ‎08-11-2017

Query (id=1344bea35ad1b2d9:2f42a5a900000000): Summary: Session ID: 445d9ad6c4309aa:2a9b7dab41e4638b Session Type: HIVESERVER2 HiveServer2 Protocol Version: V6 Start Time: 2017-08-11 10:41:09.372124000 End Time: 2017-08-11 10:43:07.479888000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.8.0-cdh5.11.1 RELEASE (build 3382c1c488dff12d5ca8d049d2b59babee605b4e) User: Connected User: Delegated User: Network Address: 10.10.16.11:37100 Default Db: default Sql Statement: select Max(`entry_date_filter`) as `Max_entry_date_filter`, Min(`entry_date_filter`) as `Min_entry_date_filter` from `default`.`trip_txn_data_view_latest` limit 1000 Coordinator: host3.local:22000 Query Options (non default): Plan: ---------------- Estimated Per-Host Requirements: Memory=3.11GB VCores=2 PLAN-ROOT SINK | 07:AGGREGATE [FINALIZE] | output: max:merge(entry_date_filter), min:merge(entry_date_filter) | limit: 1000 | hosts=5 per-host-mem=unavailable | tuple-ids=3 row-size=32B cardinality=1 | 06:EXCHANGE [UNPARTITIONED] | hosts=5 per-host-mem=unavailable | tuple-ids=3 row-size=32B cardinality=1 | 03:AGGREGATE | output: max(CAST(from_utc_timestamp((CAST((entrydate) / 1000 AS TIMESTAMP)), 'America/Los_Angeles') AS TIMESTAMP)), min(CAST(from_utc_timestamp((CAST((entrydate) / 1000 AS TIMESTAMP)), 'America/Los_Angeles') AS TIMESTAMP)) | hosts=5 per-host-mem=10.00MB | tuple-ids=3 row-size=32B cardinality=1 | 02:HASH JOIN [LEFT OUTER JOIN, PARTITIONED] | hash predicates: tx.month = tr.month, tx.year = tr.year, tx.id = tr.transactionid | hosts=5 per-host-mem=1.24GB | tuple-ids=0,1N row-size=118B cardinality=221662594 | |--05:EXCHANGE [HASH(tr.month,tr.year,tr.transactionid)] | | hosts=5 per-host-mem=0B | | tuple-ids=1 row-size=63B cardinality=96282914 | | | 01:SCAN HDFS [default.trip_txn_data_v3_latest tr, RANDOM] | partitions=18/18 files=18 size=10.75GB | table stats: 96282914 rows total | column stats: all | hosts=5 per-host-mem=1.88GB | tuple-ids=1 row-size=63B cardinality=96282914 | 04:EXCHANGE [HASH(tx.month,tx.year,tx.id)] | hosts=5 per-host-mem=0B | tuple-ids=0 row-size=55B cardinality=221662594 | 00:SCAN HDFS [default.txn_data_v3_latest tx, RANDOM] partitions=19/19 files=26 size=9.27GB table stats: 221662594 rows total column stats: all hosts=5 per-host-mem=1.46GB tuple-ids=0 row-size=55B cardinality=221662594 ---------------- Estimated Per-Host Mem: 3343422781 Estimated Per-Host VCores: 2 Request Pool: root.default Admission result: Admitted immediately ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail ------------------------------------------------------------------------------------------------------------------------------ 07:AGGREGATE 1 0.000ns 0.000ns 1 1 36.00 KB -1.00 B FINALIZE 06:EXCHANGE 1 0.000ns 0.000ns 5 1 0 -1.00 B UNPARTITIONED 03:AGGREGATE 5 1m6s 1m9s 5 1 138.74 KB 10.00 MB 02:HASH JOIN 5 24s963ms 25s882ms 221.68M 221.66M 1.88 GB 1.24 GB LEFT OUTER JOIN, PARTITIONED |--05:EXCHANGE 5 834.399ms 946.999ms 96.28M 96.28M 0 0 HASH(tr.month,tr.year,tr.tr... | 01:SCAN HDFS 5 241.599ms 307.999ms 96.28M 96.28M 64.70 MB 1.88 GB default.trip_txn_data_v3_la... 04:EXCHANGE 5 1s602ms 3s163ms 221.66M 221.66M 0 0 HASH(tx.month,tx.year,tx.id) 00:SCAN HDFS 5 369.799ms 448.999ms 221.66M 221.66M 87.37 MB 1.46 GB default.txn_data_v3_latest tx Errors: Planner Timeline: 8.681ms - Analysis finished: 2.719ms (2.719ms) - Equivalence classes computed: 3.614ms (895.594us) - Single node plan created: 5.294ms (1.679ms) - Runtime filters computed: 5.332ms (37.967us) - Distributed plan created: 5.596ms (264.419us) - Lineage info computed: 5.703ms (106.613us) - Planning finished: 8.681ms (2.978ms) Query Timeline: 1m58s - Query submitted: 0.000ns (0.000ns) - Planning finished: 12.000ms (12.000ms) - Submit for admission: 13.000ms (1.000ms) - Completed admission: 13.000ms (0.000ns) - Ready to start 16 fragment instances: 13.000ms (0.000ns) - All 16 fragment instances started: 27.000ms (14.000ms) - Rows available: 1m58s (1m58s) - First row fetched: 1m58s (3.999ms) - Unregister query: 1m58s (4.999ms) - ComputeScanRangeAssignmentTimer: 1.000ms ImpalaServer: - ClientFetchWaitTimer: 6.999ms - RowMaterializationTimer: 1.999ms Execution Profile 1344bea35ad1b2d9:2f42a5a900000000:(Total: 1m58s, non-child: 0.000ns, % non-child: 0.00%) Number of filters: 0 Filter routing table: ID Src. Node Tgt. Node(s) Targets Target type Partition filter Pending (Expected) First arrived Completed Enabled ---------------------------------------------------------------------------------------------------------------------------- Fragment instance start latencies: Count: 16, 25th %-ile: 1ms, 50th %-ile: 2ms, 75th %-ile: 2ms, 90th %-ile: 3ms, 95th %-ile: 4ms, 99.9th %-ile: 11ms Per Node Peak Memory Usage: host5.local:22000(1.97 GB) host4.local:22000(1.98 GB) host1.local:22000(1.99 GB) host2.local:22000(1.99 GB) host3.local:22000(1.98 GB) - FiltersReceived: 0 (0) - FinalizationTimer: 0.000ns Coordinator Fragment F03: Instance 1344bea35ad1b2d9:2f42a5a900000000 (host=host3.local:22000):(Total: 1m58s, non-child: 4.999ms, % non-child: 0.00%) MemoryUsage(2s000ms): 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB, 16.00 KB - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 44.02 KB (45072) - PerHostPeakMemUsage: 1.98 GB (2129589388) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 1m58s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 1m58s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 407.000us - TotalThreadsVoluntaryContextSwitches: 11 (11) Fragment Instance Lifecycle Timings: - ExecTime: 3.999ms - ExecTreeExecTime: 0.000ns - OpenTime: 1m58s - ExecTreeOpenTime: 1m58s - PrepareTime: 23.000ms - ExecTreePrepareTime: 0.000ns BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 224 (224) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 1.88 GB (2013265920) - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 23.000ms, non-child: 23.000ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 23.000ms AGGREGATION_NODE (id=7):(Total: 1m58s, non-child: 0.000ns, % non-child: 0.00%) - BuildTime: 0.000ns - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 36.00 KB (36864) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) EXCHANGE_NODE (id=6):(Total: 1m58s, non-child: 0.000ns, % non-child: 0.00%) BytesReceived(2s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 205.00 B (205) - ConvertRowBatchTime: 0.000ns - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 1m58s - PeakMemoryUsage: 0 - RowsReturned: 5 (5) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Averaged Fragment F02:(Total: 1m58s, non-child: 10.399ms, % non-child: 0.01%) split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:1m58s max:1m58s mean: 1m58s stddev:3.059ms execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 5 - AverageThreadTokens: 1.20 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.91 GB (2050393692) - PerHostPeakMemUsage: 1.98 GB (2127547096) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 24s825ms - TotalNetworkSendTime: 200.000us - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 1.51K (1510) - TotalThreadsTotalWallClockTime: 2m21s - TotalThreadsSysTime: 3s778ms - TotalThreadsUserTime: 1m30s - TotalThreadsVoluntaryContextSwitches: 74.19K (74190) Fragment Instance Lifecycle Timings: - ExecTime: 400.000us - ExecTreeExecTime: 0.000ns - OpenTime: 1m58s - ExecTreeOpenTime: 1m57s - PrepareTime: 35.399ms - ExecTreePrepareTime: 400.000us DataStreamSender (dst_id=6): - BytesSent: 41.00 B (41) - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 3.88 KB (3968) - RowsReturned: 1 (1) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 41.00 B (41) CodeGen:(Total: 870.999ms, non-child: 870.999ms, % non-child: 100.00%) - CodegenTime: 11.799ms - CompileTime: 269.799ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 194 (194) - NumInstructions: 5.44K (5436) - OptimizationTime: 570.399ms - PeakMemoryUsage: 2.65 MB (2783232) - PrepareTime: 30.599ms AGGREGATION_NODE (id=3):(Total: 1m57s, non-child: 1m6s, % non-child: 56.58%) - BuildTime: 1m6s - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 120.15 KB (123032) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HASH_JOIN_NODE (id=2):(Total: 50s883ms, non-child: 24s963ms, % non-child: 49.06%) - BuildRows: 19.26M (19256582) - BuildTime: 5s557ms - PeakMemoryUsage: 1.88 GB (2023966464) - ProbeRows: 44.33M (44332518) - ProbeRowsPartitioned: 0 (0) - ProbeTime: 19s151ms - RowsReturned: 44.34M (44335091) - RowsReturnedRate: 873.52 K/sec Hash Join Builder (join_node_id=2): - BuildRowsPartitionTime: 3s082ms - BuildRowsPartitioned: 19.26M (19256582) - GetNewBlockTime: 2.599ms - HashBuckets: 33.55M (33554432) - HashCollisions: 423.18K (423183) - HashTablesBuildTime: 2s467ms - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 1.88 GB (2023816320) - PinTime: 0.000ns - RepartitionTime: 0.000ns - SpilledPartitions: 0 (0) - UnpinTime: 0.000ns EXCHANGE_NODE (id=5):(Total: 17s964ms, non-child: 17s964ms, % non-child: 100.00%) - BytesReceived: 632.46 MB (663179241) - ConvertRowBatchTime: 310.799ms - DeserializeRowBatchTimer: 2s019ms - FirstBatchArrivalWaitTime: 0.000ns - PeakMemoryUsage: 0 - RowsReturned: 19.26M (19256582) - RowsReturnedRate: 1.07 M/sec - SendersBlockedTimer: 103.799ms - SendersBlockedTotalTimer(*): 302.999ms EXCHANGE_NODE (id=4):(Total: 7s955ms, non-child: 7s955ms, % non-child: 100.00%) - BytesReceived: 1.18 GB (1272166982) - ConvertRowBatchTime: 633.799ms - DeserializeRowBatchTimer: 4s128ms - FirstBatchArrivalWaitTime: 0.000ns - PeakMemoryUsage: 0 - RowsReturned: 44.33M (44332518) - RowsReturnedRate: 6.73 M/sec - SendersBlockedTimer: 37s365ms - SendersBlockedTotalTimer(*): 2m10s BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 224 (224) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 1.88 GB (2013265920) - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns Fragment F02: Instance 1344bea35ad1b2d9:d (host=host3.local:22000):(Total: 1m58s, non-child: 9.000ms, % non-child: 0.01%) MemoryUsage(2s000ms): 96.43 MB, 258.39 MB, 418.37 MB, 578.35 MB, 706.36 MB, 866.38 MB, 994.39 MB, 1.12 GB, 1.25 GB, 1.31 GB, 1.41 GB, 1.61 GB, 1.90 GB, 1.90 GB, 1.90 GB, 1.90 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.90 GB, 1.90 GB, 1.90 GB, 1.90 GB, 1.90 GB, 1.89 GB, 1.89 GB ThreadUsage(2s000ms): 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.20 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.91 GB (2050419405) - PerHostPeakMemUsage: 1.98 GB (2129589388) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 21s922ms - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 2.60K (2605) - TotalThreadsTotalWallClockTime: 2m21s - TotalThreadsSysTime: 3s572ms - TotalThreadsUserTime: 1m32s - TotalThreadsVoluntaryContextSwitches: 62.47K (62472) Fragment Instance Lifecycle Timings: - ExecTime: 0.000ns - ExecTreeExecTime: 0.000ns - OpenTime: 1m58s - ExecTreeOpenTime: 1m57s - PrepareTime: 36.000ms - ExecTreePrepareTime: 1.000ms DataStreamSender (dst_id=6): - BytesSent: 41.00 B (41) - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 3.88 KB (3968) - RowsReturned: 1 (1) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 41.00 B (41) CodeGen:(Total: 900.999ms, non-child: 900.999ms, % non-child: 100.00%) - CodegenTime: 10.000ms - CompileTime: 272.999ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 194 (194) - NumInstructions: 5.44K (5436) - OptimizationTime: 598.999ms - PeakMemoryUsage: 2.65 MB (2783232) - PrepareTime: 29.000ms AGGREGATION_NODE (id=3):(Total: 1m57s, non-child: 1m8s, % non-child: 58.43%) ExecOption: Codegen Enabled - BuildTime: 1m8s - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 115.50 KB (118272) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HASH_JOIN_NODE (id=2):(Total: 48s703ms, non-child: 25s681ms, % non-child: 52.73%) ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously - BuildRows: 19.25M (19253770) - BuildTime: 5s711ms - PeakMemoryUsage: 1.88 GB (2023966464) - ProbeRows: 44.33M (44325501) - ProbeRowsPartitioned: 0 (0) - ProbeTime: 19s703ms - RowsReturned: 44.33M (44328225) - RowsReturnedRate: 910.16 K/sec Hash Join Builder (join_node_id=2): ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled - BuildRowsPartitionTime: 3s156ms - BuildRowsPartitioned: 19.25M (19253770) - GetNewBlockTime: 4.999ms - HashBuckets: 33.55M (33554432) - HashCollisions: 422.10K (422097) - HashTablesBuildTime: 2s545ms - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 1.88 GB (2023816320) - PinTime: 0.000ns - RepartitionTime: 0.000ns - SpilledPartitions: 0 (0) - UnpinTime: 0.000ns EXCHANGE_NODE (id=5):(Total: 17s839ms, non-child: 946.999ms, % non-child: 5.31%) BytesReceived(2s000ms): 21.97 MB, 89.47 MB, 160.19 MB, 229.13 MB, 298.53 MB, 366.45 MB, 435.76 MB, 504.74 MB, 567.08 MB, 609.79 MB, 627.95 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB, 632.37 MB - BytesReceived: 632.37 MB (663091605) - ConvertRowBatchTime: 295.999ms - DeserializeRowBatchTimer: 2s023ms - FirstBatchArrivalWaitTime: 0.000ns - PeakMemoryUsage: 0 - RowsReturned: 19.25M (19253770) - RowsReturnedRate: 1.08 M/sec - SendersBlockedTimer: 136.999ms - SendersBlockedTotalTimer(*): 383.999ms EXCHANGE_NODE (id=4):(Total: 5s181ms, non-child: 1s302ms, % non-child: 25.14%) BytesReceived(2s000ms): 8.18 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 16.64 MB, 41.14 MB, 66.11 MB, 91.45 MB, 116.54 MB, 142.08 MB, 167.35 MB, 192.64 MB, 218.32 MB, 244.40 MB, 269.60 MB, 295.05 MB, 321.02 MB, 346.35 MB, 371.90 MB, 396.40 MB, 421.00 MB, 445.92 MB, 470.36 MB, 495.43 MB, 521.11 MB, 548.23 MB, 575.92 MB, 604.08 MB, 632.32 MB, 660.46 MB, 687.63 MB, 716.77 MB, 745.20 MB, 769.42 MB, 793.88 MB, 819.26 MB, 845.22 MB, 870.57 MB, 896.54 MB, 922.68 MB, 948.62 MB, 975.46 MB, 1003.28 MB, 1.01 GB, 1.04 GB, 1.06 GB, 1.09 GB, 1.12 GB, 1.14 GB, 1.16 GB, 1.18 GB - BytesReceived: 1.18 GB (1271952757) - ConvertRowBatchTime: 654.999ms - DeserializeRowBatchTimer: 4s238ms - FirstBatchArrivalWaitTime: 0.000ns - PeakMemoryUsage: 0 - RowsReturned: 44.33M (44325501) - RowsReturnedRate: 8.55 M/sec - SendersBlockedTimer: 23s822ms - SendersBlockedTotalTimer(*): 1m35s Instance 1344bea35ad1b2d9:b (host=host2.local:22000):(Total: 1m58s, non-child: 9.999ms, % non-child: 0.01%) MemoryUsage(2s000ms): 91.54 MB, 258.35 MB, 418.37 MB, 556.38 MB, 706.36 MB, 834.37 MB, 994.35 MB, 1.10 GB, 1.22 GB, 1.31 GB, 1.41 GB, 1.60 GB, 1.89 GB, 1.90 GB, 1.90 GB, 1.90 GB, 1.90 GB, 1.90 GB, 1.90 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.90 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.89 GB, 1.89 GB ThreadUsage(2s000ms): 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.20 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.91 GB (2050419640) - PerHostPeakMemUsage: 1.99 GB (2131438970) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 20s885ms - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 1.90K (1902) - TotalThreadsTotalWallClockTime: 2m21s - TotalThreadsSysTime: 3s917ms - TotalThreadsUserTime: 1m33s - TotalThreadsVoluntaryContextSwitches: 65.78K (65776) Fragment Instance Lifecycle Timings: - ExecTime: 0.000ns - ExecTreeExecTime: 0.000ns - OpenTime: 1m58s - ExecTreeOpenTime: 1m57s - PrepareTime: 32.999ms - ExecTreePrepareTime: 1.000ms BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 224 (224) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 1.88 GB (2013265920) - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns DataStreamSender (dst_id=6): - BytesSent: 41.00 B (41) - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 3.88 KB (3968) - RowsReturned: 1 (1) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 41.00 B (41) CodeGen:(Total: 861.999ms, non-child: 861.999ms, % non-child: 100.00%) - CodegenTime: 7.999ms - CompileTime: 304.999ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 194 (194) - NumInstructions: 5.44K (5436) - OptimizationTime: 525.999ms - PeakMemoryUsage: 2.65 MB (2783232) - PrepareTime: 30.999ms AGGREGATION_NODE (id=3):(Total: 1m57s, non-child: 1m9s, % non-child: 59.11%) ExecOption: Codegen Enabled - BuildTime: 1m9s - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 115.50 KB (118272) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HASH_JOIN_NODE (id=2):(Total: 47s928ms, non-child: 25s882ms, % non-child: 54.00%) ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously - BuildRows: 19.25M (19252092) - BuildTime: 5s674ms - PeakMemoryUsage: 1.88 GB (2023966464) - ProbeRows: 44.33M (44328023) - ProbeRowsPartitioned: 0 (0) - ProbeTime: 19s951ms - RowsReturned: 44.33M (44330541) - RowsReturnedRate: 924.92 K/sec Hash Join Builder (join_node_id=2): ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled - BuildRowsPartitionTime: 3s240ms - BuildRowsPartitioned: 19.25M (19252092) - GetNewBlockTime: 2.000ms - HashBuckets: 33.55M (33554432) - HashCollisions: 424.50K (424499) - HashTablesBuildTime: 2s429ms - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 1.88 GB (2023816320) - PinTime: 0.000ns - RepartitionTime: 0.000ns - SpilledPartitions: 0 (0) - UnpinTime: 0.000ns EXCHANGE_NODE (id=5):(Total: 17s817ms, non-child: 876.999ms, % non-child: 4.92%) BytesReceived(2s000ms): 17.90 MB, 83.71 MB, 154.78 MB, 223.24 MB, 293.07 MB, 360.79 MB, 430.14 MB, 499.56 MB, 562.39 MB, 607.86 MB, 626.51 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB, 632.30 MB - BytesReceived: 632.30 MB (663017696) - ConvertRowBatchTime: 344.999ms - DeserializeRowBatchTimer: 2s092ms - FirstBatchArrivalWaitTime: 0.000ns - PeakMemoryUsage: 0 - RowsReturned: 19.25M (19252092) - RowsReturnedRate: 1.08 M/sec - SendersBlockedTimer: 98.000ms - SendersBlockedTotalTimer(*): 197.000ms EXCHANGE_NODE (id=4):(Total: 4s227ms, non-child: 1s294ms, % non-child: 30.63%) BytesReceived(2s000ms): 7.13 MB, 10.05 MB, 10.05 MB, 10.05 MB, 10.05 MB, 10.05 MB, 10.05 MB, 10.05 MB, 10.05 MB, 10.05 MB, 10.05 MB, 10.05 MB, 15.10 MB, 38.94 MB, 64.11 MB, 89.05 MB, 114.61 MB, 140.09 MB, 165.16 MB, 190.64 MB, 216.42 MB, 242.21 MB, 267.56 MB, 292.84 MB, 318.63 MB, 344.24 MB, 369.78 MB, 394.39 MB, 419.00 MB, 443.95 MB, 468.44 MB, 493.15 MB, 518.78 MB, 545.55 MB, 573.39 MB, 601.71 MB, 629.94 MB, 658.02 MB, 685.39 MB, 714.25 MB, 742.64 MB, 767.45 MB, 791.84 MB, 817.04 MB, 843.03 MB, 868.55 MB, 894.43 MB, 920.64 MB, 946.56 MB, 973.46 MB, 1000.88 MB, 1.01 GB, 1.03 GB, 1.06 GB, 1.09 GB, 1.11 GB, 1.14 GB, 1.16 GB, 1.18 GB - BytesReceived: 1.18 GB (1272036210) - ConvertRowBatchTime: 653.999ms - DeserializeRowBatchTimer: 4s238ms - FirstBatchArrivalWaitTime: 0.000ns - PeakMemoryUsage: 0 - RowsReturned: 44.33M (44328023) - RowsReturnedRate: 10.48 M/sec - SendersBlockedTimer: 24s278ms - SendersBlockedTotalTimer(*): 1m59s Instance 1344bea35ad1b2d9:c (host=host1.local:22000):(Total: 1m58s, non-child: 8.000ms, % non-child: 0.01%) MemoryUsage(2s000ms): 90.97 MB, 258.41 MB, 418.36 MB, 546.35 MB, 706.38 MB, 834.35 MB, 994.35 MB, 1.10 GB, 1.22 GB, 1.30 GB, 1.41 GB, 1.58 GB, 1.89 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.91 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB ThreadUsage(2s000ms): 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.20 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.91 GB (2050466828) - PerHostPeakMemUsage: 1.99 GB (2136802350) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 22s999ms - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 598 (598) - TotalThreadsTotalWallClockTime: 2m21s - TotalThreadsSysTime: 4s245ms - TotalThreadsUserTime: 1m32s - TotalThreadsVoluntaryContextSwitches: 85.10K (85102) Fragment Instance Lifecycle Timings: - ExecTime: 0.000ns - ExecTreeExecTime: 0.000ns - OpenTime: 1m58s - ExecTreeOpenTime: 1m57s - PrepareTime: 38.000ms - ExecTreePrepareTime: 0.000ns BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 224 (224) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 1.88 GB (2013265920) - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns DataStreamSender (dst_id=6): - BytesSent: 41.00 B (41) - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 3.88 KB (3968) - RowsReturned: 1 (1) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 41.00 B (41) CodeGen:(Total: 875.000ms, non-child: 875.000ms, % non-child: 100.00%) - CodegenTime: 11.000ms - CompileTime: 280.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 194 (194) - NumInstructions: 5.44K (5436) - OptimizationTime: 563.000ms - PeakMemoryUsage: 2.65 MB (2783232) - PrepareTime: 31.000ms AGGREGATION_NODE (id=3):(Total: 1m57s, non-child: 1m7s, % non-child: 57.34%) ExecOption: Codegen Enabled - BuildTime: 1m7s - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 115.50 KB (118272) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HASH_JOIN_NODE (id=2):(Total: 49s995ms, non-child: 25s845ms, % non-child: 51.70%) ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously - BuildRows: 19.25M (19254820) - BuildTime: 5s601ms - PeakMemoryUsage: 1.88 GB (2023966464) - ProbeRows: 44.33M (44332859) - ProbeRowsPartitioned: 0 (0) - ProbeTime: 19s986ms - RowsReturned: 44.34M (44335352) - RowsReturnedRate: 886.79 K/sec Hash Join Builder (join_node_id=2): ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled - BuildRowsPartitionTime: 3s000ms - BuildRowsPartitioned: 19.25M (19254820) - GetNewBlockTime: 1.000ms - HashBuckets: 33.55M (33554432) - HashCollisions: 421.95K (421950) - HashTablesBuildTime: 2s594ms - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 1.88 GB (2023816320) - PinTime: 0.000ns - RepartitionTime: 0.000ns - SpilledPartitions: 0 (0) - UnpinTime: 0.000ns EXCHANGE_NODE (id=5):(Total: 18s042ms, non-child: 792.000ms, % non-child: 4.39%) BytesReceived(2s000ms): 16.92 MB, 82.52 MB, 153.59 MB, 222.05 MB, 291.92 MB, 359.66 MB, 429.11 MB, 498.42 MB, 561.31 MB, 607.38 MB, 626.23 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB, 632.40 MB - BytesReceived: 632.40 MB (663121543) - ConvertRowBatchTime: 317.000ms - DeserializeRowBatchTimer: 1s966ms - FirstBatchArrivalWaitTime: 0.000ns - PeakMemoryUsage: 0 - RowsReturned: 19.25M (19254820) - RowsReturnedRate: 1.07 M/sec - SendersBlockedTimer: 76.000ms - SendersBlockedTotalTimer(*): 76.000ms EXCHANGE_NODE (id=4):(Total: 6s108ms, non-child: 3s163ms, % non-child: 51.78%) BytesReceived(2s000ms): 6.83 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 10.04 MB, 14.78 MB, 38.42 MB, 63.62 MB, 88.56 MB, 114.11 MB, 139.69 MB, 164.73 MB, 190.08 MB, 216.02 MB, 241.82 MB, 267.05 MB, 292.42 MB, 318.16 MB, 343.83 MB, 369.21 MB, 393.92 MB, 418.46 MB, 443.37 MB, 467.99 MB, 492.46 MB, 518.23 MB, 544.95 MB, 572.86 MB, 601.05 MB, 629.32 MB, 657.36 MB, 684.72 MB, 713.50 MB, 742.03 MB, 766.84 MB, 791.07 MB, 816.35 MB, 842.19 MB, 867.87 MB, 893.60 MB, 919.76 MB, 945.90 MB, 972.62 MB, 1000.11 MB, 1.00 GB, 1.03 GB, 1.06 GB, 1.09 GB, 1.11 GB, 1.14 GB, 1.16 GB, 1.17 GB - BytesReceived: 1.18 GB (1272206154) - ConvertRowBatchTime: 674.000ms - DeserializeRowBatchTimer: 4s194ms - FirstBatchArrivalWaitTime: 0.000ns - PeakMemoryUsage: 0 - RowsReturned: 44.33M (44332859) - RowsReturnedRate: 7.26 M/sec - SendersBlockedTimer: 1m31s - SendersBlockedTotalTimer(*): 6m7s Instance 1344bea35ad1b2d9:f (host=host5.local:22000):(Total: 1m58s, non-child: 12.000ms, % non-child: 0.01%) MemoryUsage(2s000ms): 88.88 MB, 258.36 MB, 386.36 MB, 546.36 MB, 706.35 MB, 834.39 MB, 976.40 MB, 1.10 GB, 1.22 GB, 1.29 GB, 1.41 GB, 1.59 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB ThreadUsage(2s000ms): 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.20 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.91 GB (2050355330) - PerHostPeakMemUsage: 1.97 GB (2113493539) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 28s420ms - TotalNetworkSendTime: 1.000ms - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 1.25K (1245) - TotalThreadsTotalWallClockTime: 2m21s - TotalThreadsSysTime: 3s682ms - TotalThreadsUserTime: 1m27s - TotalThreadsVoluntaryContextSwitches: 79.00K (78998) Fragment Instance Lifecycle Timings: - ExecTime: 1.000ms - ExecTreeExecTime: 0.000ns - OpenTime: 1m58s - ExecTreeOpenTime: 1m57s - PrepareTime: 39.000ms - ExecTreePrepareTime: 0.000ns BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 224 (224) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 1.88 GB (2013265920) - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns DataStreamSender (dst_id=6): - BytesSent: 41.00 B (41) - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 3.88 KB (3968) - RowsReturned: 1 (1) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 41.00 B (41) CodeGen:(Total: 827.000ms, non-child: 827.000ms, % non-child: 100.00%) - CodegenTime: 20.000ms - CompileTime: 251.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 194 (194) - NumInstructions: 5.44K (5436) - OptimizationTime: 540.000ms - PeakMemoryUsage: 2.65 MB (2783232) - PrepareTime: 36.000ms AGGREGATION_NODE (id=3):(Total: 1m57s, non-child: 1m4s, % non-child: 54.63%) ExecOption: Codegen Enabled - BuildTime: 1m3s - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 115.50 KB (118272) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HASH_JOIN_NODE (id=2):(Total: 53s188ms, non-child: 23s702ms, % non-child: 44.56%) ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously - BuildRows: 19.26M (19260139) - BuildTime: 5s171ms - PeakMemoryUsage: 1.88 GB (2023966464) - ProbeRows: 44.34M (44335596) - ProbeRowsPartitioned: 0 (0) - ProbeTime: 18s277ms - RowsReturned: 44.34M (44338243) - RowsReturnedRate: 833.61 K/sec Hash Join Builder (join_node_id=2): ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled - BuildRowsPartitionTime: 2s789ms - BuildRowsPartitioned: 19.26M (19260139) - GetNewBlockTime: 3.000ms - HashBuckets: 33.55M (33554432) - HashCollisions: 423.24K (423241) - HashTablesBuildTime: 2s371ms - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 1.88 GB (2023816320) - PinTime: 0.000ns - RepartitionTime: 0.000ns - SpilledPartitions: 0 (0) - UnpinTime: 0.000ns EXCHANGE_NODE (id=5):(Total: 18s305ms, non-child: 802.000ms, % non-child: 4.38%) BytesReceived(2s000ms): 15.27 MB, 80.67 MB, 151.69 MB, 219.97 MB, 289.78 MB, 357.67 MB, 426.97 MB, 496.23 MB, 559.45 MB, 606.74 MB, 625.97 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB, 632.57 MB - BytesReceived: 632.57 MB (663297307) - ConvertRowBatchTime: 294.000ms - DeserializeRowBatchTimer: 2s119ms - FirstBatchArrivalWaitTime: 0.000ns - PeakMemoryUsage: 0 - RowsReturned: 19.26M (19260139) - RowsReturnedRate: 1.05 M/sec - SendersBlockedTimer: 71.000ms - SendersBlockedTotalTimer(*): 228.000ms EXCHANGE_NODE (id=4):(Total: 11s181ms, non-child: 1s202ms, % non-child: 10.75%) BytesReceived(2s000ms): 6.56 MB, 10.02 MB, 10.02 MB, 10.02 MB, 10.02 MB, 10.02 MB, 10.02 MB, 10.02 MB, 10.02 MB, 10.02 MB, 10.02 MB, 10.02 MB, 14.37 MB, 37.69 MB, 62.73 MB, 87.73 MB, 113.42 MB, 138.92 MB, 164.01 MB, 189.27 MB, 215.14 MB, 240.90 MB, 266.08 MB, 291.70 MB, 317.42 MB, 343.07 MB, 368.54 MB, 393.13 MB, 417.88 MB, 442.51 MB, 467.36 MB, 491.80 MB, 517.55 MB, 544.24 MB, 572.22 MB, 600.24 MB, 628.73 MB, 656.76 MB, 684.05 MB, 712.91 MB, 741.50 MB, 766.22 MB, 790.66 MB, 815.75 MB, 841.57 MB, 867.39 MB, 892.93 MB, 919.29 MB, 945.42 MB, 972.03 MB, 999.56 MB, 1.00 GB, 1.03 GB, 1.06 GB, 1.09 GB, 1.11 GB, 1.14 GB, 1.16 GB, 1.17 GB - BytesReceived: 1.18 GB (1272266175) - ConvertRowBatchTime: 627.000ms - DeserializeRowBatchTimer: 4s232ms - FirstBatchArrivalWaitTime: 0.000ns - PeakMemoryUsage: 0 - RowsReturned: 44.34M (44335596) - RowsReturnedRate: 3.97 M/sec - SendersBlockedTimer: 23s643ms - SendersBlockedTotalTimer(*): 47s209ms Instance 1344bea35ad1b2d9:e (host=host4.local:22000):(Total: 1m58s, non-child: 13.000ms, % non-child: 0.01%) MemoryUsage(2s000ms): 92.86 MB, 258.36 MB, 418.36 MB, 578.37 MB, 706.39 MB, 834.38 MB, 994.35 MB, 1.10 GB, 1.22 GB, 1.31 GB, 1.41 GB, 1.62 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB, 1.89 GB ThreadUsage(2s000ms): 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1 - AverageThreadTokens: 1.20 - BloomFilterBytes: 0 - PeakMemoryUsage: 1.91 GB (2050307259) - PerHostPeakMemUsage: 1.98 GB (2126411237) - RowsProduced: 1 (1) - TotalNetworkReceiveTime: 29s899ms - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 1.20K (1201) - TotalThreadsTotalWallClockTime: 2m21s - TotalThreadsSysTime: 3s473ms - TotalThreadsUserTime: 1m25s - TotalThreadsVoluntaryContextSwitches: 78.60K (78603) Fragment Instance Lifecycle Timings: - ExecTime: 1.000ms - ExecTreeExecTime: 0.000ns - OpenTime: 1m58s - ExecTreeOpenTime: 1m57s - PrepareTime: 31.000ms - ExecTreePrepareTime: 0.000ns DataStreamSender (dst_id=6): - BytesSent: 41.00 B (41) - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 3.88 KB (3968) - RowsReturned: 1 (1) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 41.00 B (41) CodeGen:(Total: 890.000ms, non-child: 890.000ms, % non-child: 100.00%) - CodegenTime: 10.000ms - CompileTime: 240.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 194 (194) - NumInstructions: 5.44K (5436) - OptimizationTime: 624.000ms - PeakMemoryUsage: 2.65 MB (2783232) - PrepareTime: 26.000ms AGGREGATION_NODE (id=3):(Total: 1m57s, non-child: 1m2s, % non-child: 53.40%) ExecOption: Codegen Enabled - BuildTime: 1m2s - GetResultsTime: 0.000ns - HTResizeTime: 0.000ns - HashBuckets: 0 (0) - LargestPartitionPercent: 0 (0) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 0 (0) - PeakMemoryUsage: 138.74 KB (142074) - RowsRepartitioned: 0 (0) - RowsReturned: 1 (1) - RowsReturnedRate: 0 - SpilledPartitions: 0 (0) HASH_JOIN_NODE (id=2):(Total: 54s601ms, non-child: 23s705ms, % non-child: 43.41%) ExecOption: Probe Side Codegen Enabled, Join Build-Side Prepared Asynchronously - BuildRows: 19.26M (19262093) - BuildTime: 5s630ms - PeakMemoryUsage: 1.88 GB (2023966464) - ProbeRows: 44.34M (44340615) - ProbeRowsPartitioned: 0 (0) - ProbeTime: 17s837ms - RowsReturned: 44.34M (44343095) - RowsReturnedRate: 812.13 K/sec Hash Join Builder (join_node_id=2): ExecOption: Build Side Codegen Enabled, Hash Table Construction Codegen Enabled - BuildRowsPartitionTime: 3s227ms - BuildRowsPartitioned: 19.26M (19262093) - GetNewBlockTime: 2.000ms - HashBuckets: 33.55M (33554432) - HashCollisions: 424.13K (424129) - HashTablesBuildTime: 2s397ms - LargestPartitionPercent: 6 (6) - MaxPartitionLevel: 0 (0) - NumRepartitions: 0 (0) - PartitionsCreated: 16 (16) - PeakMemoryUsage: 1.88 GB (2023816320) - PinTime: 0.000ns - RepartitionTime: 0.000ns - SpilledPartitions: 0 (0) - UnpinTime: 0.000ns EXCHANGE_NODE (id=5):(Total: 17s816ms, non-child: 754.000ms, % non-child: 4.23%) BytesReceived(2s000ms): 19.07 MB, 85.45 MB, 156.54 MB, 225.12 MB, 294.75 MB, 362.43 MB, 431.91 MB, 501.29 MB, 564.01 MB, 608.66 MB, 627.17 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB, 632.64 MB - BytesReceived: 632.64 MB (663368055) - ConvertRowBatchTime: 302.000ms - DeserializeRowBatchTimer: 1s895ms - FirstBatchArrivalWaitTime: 0.000ns - PeakMemoryUsage: 0 - RowsReturned: 19.26M (19262093) - RowsReturnedRate: 1.08 M/sec - SendersBlockedTimer: 137.000ms - SendersBlockedTotalTimer(*): 630.000ms EXCHANGE_NODE (id=4):(Total: 13s080ms, non-child: 1s052ms, % non-child: 8.04%) BytesReceived(2s000ms): 7.41 MB, 10.01 MB, 10.01 MB, 10.01 MB, 10.01 MB, 10.01 MB, 10.01 MB, 10.01 MB, 10.01 MB, 10.01 MB, 10.01 MB, 10.01 MB, 15.46 MB, 39.47 MB, 64.67 MB, 89.61 MB, 115.15 MB, 140.71 MB, 165.89 MB, 191.33 MB, 217.09 MB, 243.03 MB, 268.46 MB, 293.68 MB, 319.50 MB, 345.11 MB, 370.68 MB, 395.29 MB, 419.88 MB, 444.84 MB, 469.31 MB, 494.18 MB, 519.75 MB, 546.63 MB, 574.48 MB, 602.57 MB, 630.83 MB, 658.99 MB, 686.21 MB, 715.14 MB, 743.73 MB, 768.41 MB, 792.68 MB, 817.87 MB, 843.94 MB, 869.39 MB, 895.31 MB, 921.56 MB, 947.43 MB, 974.39 MB, 1001.82 MB, 1.01 GB, 1.04 GB, 1.06 GB, 1.09 GB, 1.11 GB, 1.14 GB, 1.16 GB, 1.18 GB - BytesReceived: 1.18 GB (1272373617) - ConvertRowBatchTime: 559.000ms - DeserializeRowBatchTimer: 3s738ms - FirstBatchArrivalWaitTime: 0.000ns - PeakMemoryUsage: 0 - RowsReturned: 44.34M (44340615) - RowsReturnedRate: 3.39 M/sec - SendersBlockedTimer: 23s664ms - SendersBlockedTotalTimer(*): 23s683ms Averaged Fragment F00:(Total: 1m49s, non-child: 1m37s, % non-child: 88.87%) split sizes: min: 1.62 GB, max: 2.10 GB, avg: 1.85 GB, stddev: 209.45 MB completion times: min:1m39s max:1m58s mean: 1m49s stddev:7s801ms execution rates: min:15.09 MB/sec max:18.89 MB/sec mean:17.34 MB/sec stddev:1.39 MB/sec num instances: 5 - AverageThreadTokens: 2.94 - BloomFilterBytes: 0 - PeakMemoryUsage: 84.72 MB (88831493) - PerHostPeakMemUsage: 1.98 GB (2127547096) - RowsProduced: 44.33M (44332518) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 1m29s - TotalStorageWaitTime: 256.999ms - TotalThreadsInvoluntaryContextSwitches: 22 (22) - TotalThreadsTotalWallClockTime: 5m21s - TotalThreadsSysTime: 1s663ms - TotalThreadsUserTime: 22s052ms - TotalThreadsVoluntaryContextSwitches: 69.18K (69179) Fragment Instance Lifecycle Timings: - ExecTime: 1m49s - ExecTreeExecTime: 414.399ms - OpenTime: 18.799ms - ExecTreeOpenTime: 200.000us - PrepareTime: 21.999ms - ExecTreePrepareTime: 0.000ns DataStreamSender (dst_id=4):(Total: 11s775ms, non-child: 11s775ms, % non-child: 100.00%) - BytesSent: 1.18 GB (1272166982) - NetworkThroughput(*): 401.05 MB/sec - OverallThroughput: 103.56 MB/sec - PeakMemoryUsage: 281.05 KB (287792) - RowsReturned: 44.33M (44332518) - SerializeBatchTime: 8s714ms - TransmitDataRPCTime: 3s051ms - UncompressedRowBatchSize: 2.80 GB (3006022241) CodeGen:(Total: 38.799ms, non-child: 38.799ms, % non-child: 100.00%) - CodegenTime: 200.000us - CompileTime: 5.199ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 5 (5) - NumInstructions: 98 (98) - OptimizationTime: 10.799ms - PeakMemoryUsage: 49.00 KB (50176) - PrepareTime: 22.199ms HDFS_SCAN_NODE (id=0):(Total: 369.799ms, non-child: 369.799ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.00 - AverageScannerThreadConcurrency: 1.94 - BytesRead: 541.70 MB (568013343) - BytesReadDataNodeCache: 0 - BytesReadLocal: 541.62 MB (567931423) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 541.62 MB (567931423) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 1 (1) - NumDisksAccessed: 2 (2) - NumRowGroups: 21 (21) - NumScannerThreadsStarted: 2 (2) - PeakMemoryUsage: 82.97 MB (87003562) - PerReadThreadRawHdfsThroughput: 1.84 GB/sec - RemoteScanRanges: 0 (0) - RowBatchQueueGetWaitTime: 40.599ms - RowBatchQueuePutWaitTime: 3m30s - RowsRead: 44.33M (44332518) - RowsReturned: 44.33M (44332518) - RowsReturnedRate: 121.37 M/sec - ScanRangesComplete: 7 (7) - ScannerThreadsInvoluntaryContextSwitches: 0 (0) - ScannerThreadsTotalWallClockTime: 3m32s - MaterializeTupleTime(*): 1s505ms - ScannerThreadsSysTime: 321.218ms - ScannerThreadsUserTime: 2s059ms - ScannerThreadsVoluntaryContextSwitches: 42.96K (42961) - TotalRawHdfsReadTime(*): 290.199ms - TotalReadThroughput: 4.94 MB/sec

ponypony · ‎08-11-2017

Posted here, thank you for helping out. ----------------------------------------------------------- Query (id=d9451b2a703b10ea:7b8ec0b200000000): Summary: Session ID: 3c45d712aca91b57:cdd15e04d5ca5997 Session Type: HIVESERVER2 HiveServer2 Protocol Version: V6 Start Time: 2017-08-11 13:03:23.398808000 End Time: 2017-08-11 13:05:37.772410000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.8.0-cdh5.11.1 RELEASE (build 3382c1c488dff12d5ca8d049d2b59babee605b4e) User: root Connected User: root Delegated User: Network Address: 10.10.16.7:47208 Default Db: default Sql Statement: select id from image_review_image_data_v3 v where transactionid ='304641412' Coordinator: host.local:22000 Query Options (non default): QUERY_TIMEOUT_S=600 Plan: ---------------- Estimated Per-Host Requirements: Memory=176.00MB VCores=1 PLAN-ROOT SINK | 01:EXCHANGE [UNPARTITIONED] | hosts=5 per-host-mem=unavailable | tuple-ids=0 row-size=50B cardinality=28 | 00:SCAN HDFS [default.image_review_image_data_v3 v, RANDOM] partitions=18/18 files=138 size=520.24GB predicates: transactionid = '304641412' table stats: 2961307241 rows total column stats: all hosts=5 per-host-mem=176.00MB tuple-ids=0 row-size=50B cardinality=28 ---------------- Estimated Per-Host Mem: 184549376 Estimated Per-Host VCores: 1 Request Pool: root.root Admission result: Admitted immediately ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail ---------------------------------------------------------------------------------------------------------------------- 01:EXCHANGE 1 0.000ns 0.000ns 24 28 0 -1.00 B UNPARTITIONED 00:SCAN HDFS 5 1m56s 2m4s 24 28 61.42 MB 176.00 MB default.image_review_image_... Errors: Planner Timeline: 3.336ms - Analysis finished: 655.820us (655.820us) - Equivalence classes computed: 735.594us (79.774us) - Single node plan created: 1.969ms (1.234ms) - Runtime filters computed: 2.068ms (98.686us) - Distributed plan created: 2.155ms (87.078us) - Lineage info computed: 2.201ms (46.370us) - Planning finished: 3.336ms (1.134ms) Query Timeline: 2m14s - Query submitted: 0.000ns (0.000ns) - Planning finished: 5s431ms (5s431ms) - Submit for admission: 5s435ms (4.000ms) - Completed admission: 5s435ms (0.000ns) - Ready to start 6 fragment instances: 5s437ms (2.000ms) - All 6 fragment instances started: 5s447ms (10.000ms) - Rows available: 11s084ms (5s637ms) - First row fetched: 12s195ms (1s111ms) - Unregister query: 2m14s (2m2s) - ComputeScanRangeAssignmentTimer: 2.000ms ImpalaServer: - ClientFetchWaitTimer: 1s658ms - RowMaterializationTimer: 2m1s Execution Profile d9451b2a703b10ea:7b8ec0b200000000:(Total: 2m7s, non-child: 0.000ns, % non-child: 0.00%) Number of filters: 0 Filter routing table: ID Src. Node Tgt. Node(s) Targets Target type Partition filter Pending (Expected) First arrived Completed Enabled ---------------------------------------------------------------------------------------------------------------------------- Fragment instance start latencies: Count: 6, 25th %-ile: 2ms, 50th %-ile: 2ms, 75th %-ile: 4ms, 90th %-ile: 4ms, 95th %-ile: 8ms, 99.9th %-ile: 8ms Per Node Peak Memory Usage: host2.local:22000(61.43 MB) host5.local.local:22000(62.37 MB) host4.local.local:22000(56.81 MB) host3.local:22000(60.35 MB) host.local:22000(64.13 MB) - FiltersReceived: 0 (0) - FinalizationTimer: 0.000ns Coordinator Fragment F01: Instance d9451b2a703b10ea:7b8ec0b200000000 (host=host.local:22000):(Total: 2m8s, non-child: 1.000ms, % non-child: 0.00%) MemoryUsage(4s000ms): 8.00 KB, 10.58 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 12.12 KB, 13.60 KB, 16.06 KB, 16.06 KB, 16.06 KB, 16.06 KB, 16.06 KB, 16.06 KB, 16.06 KB - AverageThreadTokens: 0.00 - BloomFilterBytes: 0 - PeakMemoryUsage: 16.19 KB (16576) - PerHostPeakMemUsage: 64.13 MB (67250313) - RowsProduced: 24 (24) - TotalNetworkReceiveTime: 2m8s - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 0.000ns - TotalThreadsInvoluntaryContextSwitches: 0 (0) - TotalThreadsTotalWallClockTime: 2m8s - TotalThreadsSysTime: 0.000ns - TotalThreadsUserTime: 1.091ms - TotalThreadsVoluntaryContextSwitches: 6 (6) Fragment Instance Lifecycle Timings: - ExecTime: 2m2s - ExecTreeExecTime: 2m2s - OpenTime: 5s626ms - ExecTreeOpenTime: 5s625ms - PrepareTime: 19.000ms - ExecTreePrepareTime: 0.000ns BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 0 - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns PLAN_ROOT_SINK: - PeakMemoryUsage: 0 CodeGen:(Total: 19.000ms, non-child: 19.000ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 0.000ns - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 0 (0) - NumInstructions: 0 (0) - OptimizationTime: 0.000ns - PeakMemoryUsage: 0 - PrepareTime: 19.000ms EXCHANGE_NODE (id=1):(Total: 2m8s, non-child: 0.000ns, % non-child: 0.00%) BytesReceived(4s000ms): 0, 190.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 304.00 B, 366.00 B, 470.00 B, 470.00 B, 470.00 B, 470.00 B, 470.00 B, 470.00 B, 470.00 B - BytesReceived: 470.00 B (470) - ConvertRowBatchTime: 0.000ns - DeserializeRowBatchTimer: 0.000ns - FirstBatchArrivalWaitTime: 5s625ms - PeakMemoryUsage: 0 - RowsReturned: 24 (24) - RowsReturnedRate: 0 - SendersBlockedTimer: 0.000ns - SendersBlockedTotalTimer(*): 0.000ns Averaged Fragment F00:(Total: 2m, non-child: 3s765ms, % non-child: 3.12%) split sizes: min: 85.20 GB, max: 109.11 GB, avg: 104.05 GB, stddev: 9.43 GB completion times: min:1m49s max:2m8s mean: 2m stddev:6s502ms execution rates: min:798.37 MB/sec max:940.70 MB/sec mean:881.14 MB/sec stddev:48.68 MB/sec num instances: 5 - AverageThreadTokens: 8.95 - BloomFilterBytes: 0 - PeakMemoryUsage: 61.02 MB (63980069) - PerHostPeakMemUsage: 61.02 MB (63982552) - RowsProduced: 4 (4) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 15m3s - TotalThreadsInvoluntaryContextSwitches: 68 (68) - TotalThreadsTotalWallClockTime: 18m - TotalThreadsSysTime: 14s479ms - TotalThreadsUserTime: 22s701ms - TotalThreadsVoluntaryContextSwitches: 10.65K (10647) Fragment Instance Lifecycle Timings: - ExecTime: 2m - ExecTreeExecTime: 1m56s - OpenTime: 28.399ms - ExecTreeOpenTime: 200.000us - PrepareTime: 20.199ms - ExecTreePrepareTime: 999.999us DataStreamSender (dst_id=1): - BytesSent: 94.00 B (94) - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 3.88 KB (3968) - RowsReturned: 4 (4) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 265.00 B (265) CodeGen:(Total: 46.399ms, non-child: 46.399ms, % non-child: 100.00%) - CodegenTime: 800.000us - CompileTime: 6.599ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 18 (18) - NumInstructions: 289 (289) - OptimizationTime: 20.399ms - PeakMemoryUsage: 144.50 KB (147968) - PrepareTime: 18.999ms HDFS_SCAN_NODE (id=0):(Total: 1m56s, non-child: 1m56s, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 0.97 - AverageScannerThreadConcurrency: 7.95 - BytesRead: 1.81 GB (1947540891) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.80 GB (1935027176) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 1.80 GB (1935027176) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDisksAccessed: 2 (2) - NumRowGroups: 1.16K (1159) - NumScannerThreadsStarted: 8 (8) - PeakMemoryUsage: 58.13 MB (60950173) - PerReadThreadRawHdfsThroughput: 15.82 MB/sec - RemoteScanRanges: 119 (119) - RowBatchQueueGetWaitTime: 1m56s - RowBatchQueuePutWaitTime: 0.000ns - RowsRead: 592.26M (592261448) - RowsReturned: 4 (4) - RowsReturnedRate: 0 - ScanRangesComplete: 222 (222) - ScannerThreadsInvoluntaryContextSwitches: 62 (62) - ScannerThreadsTotalWallClockTime: 15m59s - MaterializeTupleTime(*): 29s997ms - ScannerThreadsSysTime: 11s399ms - ScannerThreadsUserTime: 21s866ms - ScannerThreadsVoluntaryContextSwitches: 6.90K (6904) - TotalRawHdfsReadTime(*): 1m57s - TotalReadThroughput: 15.34 MB/sec BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 0 - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns Fragment F00: Instance d9451b2a703b10ea:2 (host=host4.local.local:22000):(Total: 2m8s, non-child: 3s584ms, % non-child: 2.79%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:233/107.89 GB MemoryUsage(4s000ms): 28.26 MB, 25.90 MB, 24.97 MB, 30.79 MB, 26.85 MB, 21.63 MB, 28.63 MB, 29.20 MB, 27.14 MB, 23.07 MB, 28.53 MB, 22.50 MB, 27.40 MB, 26.63 MB, 30.06 MB, 30.50 MB, 29.07 MB, 27.06 MB, 28.04 MB, 34.46 MB, 26.72 MB, 33.66 MB, 28.47 MB, 28.68 MB, 31.87 MB, 27.85 MB, 38.58 MB, 33.31 MB, 29.26 MB, 30.06 MB, 31.92 MB, 29.39 MB ThreadUsage(4s000ms): 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8 - AverageThreadTokens: 8.96 - BloomFilterBytes: 0 - PeakMemoryUsage: 56.81 MB (59570649) - PerHostPeakMemUsage: 56.81 MB (59570649) - RowsProduced: 8 (8) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 16m5s - TotalThreadsInvoluntaryContextSwitches: 55 (55) - TotalThreadsTotalWallClockTime: 19m10s - TotalThreadsSysTime: 13s240ms - TotalThreadsUserTime: 21s668ms - TotalThreadsVoluntaryContextSwitches: 10.96K (10957) Fragment Instance Lifecycle Timings: - ExecTime: 2m8s - ExecTreeExecTime: 2m4s - OpenTime: 27.000ms - ExecTreeOpenTime: 1.000ms - PrepareTime: 21.000ms - ExecTreePrepareTime: 1.000ms BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 0 - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns DataStreamSender (dst_id=1): - BytesSent: 166.00 B (166) - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 3.88 KB (3968) - RowsReturned: 8 (8) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 444.00 B (444) CodeGen:(Total: 45.000ms, non-child: 45.000ms, % non-child: 100.00%) - CodegenTime: 1.000ms - CompileTime: 6.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 18 (18) - NumInstructions: 289 (289) - OptimizationTime: 19.000ms - PeakMemoryUsage: 144.50 KB (147968) - PrepareTime: 19.000ms HDFS_SCAN_NODE (id=0):(Total: 2m4s, non-child: 2m4s, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:233/107.89 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 233 out of 233 Hdfs Read Thread Concurrency Bucket: 0:8.171% 1:88.33% 2:3.113% 3:0% 4:0.3891% File Formats: PARQUET/NONE:466 BytesRead(4s000ms): 27.80 MB, 85.14 MB, 142.67 MB, 207.51 MB, 265.22 MB, 318.69 MB, 375.42 MB, 427.20 MB, 488.50 MB, 538.53 MB, 589.01 MB, 643.03 MB, 694.78 MB, 745.85 MB, 799.44 MB, 869.70 MB, 939.54 MB, 1004.95 MB, 1.04 GB, 1.09 GB, 1.15 GB, 1.20 GB, 1.26 GB, 1.30 GB, 1.36 GB, 1.42 GB, 1.48 GB, 1.54 GB, 1.59 GB, 1.66 GB, 1.71 GB, 1.78 GB - FooterProcessingTime: (Avg: 335.622ms ; Min: 2.000ms ; Max: 1s834ms ; Number of samples: 233) - AverageHdfsReadThreadConcurrency: 0.96 - AverageScannerThreadConcurrency: 7.96 - BytesRead: 1.81 GB (1943785374) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.79 GB (1920996399) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 1.79 GB (1920996399) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDisksAccessed: 2 (2) - NumRowGroups: 1.21K (1207) - NumScannerThreadsStarted: 8 (8) - PeakMemoryUsage: 54.80 MB (57460818) - PerReadThreadRawHdfsThroughput: 14.82 MB/sec - RemoteScanRanges: 217 (217) - RowBatchQueueGetWaitTime: 2m4s - RowBatchQueuePutWaitTime: 0.000ns - RowsRead: 610.88M (610884328) - RowsReturned: 8 (8) - RowsReturnedRate: 0 - ScanRangesComplete: 233 (233) - ScannerThreadsInvoluntaryContextSwitches: 48 (48) - ScannerThreadsTotalWallClockTime: 17m1s - MaterializeTupleTime(*): 28s089ms - ScannerThreadsSysTime: 10s250ms - ScannerThreadsUserTime: 20s926ms - ScannerThreadsVoluntaryContextSwitches: 7.15K (7147) - TotalRawHdfsReadTime(*): 2m5s - TotalReadThroughput: 14.42 MB/sec Instance d9451b2a703b10ea:3 (host=host.local:22000):(Total: 2m4s, non-child: 3s997ms, % non-child: 3.21%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:232/108.95 GB MemoryUsage(2s000ms): 25.59 MB, 43.03 MB, 24.67 MB, 31.96 MB, 22.78 MB, 26.98 MB, 27.02 MB, 23.24 MB, 34.10 MB, 28.42 MB, 32.10 MB, 21.83 MB, 30.47 MB, 28.60 MB, 22.12 MB, 29.12 MB, 42.88 MB, 21.27 MB, 29.99 MB, 25.76 MB, 25.38 MB, 24.17 MB, 29.60 MB, 28.41 MB, 25.95 MB, 25.93 MB, 24.21 MB, 24.41 MB, 25.95 MB, 29.78 MB, 23.64 MB, 27.39 MB, 31.07 MB, 26.80 MB, 33.48 MB, 36.65 MB, 26.36 MB, 40.59 MB, 26.20 MB, 39.60 MB, 26.63 MB, 34.87 MB, 32.94 MB, 31.29 MB, 33.99 MB, 27.84 MB, 25.46 MB, 22.86 MB, 42.47 MB, 34.66 MB, 28.53 MB, 28.16 MB, 28.88 MB, 34.86 MB, 39.26 MB, 33.16 MB, 24.88 MB, 32.04 MB, 32.25 MB, 36.52 MB, 25.70 MB, 33.52 MB ThreadUsage(2s000ms): 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 7 - AverageThreadTokens: 8.95 - BloomFilterBytes: 0 - PeakMemoryUsage: 64.12 MB (67237897) - PerHostPeakMemUsage: 64.13 MB (67250313) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 15m31s - TotalThreadsInvoluntaryContextSwitches: 73 (73) - TotalThreadsTotalWallClockTime: 18m33s - TotalThreadsSysTime: 15s241ms - TotalThreadsUserTime: 24s078ms - TotalThreadsVoluntaryContextSwitches: 11.52K (11520) Fragment Instance Lifecycle Timings: - ExecTime: 2m4s - ExecTreeExecTime: 2m - OpenTime: 29.000ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 20.000ms - ExecTreePrepareTime: 1.000ms DataStreamSender (dst_id=1): - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 3.88 KB (3968) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 48.000ms, non-child: 48.000ms, % non-child: 100.00%) - CodegenTime: 1.000ms - CompileTime: 6.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 18 (18) - NumInstructions: 289 (289) - OptimizationTime: 22.000ms - PeakMemoryUsage: 144.50 KB (147968) - PrepareTime: 19.000ms HDFS_SCAN_NODE (id=0):(Total: 2m, non-child: 2m, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:232/108.95 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 232 out of 232 Hdfs Read Thread Concurrency Bucket: 0:4.418% 1:92.77% 2:2.008% 3:0.4016% 4:0.4016% File Formats: PARQUET/NONE:464 BytesRead(2s000ms): 11.93 MB, 46.91 MB, 75.47 MB, 104.71 MB, 132.90 MB, 159.49 MB, 195.09 MB, 223.49 MB, 260.53 MB, 293.25 MB, 331.14 MB, 356.24 MB, 380.49 MB, 407.13 MB, 429.02 MB, 450.06 MB, 489.78 MB, 532.49 MB, 562.91 MB, 590.18 MB, 612.37 MB, 642.80 MB, 678.62 MB, 708.21 MB, 734.52 MB, 764.60 MB, 795.61 MB, 833.28 MB, 863.53 MB, 895.09 MB, 928.86 MB, 956.50 MB, 981.44 MB, 1012.94 MB, 1.03 GB, 1.06 GB, 1.10 GB, 1.13 GB, 1.16 GB, 1.20 GB, 1.23 GB, 1.27 GB, 1.30 GB, 1.33 GB, 1.37 GB, 1.40 GB, 1.43 GB, 1.46 GB, 1.49 GB, 1.53 GB, 1.56 GB, 1.58 GB, 1.62 GB, 1.65 GB, 1.69 GB, 1.71 GB, 1.74 GB, 1.77 GB, 1.80 GB, 1.82 GB, 1.86 GB, 1.90 GB - FooterProcessingTime: (Avg: 294.689ms ; Min: 2.000ms ; Max: 1s752ms ; Number of samples: 232) - AverageHdfsReadThreadConcurrency: 1.00 - AverageScannerThreadConcurrency: 7.95 - BytesRead: 1.92 GB (2063038437) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.92 GB (2058780151) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 1.92 GB (2058780151) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDisksAccessed: 2 (2) - NumRowGroups: 1.22K (1222) - NumScannerThreadsStarted: 8 (8) - PeakMemoryUsage: 61.42 MB (64400766) - PerReadThreadRawHdfsThroughput: 16.19 MB/sec - RemoteScanRanges: 41 (41) - RowBatchQueueGetWaitTime: 2m - RowBatchQueuePutWaitTime: 0.000ns - RowsRead: 622.54M (622540363) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 232 (232) - ScannerThreadsInvoluntaryContextSwitches: 67 (67) - ScannerThreadsTotalWallClockTime: 16m29s - MaterializeTupleTime(*): 31s752ms - ScannerThreadsSysTime: 12s047ms - ScannerThreadsUserTime: 23s131ms - ScannerThreadsVoluntaryContextSwitches: 7.40K (7404) - TotalRawHdfsReadTime(*): 2m1s - TotalReadThroughput: 15.80 MB/sec Instance d9451b2a703b10ea:1 (host=host3.local:22000):(Total: 2m2s, non-child: 4s327ms, % non-child: 3.52%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:232/109.11 GB MemoryUsage(2s000ms): 20.27 MB, 36.05 MB, 16.82 MB, 37.59 MB, 30.31 MB, 26.77 MB, 27.86 MB, 33.61 MB, 26.30 MB, 28.51 MB, 33.65 MB, 26.65 MB, 28.53 MB, 22.00 MB, 28.69 MB, 25.45 MB, 23.24 MB, 35.02 MB, 28.50 MB, 25.77 MB, 31.60 MB, 25.22 MB, 23.69 MB, 27.16 MB, 21.06 MB, 26.95 MB, 35.30 MB, 19.66 MB, 31.87 MB, 26.15 MB, 26.14 MB, 27.41 MB, 34.45 MB, 33.38 MB, 32.25 MB, 29.14 MB, 31.75 MB, 21.83 MB, 31.45 MB, 33.07 MB, 31.77 MB, 26.28 MB, 26.96 MB, 32.46 MB, 34.04 MB, 34.71 MB, 33.02 MB, 32.82 MB, 23.38 MB, 36.66 MB, 38.82 MB, 36.48 MB, 28.91 MB, 30.31 MB, 32.37 MB, 28.13 MB, 36.17 MB, 24.15 MB, 27.20 MB, 30.05 MB, 36.03 MB ThreadUsage(2s000ms): 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8 - AverageThreadTokens: 8.96 - BloomFilterBytes: 0 - PeakMemoryUsage: 60.35 MB (63283432) - PerHostPeakMemUsage: 60.35 MB (63283432) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 15m16s - TotalThreadsInvoluntaryContextSwitches: 89 (89) - TotalThreadsTotalWallClockTime: 18m20s - TotalThreadsSysTime: 16s903ms - TotalThreadsUserTime: 25s652ms - TotalThreadsVoluntaryContextSwitches: 10.99K (10989) Fragment Instance Lifecycle Timings: - ExecTime: 2m2s - ExecTreeExecTime: 1m58s - OpenTime: 32.999ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 19.999ms - ExecTreePrepareTime: 999.999us BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 0 - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns DataStreamSender (dst_id=1): - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 3.88 KB (3968) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 49.999ms, non-child: 49.999ms, % non-child: 100.00%) - CodegenTime: 1.000ms - CompileTime: 8.999ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 18 (18) - NumInstructions: 289 (289) - OptimizationTime: 22.999ms - PeakMemoryUsage: 144.50 KB (147968) - PrepareTime: 17.999ms HDFS_SCAN_NODE (id=0):(Total: 1m58s, non-child: 1m58s, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:232/109.11 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 232 out of 232 Hdfs Read Thread Concurrency Bucket: 0:7.317% 1:89.02% 2:3.252% 3:0.4065% 4:0% File Formats: PARQUET/NONE:464 BytesRead(2s000ms): 10.81 MB, 47.09 MB, 69.94 MB, 101.26 MB, 136.16 MB, 163.45 MB, 193.85 MB, 229.89 MB, 270.82 MB, 300.56 MB, 337.38 MB, 371.32 MB, 408.27 MB, 442.79 MB, 476.09 MB, 511.15 MB, 541.57 MB, 578.83 MB, 607.60 MB, 641.31 MB, 683.05 MB, 711.09 MB, 736.45 MB, 771.75 MB, 805.29 MB, 833.93 MB, 861.38 MB, 888.27 MB, 915.50 MB, 946.55 MB, 977.73 MB, 1006.97 MB, 1.02 GB, 1.06 GB, 1.09 GB, 1.12 GB, 1.15 GB, 1.18 GB, 1.21 GB, 1.24 GB, 1.27 GB, 1.30 GB, 1.33 GB, 1.36 GB, 1.39 GB, 1.43 GB, 1.46 GB, 1.50 GB, 1.52 GB, 1.55 GB, 1.59 GB, 1.63 GB, 1.66 GB, 1.69 GB, 1.72 GB, 1.75 GB, 1.79 GB, 1.81 GB, 1.84 GB, 1.87 GB, 1.90 GB - FooterProcessingTime: (Avg: 298.749ms ; Min: 1.999ms ; Max: 1s567ms ; Number of samples: 232) - AverageHdfsReadThreadConcurrency: 0.97 - AverageScannerThreadConcurrency: 7.96 - BytesRead: 1.93 GB (2073824728) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.93 GB (2067526565) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 1.93 GB (2067526565) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDisksAccessed: 2 (2) - NumRowGroups: 1.22K (1216) - NumScannerThreadsStarted: 8 (8) - PeakMemoryUsage: 58.54 MB (61378909) - PerReadThreadRawHdfsThroughput: 16.61 MB/sec - RemoteScanRanges: 60 (60) - RowBatchQueueGetWaitTime: 1m58s - RowBatchQueuePutWaitTime: 0.000ns - RowsRead: 622.14M (622140244) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 232 (232) - ScannerThreadsInvoluntaryContextSwitches: 78 (78) - ScannerThreadsTotalWallClockTime: 16m17s - MaterializeTupleTime(*): 34s311ms - ScannerThreadsSysTime: 13s374ms - ScannerThreadsUserTime: 24s648ms - ScannerThreadsVoluntaryContextSwitches: 7.13K (7127) - TotalRawHdfsReadTime(*): 1m59s - TotalReadThroughput: 16.05 MB/sec Instance d9451b2a703b10ea:5 (host=host2.local:22000):(Total: 1m58s, non-child: 3s923ms, % non-child: 3.30%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:233/109.09 GB MemoryUsage(2s000ms): 26.86 MB, 38.67 MB, 20.91 MB, 23.20 MB, 30.55 MB, 33.23 MB, 32.38 MB, 27.99 MB, 29.41 MB, 27.31 MB, 28.17 MB, 25.18 MB, 26.16 MB, 25.07 MB, 22.63 MB, 32.02 MB, 24.45 MB, 32.23 MB, 26.54 MB, 22.92 MB, 27.61 MB, 29.19 MB, 30.77 MB, 29.01 MB, 31.00 MB, 28.45 MB, 27.06 MB, 25.67 MB, 28.82 MB, 29.03 MB, 32.36 MB, 27.41 MB, 25.43 MB, 37.01 MB, 28.27 MB, 31.88 MB, 29.63 MB, 30.96 MB, 29.69 MB, 37.05 MB, 29.90 MB, 25.39 MB, 31.32 MB, 29.52 MB, 31.78 MB, 31.69 MB, 32.83 MB, 30.29 MB, 28.75 MB, 27.28 MB, 31.85 MB, 34.23 MB, 28.71 MB, 27.12 MB, 30.69 MB, 38.23 MB, 28.43 MB, 30.10 MB, 28.36 MB ThreadUsage(2s000ms): 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 7 - AverageThreadTokens: 8.95 - BloomFilterBytes: 0 - PeakMemoryUsage: 61.43 MB (64412714) - PerHostPeakMemUsage: 61.43 MB (64412714) - RowsProduced: 0 (0) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 14m45s - TotalThreadsInvoluntaryContextSwitches: 64 (64) - TotalThreadsTotalWallClockTime: 17m41s - TotalThreadsSysTime: 15s453ms - TotalThreadsUserTime: 23s275ms - TotalThreadsVoluntaryContextSwitches: 10.98K (10979) Fragment Instance Lifecycle Timings: - ExecTime: 1m58s - ExecTreeExecTime: 1m54s - OpenTime: 25.999ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 19.999ms - ExecTreePrepareTime: 999.998us BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 0 - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns DataStreamSender (dst_id=1): - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 3.88 KB (3968) - RowsReturned: 0 (0) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 0 CodeGen:(Total: 44.999ms, non-child: 44.999ms, % non-child: 100.00%) - CodegenTime: 1.000ms - CompileTime: 5.999ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 18 (18) - NumInstructions: 289 (289) - OptimizationTime: 18.999ms - PeakMemoryUsage: 144.50 KB (147968) - PrepareTime: 19.999ms HDFS_SCAN_NODE (id=0):(Total: 1m54s, non-child: 1m54s, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:233/109.09 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 233 out of 233 Hdfs Read Thread Concurrency Bucket: 0:8.017% 1:88.19% 2:3.376% 3:0% 4:0.4219% File Formats: PARQUET/NONE:466 BytesRead(2s000ms): 15.45 MB, 65.80 MB, 102.98 MB, 133.78 MB, 165.21 MB, 203.47 MB, 240.87 MB, 283.02 MB, 316.20 MB, 349.66 MB, 382.79 MB, 416.64 MB, 448.05 MB, 476.41 MB, 498.51 MB, 533.48 MB, 563.70 MB, 593.07 MB, 615.87 MB, 643.39 MB, 672.48 MB, 700.83 MB, 731.61 MB, 766.05 MB, 802.69 MB, 837.45 MB, 867.75 MB, 894.95 MB, 924.89 MB, 959.66 MB, 993.46 MB, 1.00 GB, 1.04 GB, 1.07 GB, 1.11 GB, 1.14 GB, 1.18 GB, 1.20 GB, 1.23 GB, 1.27 GB, 1.31 GB, 1.33 GB, 1.36 GB, 1.39 GB, 1.43 GB, 1.46 GB, 1.49 GB, 1.52 GB, 1.56 GB, 1.59 GB, 1.62 GB, 1.66 GB, 1.69 GB, 1.72 GB, 1.75 GB, 1.79 GB, 1.83 GB, 1.85 GB, 1.88 GB - FooterProcessingTime: (Avg: 311.635ms ; Min: 1.999ms ; Max: 1s963ms ; Number of samples: 233) - AverageHdfsReadThreadConcurrency: 0.97 - AverageScannerThreadConcurrency: 7.95 - BytesRead: 1.91 GB (2050873759) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.90 GB (2043531704) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 1.90 GB (2043531704) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDisksAccessed: 2 (2) - NumRowGroups: 1.20K (1202) - NumScannerThreadsStarted: 8 (8) - PeakMemoryUsage: 57.17 MB (59946439) - PerReadThreadRawHdfsThroughput: 16.96 MB/sec - RemoteScanRanges: 70 (70) - RowBatchQueueGetWaitTime: 1m54s - RowBatchQueuePutWaitTime: 0.000ns - RowsRead: 621.15M (621148641) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 233 (233) - ScannerThreadsInvoluntaryContextSwitches: 62 (62) - ScannerThreadsTotalWallClockTime: 15m42s - MaterializeTupleTime(*): 31s204ms - ScannerThreadsSysTime: 12s266ms - ScannerThreadsUserTime: 22s397ms - ScannerThreadsVoluntaryContextSwitches: 7.12K (7115) - TotalRawHdfsReadTime(*): 1m55s - TotalReadThroughput: 16.46 MB/sec Instance d9451b2a703b10ea:4 (host=host5.local.local:22000):(Total: 1m49s, non-child: 2s994ms, % non-child: 2.74%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:181/85.20 GB MemoryUsage(2s000ms): 18.21 MB, 38.53 MB, 27.08 MB, 26.53 MB, 32.62 MB, 30.29 MB, 34.66 MB, 29.49 MB, 26.05 MB, 31.58 MB, 25.94 MB, 25.67 MB, 27.96 MB, 24.87 MB, 23.64 MB, 25.64 MB, 32.55 MB, 24.18 MB, 23.78 MB, 24.56 MB, 17.69 MB, 31.04 MB, 36.53 MB, 27.76 MB, 30.91 MB, 29.22 MB, 27.76 MB, 33.72 MB, 31.42 MB, 31.44 MB, 43.47 MB, 25.10 MB, 33.64 MB, 33.70 MB, 31.30 MB, 33.17 MB, 27.18 MB, 28.23 MB, 31.66 MB, 37.66 MB, 33.80 MB, 25.78 MB, 34.78 MB, 31.03 MB, 30.89 MB, 35.60 MB, 25.06 MB, 32.50 MB, 32.21 MB, 29.77 MB, 33.05 MB, 37.72 MB, 26.45 MB, 31.92 MB ThreadUsage(2s000ms): 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 9, 8 - AverageThreadTokens: 8.93 - BloomFilterBytes: 0 - PeakMemoryUsage: 62.37 MB (65395653) - PerHostPeakMemUsage: 62.37 MB (65395653) - RowsProduced: 16 (16) - TotalNetworkReceiveTime: 0.000ns - TotalNetworkSendTime: 0.000ns - TotalStorageWaitTime: 13m38s - TotalThreadsInvoluntaryContextSwitches: 59 (59) - TotalThreadsTotalWallClockTime: 16m15s - TotalThreadsSysTime: 11s556ms - TotalThreadsUserTime: 18s833ms - TotalThreadsVoluntaryContextSwitches: 8.79K (8792) Fragment Instance Lifecycle Timings: - ExecTime: 1m49s - ExecTreeExecTime: 1m46s - OpenTime: 27.000ms - ExecTreeOpenTime: 0.000ns - PrepareTime: 20.000ms - ExecTreePrepareTime: 1.000ms BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 17.60 GB (18897856512) - PeakMemoryUsage: 0 - ScratchFileUsedBytes: 0 - TotalBufferWaitTime: 0.000ns - TotalEncryptionTime: 0.000ns - TotalReadBlockTime: 0.000ns DataStreamSender (dst_id=1): - BytesSent: 304.00 B (304) - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 3.88 KB (3968) - RowsReturned: 16 (16) - SerializeBatchTime: 0.000ns - TransmitDataRPCTime: 0.000ns - UncompressedRowBatchSize: 884.00 B (884) CodeGen:(Total: 44.000ms, non-child: 44.000ms, % non-child: 100.00%) - CodegenTime: 0.000ns - CompileTime: 6.000ms - LoadTime: 0.000ns - ModuleBitcodeSize: 1.97 MB (2070236) - NumFunctions: 18 (18) - NumInstructions: 289 (289) - OptimizationTime: 19.000ms - PeakMemoryUsage: 144.50 KB (147968) - PrepareTime: 19.000ms HDFS_SCAN_NODE (id=0):(Total: 1m46s, non-child: 1m46s, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:181/85.20 GB ExecOption: PARQUET Codegen Enabled, Codegen enabled: 181 out of 181 Hdfs Read Thread Concurrency Bucket: 0:5.936% 1:90.87% 2:2.74% 3:0.4566% 4:0% File Formats: PARQUET/NONE:362 BytesRead(2s000ms): 8.09 MB, 40.45 MB, 69.02 MB, 90.99 MB, 123.35 MB, 149.26 MB, 175.30 MB, 204.01 MB, 226.35 MB, 252.84 MB, 289.11 MB, 315.11 MB, 341.50 MB, 366.53 MB, 388.34 MB, 408.49 MB, 431.86 MB, 458.16 MB, 480.48 MB, 501.06 MB, 521.84 MB, 547.29 MB, 580.90 MB, 605.54 MB, 635.80 MB, 667.92 MB, 703.79 MB, 736.70 MB, 764.60 MB, 798.15 MB, 841.56 MB, 867.56 MB, 889.01 MB, 917.82 MB, 948.27 MB, 982.54 MB, 1009.91 MB, 1.01 GB, 1.04 GB, 1.08 GB, 1.11 GB, 1.14 GB, 1.17 GB, 1.20 GB, 1.22 GB, 1.25 GB, 1.27 GB, 1.29 GB, 1.32 GB, 1.35 GB, 1.37 GB, 1.41 GB, 1.44 GB, 1.47 GB - FooterProcessingTime: (Avg: 352.397ms ; Min: 5.000ms ; Max: 1s916ms ; Number of samples: 181) - AverageHdfsReadThreadConcurrency: 0.98 - AverageScannerThreadConcurrency: 7.93 - BytesRead: 1.50 GB (1606182161) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.48 GB (1584301061) - BytesReadRemoteUnexpected: 0 - BytesReadShortCircuit: 1.48 GB (1584301061) - DecompressionTime: 0.000ns - MaxCompressedTextFileLength: 0 - NumColumns: 2 (2) - NumDisksAccessed: 2 (2) - NumRowGroups: 950 (950) - NumScannerThreadsStarted: 8 (8) - PeakMemoryUsage: 58.71 MB (61563937) - PerReadThreadRawHdfsThroughput: 14.50 MB/sec - RemoteScanRanges: 208 (208) - RowBatchQueueGetWaitTime: 1m46s - RowBatchQueuePutWaitTime: 0.000ns - RowsRead: 484.59M (484593665) - RowsReturned: 16 (16) - RowsReturnedRate: 0 - ScanRangesComplete: 181 (181) - ScannerThreadsInvoluntaryContextSwitches: 57 (57) - ScannerThreadsTotalWallClockTime: 14m26s - MaterializeTupleTime(*): 24s629ms - ScannerThreadsSysTime: 9s060ms - ScannerThreadsUserTime: 18s227ms - ScannerThreadsVoluntaryContextSwitches: 5.73K (5731) - TotalRawHdfsReadTime(*): 1m45s - TotalReadThroughput: 13.98 MB/sec

Online	Offline
Last Visited	‎10-03-2017 12:21 PM

Member Since	‎09-14-2016 10:37 AM
Last Visited	‎10-03-2017 12:21 PM
Posts	116
Kudos received	2

Cloudera Community

Re: Two external tables point to same hdfs locatio...

Re: Failed to load metadata

Failed to load metadata

Insert overwrite partitioned table

Re: Partition, file size and number of files quest...

Re: Partition, file size and number of files quest...

Re: Impala max/min slow

Re: Impala max/min slow

Re: Impala max/min slow

Re: Impala max/min slow

Re: Partition, file size and number of files quest...