Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Impala Query Profile: SendersBlockedTimer

Impala Query Profile: SendersBlockedTimer

Explorer

I am struggling to interpret my Impala Query Profile and would like some assistance please from persons more knowledgable than I am.

 

My understanding is that it took 39s394ms (Source: Query Timeline: 39s394ms) to return all rows back to the client.

 

The coordinator spent 2s544ms writing results to an ascii format that the client can read (Source: RowMaterializationTimer: 2s544ms).

 

Impala was doing nothing for 36s067ms, waiting for the client to retrieve rows that were already available (Source: ClientFetchWaitTimer: 36s067ms).

 

Now here are the items I don't understand:

 

If I take one of the 9 fragments as an example (Instance 8f4fe258fc27f5de:86cdd2f9cdda16b5 (host=myhdpnode04.mydomain:22000)) the stats seem to be telling me:

 

1. The HDFS_SCAN_NODE operation took 439.831ms to read 13.98 MB (14660420) of data
2. The DataStreamSender sent 13.26 MB (13904319) (Source: BytesSent) in 35s723ms (Source: ThriftTransmitTime(*)) to dst_id=1 (the coordinator EXCHANGE_NODE)

 

If I now go to the coordinator fragment:

 

1. Total time spent is 537.213ms

 

Q1: How does this connect up to RowMaterializationTimer: 2s544ms from above?

2. Network receive time of 511.948ms

 

Q2: How does this connect up to ThriftTransmitTime(*) from the DataStreamSender 35s723ms from above?

3. The EXCHANGE_NODE received the bytes from all of the 9 DataStreamSender

 

Q3: What is SendersBlockedTimer: 35s657ms? Is this how long it took to receive all of the data from all 9 DataStreamSender?
Q4: Is it receiving the bytes in chunks? BytesReceived(1s000ms): 5.42 MB, 13.30 MB, 16.38 MB etc. and then passing those onto the coordinator which in turn passes them back to the client?


I'm basically trying to break down the total query time of 39s394m into HDFS scanning time, inter process communication and passing back to the client, but I'm struggling to make sense of what these metrics are showing.

 

Am I missing some key piece of documentation on how to interpret the profile.

 

Thanks in advance,
Steve

3 REPLIES 3

Re: Impala Query Profile: SendersBlockedTimer

Explorer
 Query (id=8f4fe258fc27f5de:86cdd2f9cdda16ae):
  Summary:
    Session ID: 7f4c7f3c34af08d3:ffaa8647743963a1
    Session Type: HIVESERVER2
    HiveServer2 Protocol Version: V6
    Start Time: 2017-01-25 17:53:00.235665000
    End Time: 
    Query Type: QUERY
    Query State: CREATED
    Query Status: OK
    Impala Version: impalad version 2.5.0-cdh5.7.0 RELEASE (build ad3f5adabedf56fe6bd9eea39147c067cc552703)
    Network Address: 10.11.80.4:61535
    Default Db: default
    Sql Statement: SELECT 
	MYUDF_LENGTH(CONCAT('\0\0', MYUDF_LENGTH(NVL(`col1`, ''), 1), '\0\0\0\0\0\0\0\0\0\0', MYUDF_LENGTH(NVL(`col2`, ''), 1), '\0\0\0\0\0\0\0\0\0\0', MYUDF_LENGTH(NVL(`col3`, ''), 1), '\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0', MYUDF_LENGTH(NVL(`col4`, ''), 1), '\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0', MYUDF_DATE(`col5`), '\0\0\0\0\0\0\0\0\0\0', MYUDF_LENGTH(NVL(`col6`, ''), 1), '\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0', MYUDF_LENGTH(NVL(`col7`, ''), 1)), 5) 
FROM 
	`MYDB`.`mytab` 
WHERE (`col5`='2016-10-01 00:00:00.000000000' AND `part_col5`='2016-10-01') AND `col6`='C_RES' AND `col1`='Y' AND `col2`='A'
    Coordinator: myhdpnode05.mydomain:22000
    Query Options (non default): BATCH_SIZE=1024,SYNC_DDL=1
    Plan: 
----------------
Estimated Per-Host Requirements: Memory=504.00MB VCores=1
WARNING: The following tables are missing relevant table and/or column statistics.
mydb.mytab

01:EXCHANGE [UNPARTITIONED]
|  hosts=9 per-host-mem=unavailable
|  tuple-ids=0 row-size=106B cardinality=unavailable
|
00:SCAN HDFS [mydb.mytab, RANDOM]
   partitions=16/6736 files=32 size=1.79GB
   predicates: col5 = '2016-10-01 00:00:00.000000000', col6 = 'C_RES', col1 = 'Y', col2 = 'A'
   table stats: unavailable
   columns missing stats: col1, col2, col3, col4, col5, col6, col7
   hosts=9 per-host-mem=504.00MB
   tuple-ids=0 row-size=106B cardinality=unavailable
----------------
    Estimated Per-Host Mem: 528482304
    Estimated Per-Host VCores: 1
    Tables Missing Stats: mydb.mytab
    Request Pool: default-pool
    Planner Timeline: 23.121ms
       - Analysis finished: 6.242ms (6.242ms)
       - Equivalence classes computed: 6.561ms (319.064us)
       - Single node plan created: 9.494ms (2.933ms)
       - Runtime filters computed: 9.520ms (25.479us)
       - Distributed plan created: 14.885ms (5.365ms)
       - Lineage info computed: 16.396ms (1.510ms)
       - Planning finished: 23.121ms (6.725ms)
    Query Timeline: 39s394ms
       - Start execution: 30.989us (30.989us)
       - Planning finished: 29.209ms (29.178ms)
       - Ready to start 9 remote fragments: 29.964ms (755.205us)
       - All 9 remote fragments started: 33.367ms (3.402ms)
       - Rows available: 483.564ms (450.197ms)
       - First row fetched: 483.630ms (66.052us)
  ImpalaServer:
     - ClientFetchWaitTimer: 36s067ms
     - RowMaterializationTimer: 2s544ms
  Execution Profile 8f4fe258fc27f5de:86cdd2f9cdda16ae:(Total: 563.710ms, non-child: 0.000ns, % non-child: 0.00%)
    Number of filters: 0
    Filter routing table: 
 ID  Src. Node  Tgt. Node  Targets  Type   Partition filter
-----------------------------------------------------------

    Fragment start latencies: Count: 9, 25th %-ile: 2ms, 50th %-ile: 2ms, 75th %-ile: 2ms, 90th %-ile: 2ms, 95th %-ile: 3ms, 99.9th %-ile: 3ms
    Per Node Peak Memory Usage: myhdpnode04.mydomain:22000(45.81 MB) myhdpnode07.mydomain:22000(45.97 MB) myhdpnode09.mydomain:22000(44.58 MB) myhdpnode08.mydomain:22000(37.28 MB) myhdpnodell02.mydomain:22000(37.06 MB) myhdpnode10.mydomain:22000(31.23 MB) myhdpnodell01.mydomain:22000(34.93 MB) myhdpnode06.mydomain:22000(31.80 MB) myhdpnode05.mydomain:22000(69.90 MB) 
     - FiltersReceived: 0 (0)
     - FinalizationTimer: 0.000ns
    Coordinator Fragment F01:(Total: 537.213ms, non-child: 1.602ms, % non-child: 0.30%)
      MemoryUsage(1s000ms): 11.85 MB, 23.57 MB, 23.39 MB, 23.20 MB, 22.86 MB, 22.86 MB, 22.98 MB, 22.98 MB, 23.16 MB, 23.08 MB, 22.99 MB, 22.62 MB, 22.41 MB, 22.32 MB, 22.50 MB, 22.74 MB, 22.70 MB, 22.42 MB, 22.38 MB, 22.63 MB, 22.69 MB, 22.79 MB, 22.62 MB, 22.46 MB, 22.24 MB, 22.39 MB, 22.52 MB, 22.87 MB, 22.79 MB, 22.73 MB, 22.71 MB, 22.91 MB, 22.72 MB, 22.36 MB, 22.15 MB, 22.12 MB, 22.08 MB, 16.32 MB, 10.26 MB
       - AverageThreadTokens: 0.00 
       - BloomFilterBytes: 0
       - PeakMemoryUsage: 23.69 MB (24837264)
       - PerHostPeakMemUsage: 0
       - PrepareTime: 21.404us
       - RowsProduced: 1.43M (1434524)
       - TotalCpuTime: 38s826ms
       - TotalNetworkReceiveTime: 511.948ms
       - TotalNetworkSendTime: 0.000ns
       - TotalStorageWaitTime: 0.000ns
      BlockMgr:
         - BlockWritesOutstanding: 0 (0)
         - BlocksCreated: 0 (0)
         - BlocksRecycled: 0 (0)
         - BufferedPins: 0 (0)
         - BytesWritten: 0
         - MaxBlockSize: 8.00 MB (8388608)
         - MemoryLimit: 19.20 GB (20615843840)
         - PeakMemoryUsage: 0
         - TotalBufferWaitTime: 0.000ns
         - TotalEncryptionTime: 0.000ns
         - TotalIntegrityCheckTime: 0.000ns
         - TotalReadBlockTime: 0.000ns
      EXCHANGE_NODE (id=1):(Total: 535.610ms, non-child: 85.682ms, % non-child: 16.00%)
        BytesReceived(1s000ms): 5.42 MB, 13.30 MB, 16.38 MB, 19.52 MB, 22.74 MB, 26.16 MB, 29.54 MB, 32.97 MB, 36.46 MB, 39.83 MB, 43.17 MB, 46.41 MB, 49.53 MB, 52.88 MB, 56.32 MB, 59.75 MB, 63.33 MB, 66.65 MB, 70.01 MB, 73.43 MB, 76.74 MB, 80.11 MB, 83.40 MB, 86.74 MB, 90.21 MB, 93.75 MB, 97.17 MB, 100.58 MB, 104.08 MB, 107.53 MB, 110.83 MB, 114.19 MB, 117.23 MB, 120.53 MB, 123.78 MB, 127.10 MB, 129.79 MB, 130.11 MB, 130.11 MB
         - BytesReceived: 130.11 MB (136433507)
         - ConvertRowBatchTime: 21.318ms
         - DeserializeRowBatchTimer: 242.065ms
         - FirstBatchArrivalWaitTime: 449.926ms
         - PeakMemoryUsage: 0
         - RowsReturned: 1.43M (1434524)
         - RowsReturnedRate: 2.68 M/sec
         - SendersBlockedTimer: 35s657ms
         - SendersBlockedTotalTimer(*): 4m37s
    Averaged Fragment F00:(Total: 31s377ms, non-child: 0.000ns, % non-child: 0.00%)
      split sizes:  min: 153.01 MB, max: 253.92 MB, avg: 203.40 MB, stddev: 34.34 MB
      completion times: min:21s486ms  max:36s246ms  mean: 31s580ms  stddev:4s518ms
      execution rates: min:4.90 MB/sec  max:7.89 MB/sec  mean:6.47 MB/sec  stddev:821.48 KB/sec
      num instances: 9
       - AverageThreadTokens: 2.73 
       - BloomFilterBytes: 0
       - PeakMemoryUsage: 39.43 MB (41348016)
       - PerHostPeakMemUsage: 42.06 MB (44105068)
       - PrepareTime: 135.206ms
       - RowsProduced: 159.39K (159391)
       - TotalCpuTime: 57s019ms
       - TotalNetworkReceiveTime: 0.000ns
       - TotalNetworkSendTime: 30s985ms
       - TotalStorageWaitTime: 668.399ms
      CodeGen:(Total: 334.638ms, non-child: 334.638ms, % non-child: 100.00%)
         - CodegenTime: 0.000ns
         - CompileTime: 6.389ms
         - LoadTime: 0.000ns
         - ModuleBitcodeSize: 1.85 MB (1939948)
         - OptimizationTime: 194.119ms
         - PrepareTime: 132.984ms
      DataStreamSender (dst_id=1):(Total: 31s022ms, non-child: 31s022ms, % non-child: 100.00%)
         - BytesSent: 14.46 MB (15159278)
         - NetworkThroughput(*): 479.27 KB/sec
         - OverallThroughput: 480.12 KB/sec
         - PeakMemoryUsage: 1.06 KB (1088)
         - SerializeBatchTime: 98.086ms
         - ThriftTransmitTime(*): 31s078ms
         - UncompressedRowBatchSize: 27.64 MB (28980535)
      HDFS_SCAN_NODE (id=0):(Total: 425.236ms, non-child: 425.236ms, % non-child: 100.00%)
         - AverageHdfsReadThreadConcurrency: 0.04 
         - AverageScannerThreadConcurrency: 1.75 
         - BytesRead: 13.09 MB (13725347)
         - BytesReadDataNodeCache: 0
         - BytesReadLocal: 0
         - BytesReadRemoteUnexpected: 0
         - BytesReadShortCircuit: 0
         - DecompressionTime: 22.215ms
         - MaxCompressedTextFileLength: 0
         - NumColumns: 7 (7)
         - NumDisksAccessed: 1 (1)
         - NumRowGroups: 3 (3)
         - NumScannerThreadsStarted: 3 (3)
         - PeakMemoryUsage: 38.88 MB (40773221)
         - PerReadThreadRawHdfsThroughput: 18.49 MB/sec
         - RemoteScanRanges: 28 (28)
         - RowsRead: 473.53K (473529)
         - RowsReturned: 159.39K (159391)
         - RowsReturnedRate: 373.99 K/sec
         - ScanRangesComplete: 3 (3)
         - ScannerThreadsInvoluntaryContextSwitches: 57 (57)
         - ScannerThreadsTotalWallClockTime: 56s988ms
           - MaterializeTupleTime(*): 54s451ms
           - ScannerThreadsSysTime: 2.665ms
           - ScannerThreadsUserTime: 224.631ms
         - ScannerThreadsVoluntaryContextSwitches: 140 (140)
         - TotalRawHdfsReadTime(*): 710.516ms
         - TotalReadThroughput: 429.16 KB/sec
      BlockMgr:
         - BlockWritesOutstanding: 0 (0)
         - BlocksCreated: 0 (0)
         - BlocksRecycled: 0 (0)
         - BufferedPins: 0 (0)
         - BytesWritten: 0
         - MaxBlockSize: 8.00 MB (8388608)
         - MemoryLimit: 19.20 GB (20615843840)
         - PeakMemoryUsage: 0
         - TotalBufferWaitTime: 0.000ns
         - TotalEncryptionTime: 0.000ns
         - TotalIntegrityCheckTime: 0.000ns
         - TotalReadBlockTime: 0.000ns
    Fragment F00:
      Instance 8f4fe258fc27f5de:86cdd2f9cdda16b5 (host=myhdpnode04.mydomain:22000):(Total: 36s044ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:4/216.82 MB 
        MemoryUsage(1s000ms): 22.23 MB, 44.75 MB, 44.93 MB, 45.40 MB, 45.54 MB, 45.69 MB, 45.69 MB, 45.63 MB, 45.34 MB, 45.14 MB, 42.11 MB, 39.13 MB, 38.98 MB, 38.88 MB, 38.84 MB, 36.22 MB, 33.54 MB, 33.67 MB, 33.79 MB, 34.07 MB, 34.20 MB, 34.48 MB, 34.86 MB, 34.98 MB, 35.20 MB, 35.42 MB, 35.73 MB, 35.89 MB, 34.76 MB, 33.47 MB, 33.64 MB, 28.47 MB, 25.31 MB, 19.68 MB, 16.80 MB, 13.64 MB
        ThreadUsage(1s000ms): 3, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 1, 1, 1
         - AverageThreadTokens: 3.49 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 45.81 MB (48037304)
         - PerHostPeakMemUsage: 45.81 MB (48037304)
         - PrepareTime: 140.386ms
         - RowsProduced: 170.17K (170172)
         - TotalCpuTime: 1m31s
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 35s647ms
         - TotalStorageWaitTime: 748.028ms
        BlockMgr:
           - BlockWritesOutstanding: 0 (0)
           - BlocksCreated: 0 (0)
           - BlocksRecycled: 0 (0)
           - BufferedPins: 0 (0)
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 19.20 GB (20615843840)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0.000ns
           - TotalEncryptionTime: 0.000ns
           - TotalIntegrityCheckTime: 0.000ns
           - TotalReadBlockTime: 0.000ns
        CodeGen:(Total: 339.074ms, non-child: 339.074ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 6.328ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.85 MB (1939948)
           - OptimizationTime: 193.462ms
           - PrepareTime: 138.197ms
        DataStreamSender (dst_id=1):(Total: 35s714ms, non-child: 35s714ms, % non-child: 100.00%)
           - BytesSent: 13.26 MB (13904319)
           - NetworkThroughput(*): 380.10 KB/sec
           - OverallThroughput: 380.19 KB/sec
           - PeakMemoryUsage: 1.06 KB (1088)
           - SerializeBatchTime: 92.802ms
           - ThriftTransmitTime(*): 35s723ms
           - UncompressedRowBatchSize: 29.52 MB (30958099)
        HDFS_SCAN_NODE (id=0):(Total: 439.831ms, non-child: 439.831ms, % non-child: 100.00%)
          ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 4
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:4/216.82 MB 
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 
          File Formats: PARQUET/SNAPPY:28 
          BytesRead(1s000ms): 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB, 13.98 MB
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 2.53 
           - BytesRead: 13.98 MB (14660420)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 25.014ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 7 (7)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 4 (4)
           - NumScannerThreadsStarted: 4 (4)
           - PeakMemoryUsage: 45.56 MB (47774072)
           - PerReadThreadRawHdfsThroughput: 18.62 MB/sec
           - RemoteScanRanges: 32 (32)
           - RowsRead: 504.44K (504443)
           - RowsReturned: 170.17K (170172)
           - RowsReturnedRate: 386.90 K/sec
           - ScanRangesComplete: 4 (4)
           - ScannerThreadsInvoluntaryContextSwitches: 45 (45)
           - ScannerThreadsTotalWallClockTime: 1m31s
             - MaterializeTupleTime(*): 1m27s
             - ScannerThreadsSysTime: 7.997ms
             - ScannerThreadsUserTime: 244.961ms
           - ScannerThreadsVoluntaryContextSwitches: 170 (170)
           - TotalRawHdfsReadTime(*): 750.827ms
           - TotalReadThroughput: 397.69 KB/sec
      Instance 8f4fe258fc27f5de:86cdd2f9cdda16b8 (host=myhdpnode05.mydomain:22000):(Total: 35s430ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:4/253.92 MB 
        MemoryUsage(1s000ms): 16.61 MB, 46.11 MB, 43.12 MB, 40.35 MB, 39.85 MB, 39.76 MB, 39.73 MB, 39.57 MB, 39.36 MB, 39.16 MB, 39.10 MB, 39.22 MB, 39.10 MB, 39.22 MB, 39.22 MB, 39.60 MB, 36.82 MB, 37.10 MB, 37.60 MB, 37.97 MB, 35.83 MB, 33.33 MB, 33.52 MB, 33.76 MB, 33.87 MB, 34.00 MB, 33.94 MB, 33.97 MB, 34.00 MB, 31.28 MB, 22.49 MB, 19.47 MB, 15.35 MB, 13.22 MB, 10.66 MB, 7.56 MB
        ThreadUsage(1s000ms): 3, 5, 5, 5, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 3.32 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 46.23 MB (48477192)
         - PerHostPeakMemUsage: 69.90 MB (73290664)
         - PrepareTime: 133.263ms
         - RowsProduced: 199.01K (199014)
         - TotalCpuTime: 1m23s
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 34s992ms
         - TotalStorageWaitTime: 837.125ms
        CodeGen:(Total: 333.181ms, non-child: 333.181ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 6.407ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.85 MB (1939948)
           - OptimizationTime: 194.569ms
           - PrepareTime: 131.031ms
        DataStreamSender (dst_id=1):(Total: 35s101ms, non-child: 35s101ms, % non-child: 100.00%)
           - BytesSent: 18.59 MB (19490715)
           - NetworkThroughput(*): 541.90 KB/sec
           - OverallThroughput: 542.25 KB/sec
           - PeakMemoryUsage: 1.06 KB (1088)
           - SerializeBatchTime: 127.747ms
           - ThriftTransmitTime(*): 35s124ms
           - UncompressedRowBatchSize: 34.47 MB (36148741)
        HDFS_SCAN_NODE (id=0):(Total: 429.520ms, non-child: 429.520ms, % non-child: 100.00%)
          ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 4
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:4/253.92 MB 
          Hdfs Read Thread Concurrency Bucket: 0:98.59% 1:0% 2:0% 3:0% 4:1.408% 5:0% 
          File Formats: PARQUET/SNAPPY:28 
          BytesRead(1s000ms): 13.39 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB, 16.28 MB
           - AverageHdfsReadThreadConcurrency: 0.06 
           - AverageScannerThreadConcurrency: 2.35 
           - BytesRead: 16.28 MB (17073062)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 27.979ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 7 (7)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 4 (4)
           - NumScannerThreadsStarted: 4 (4)
           - PeakMemoryUsage: 45.98 MB (48213960)
           - PerReadThreadRawHdfsThroughput: 20.49 MB/sec
           - RemoteScanRanges: 32 (32)
           - RowsRead: 592.00K (592005)
           - RowsReturned: 199.01K (199014)
           - RowsReturnedRate: 463.34 K/sec
           - ScanRangesComplete: 4 (4)
           - ScannerThreadsInvoluntaryContextSwitches: 40 (40)
           - ScannerThreadsTotalWallClockTime: 1m23s
             - MaterializeTupleTime(*): 1m19s
             - ScannerThreadsSysTime: 1.999ms
             - ScannerThreadsUserTime: 274.956ms
           - ScannerThreadsVoluntaryContextSwitches: 190 (190)
           - TotalRawHdfsReadTime(*): 794.747ms
           - TotalReadThroughput: 469.66 KB/sec
      Instance 8f4fe258fc27f5de:86cdd2f9cdda16b4 (host=myhdpnode07.mydomain:22000):(Total: 35s129ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:4/228.52 MB 
        MemoryUsage(1s000ms): 22.35 MB, 45.03 MB, 45.12 MB, 45.34 MB, 45.50 MB, 45.80 MB, 45.88 MB, 45.75 MB, 45.72 MB, 45.56 MB, 42.77 MB, 39.97 MB, 39.48 MB, 39.17 MB, 39.08 MB, 39.17 MB, 39.36 MB, 39.73 MB, 40.11 MB, 40.36 MB, 40.67 MB, 38.35 MB, 35.78 MB, 34.63 MB, 33.39 MB, 33.65 MB, 33.86 MB, 31.21 MB, 28.37 MB, 28.34 MB, 22.13 MB, 18.47 MB, 15.46 MB, 12.71 MB, 8.05 MB
        ThreadUsage(1s000ms): 3, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 2, 2, 1, 1, 1, 1, 1
         - AverageThreadTokens: 3.54 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 45.97 MB (48204168)
         - PerHostPeakMemUsage: 45.97 MB (48204168)
         - PrepareTime: 135.033ms
         - RowsProduced: 178.75K (178753)
         - TotalCpuTime: 1m31s
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 34s684ms
         - TotalStorageWaitTime: 833.298ms
        BlockMgr:
           - BlockWritesOutstanding: 0 (0)
           - BlocksCreated: 0 (0)
           - BlocksRecycled: 0 (0)
           - BufferedPins: 0 (0)
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 19.20 GB (20615843840)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0.000ns
           - TotalEncryptionTime: 0.000ns
           - TotalIntegrityCheckTime: 0.000ns
           - TotalReadBlockTime: 0.000ns
        CodeGen:(Total: 335.442ms, non-child: 335.442ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 6.384ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.85 MB (1939948)
           - OptimizationTime: 195.063ms
           - PrepareTime: 132.908ms
        DataStreamSender (dst_id=1):(Total: 34s811ms, non-child: 34s811ms, % non-child: 100.00%)
           - BytesSent: 17.68 MB (18535810)
           - NetworkThroughput(*): 520.47 KB/sec
           - OverallThroughput: 519.98 KB/sec
           - PeakMemoryUsage: 1.06 KB (1088)
           - SerializeBatchTime: 123.929ms
           - ThriftTransmitTime(*): 34s778ms
           - UncompressedRowBatchSize: 30.99 MB (32499125)
        HDFS_SCAN_NODE (id=0):(Total: 451.752ms, non-child: 451.752ms, % non-child: 100.00%)
          ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 4
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:4/228.52 MB 
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 
          File Formats: PARQUET/SNAPPY:28 
          BytesRead(1s000ms): 7.35 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB, 14.70 MB
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 2.57 
           - BytesRead: 14.70 MB (15417381)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 24.181ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 7 (7)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 4 (4)
           - NumScannerThreadsStarted: 4 (4)
           - PeakMemoryUsage: 45.72 MB (47940952)
           - PerReadThreadRawHdfsThroughput: 20.52 MB/sec
           - RemoteScanRanges: 32 (32)
           - RowsRead: 531.95K (531948)
           - RowsReturned: 178.75K (178753)
           - RowsReturnedRate: 395.69 K/sec
           - ScanRangesComplete: 4 (4)
           - ScannerThreadsInvoluntaryContextSwitches: 15 (15)
           - ScannerThreadsTotalWallClockTime: 1m31s
             - MaterializeTupleTime(*): 1m27s
             - ScannerThreadsSysTime: 999.000us
             - ScannerThreadsUserTime: 243.960ms
           - ScannerThreadsVoluntaryContextSwitches: 177 (177)
           - TotalRawHdfsReadTime(*): 716.586ms
           - TotalReadThroughput: 424.11 KB/sec
      Instance 8f4fe258fc27f5de:86cdd2f9cdda16b2 (host=myhdpnode08.mydomain:22000):(Total: 33s483ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:3/165.08 MB 
        MemoryUsage(1s000ms): 18.39 MB, 36.78 MB, 36.90 MB, 37.03 MB, 37.21 MB, 34.69 MB, 32.20 MB, 32.11 MB, 32.17 MB, 32.17 MB, 32.17 MB, 32.21 MB, 32.39 MB, 32.54 MB, 27.20 MB, 27.17 MB, 27.33 MB, 27.39 MB, 27.54 MB, 27.58 MB, 22.36 MB, 21.79 MB, 21.29 MB, 20.61 MB, 20.02 MB, 19.27 MB, 18.77 MB, 14.21 MB, 12.77 MB, 11.84 MB, 11.21 MB, 8.37 MB, 4.91 MB
        ThreadUsage(1s000ms): 2, 4, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 2.21 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 37.28 MB (39087656)
         - PerHostPeakMemUsage: 37.28 MB (39087656)
         - PrepareTime: 135.335ms
         - RowsProduced: 129.37K (129372)
         - TotalCpuTime: 41s097ms
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 33s113ms
         - TotalStorageWaitTime: 457.415ms
        BlockMgr:
           - BlockWritesOutstanding: 0 (0)
           - BlocksCreated: 0 (0)
           - BlocksRecycled: 0 (0)
           - BufferedPins: 0 (0)
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 19.20 GB (20615843840)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0.000ns
           - TotalEncryptionTime: 0.000ns
           - TotalIntegrityCheckTime: 0.000ns
           - TotalReadBlockTime: 0.000ns
        CodeGen:(Total: 337.278ms, non-child: 337.278ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 6.426ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.85 MB (1939948)
           - OptimizationTime: 196.635ms
           - PrepareTime: 133.161ms
        DataStreamSender (dst_id=1):(Total: 33s118ms, non-child: 33s118ms, % non-child: 100.00%)
           - BytesSent: 12.28 MB (12874371)
           - NetworkThroughput(*): 378.80 KB/sec
           - OverallThroughput: 379.62 KB/sec
           - PeakMemoryUsage: 1.06 KB (1088)
           - SerializeBatchTime: 83.181ms
           - ThriftTransmitTime(*): 33s190ms
           - UncompressedRowBatchSize: 22.44 MB (23533194)
        HDFS_SCAN_NODE (id=0):(Total: 419.264ms, non-child: 419.264ms, % non-child: 100.00%)
          ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 3
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:3/165.08 MB 
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 
          File Formats: PARQUET/SNAPPY:21 
          BytesRead(1s000ms): 5.32 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 1.23 
           - BytesRead: 10.64 MB (11159235)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 19.749ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 7 (7)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 3 (3)
           - NumScannerThreadsStarted: 3 (3)
           - PeakMemoryUsage: 37.09 MB (38889960)
           - PerReadThreadRawHdfsThroughput: 20.33 MB/sec
           - RemoteScanRanges: 24 (24)
           - RowsRead: 384.26K (384263)
           - RowsReturned: 129.37K (129372)
           - RowsReturnedRate: 308.57 K/sec
           - ScanRangesComplete: 3 (3)
           - ScannerThreadsInvoluntaryContextSwitches: 120 (120)
           - ScannerThreadsTotalWallClockTime: 40s980ms
             - MaterializeTupleTime(*): 37s658ms
             - ScannerThreadsSysTime: 2.999ms
             - ScannerThreadsUserTime: 184.971ms
           - ScannerThreadsVoluntaryContextSwitches: 88 (88)
           - TotalRawHdfsReadTime(*): 523.496ms
           - TotalReadThroughput: 325.30 KB/sec
      Instance 8f4fe258fc27f5de:86cdd2f9cdda16b3 (host=myhdpnode09.mydomain:22000):(Total: 32s991ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:4/216.07 MB 
        MemoryUsage(1s000ms): 24.79 MB, 43.49 MB, 43.90 MB, 44.16 MB, 44.03 MB, 43.99 MB, 44.31 MB, 44.45 MB, 39.49 MB, 39.63 MB, 39.77 MB, 40.15 MB, 40.40 MB, 40.52 MB, 41.01 MB, 41.18 MB, 41.40 MB, 38.94 MB, 36.35 MB, 34.81 MB, 30.86 MB, 28.08 MB, 28.24 MB, 28.31 MB, 25.56 MB, 21.45 MB, 19.76 MB, 19.10 MB, 15.49 MB, 12.32 MB, 7.61 MB, 6.15 MB, 4.11 MB
        ThreadUsage(1s000ms): 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 3.20 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 44.58 MB (46746664)
         - PerHostPeakMemUsage: 44.58 MB (46746664)
         - PrepareTime: 135.480ms
         - RowsProduced: 168.98K (168976)
         - TotalCpuTime: 1m11s
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 32s568ms
         - TotalStorageWaitTime: 864.729ms
        BlockMgr:
           - BlockWritesOutstanding: 0 (0)
           - BlocksCreated: 0 (0)
           - BlocksRecycled: 0 (0)
           - BufferedPins: 0 (0)
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 19.20 GB (20615843840)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0.000ns
           - TotalEncryptionTime: 0.000ns
           - TotalIntegrityCheckTime: 0.000ns
           - TotalReadBlockTime: 0.000ns
        CodeGen:(Total: 334.216ms, non-child: 334.216ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 6.309ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.85 MB (1939948)
           - OptimizationTime: 193.524ms
           - PrepareTime: 133.288ms
        DataStreamSender (dst_id=1):(Total: 32s560ms, non-child: 32s560ms, % non-child: 100.00%)
           - BytesSent: 15.50 MB (16251429)
           - NetworkThroughput(*): 485.82 KB/sec
           - OverallThroughput: 487.41 KB/sec
           - PeakMemoryUsage: 1.06 KB (1088)
           - SerializeBatchTime: 101.660ms
           - ThriftTransmitTime(*): 32s667ms
           - UncompressedRowBatchSize: 29.32 MB (30741134)
        HDFS_SCAN_NODE (id=0):(Total: 454.339ms, non-child: 454.339ms, % non-child: 100.00%)
          ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 4
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:4/216.07 MB 
          Hdfs Read Thread Concurrency Bucket: 0:98.48% 1:0% 2:0% 3:0% 4:0% 5:1.515% 
          File Formats: PARQUET/SNAPPY:28 
          BytesRead(1s000ms): 7.12 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB, 13.94 MB
           - AverageHdfsReadThreadConcurrency: 0.08 
           - AverageScannerThreadConcurrency: 2.20 
           - BytesRead: 13.94 MB (14613412)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 22.714ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 7 (7)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 4 (4)
           - NumScannerThreadsStarted: 4 (4)
           - PeakMemoryUsage: 44.38 MB (46540792)
           - PerReadThreadRawHdfsThroughput: 15.40 MB/sec
           - RemoteScanRanges: 32 (32)
           - RowsRead: 502.61K (502614)
           - RowsReturned: 168.98K (168976)
           - RowsReturnedRate: 371.92 K/sec
           - ScanRangesComplete: 4 (4)
           - ScannerThreadsInvoluntaryContextSwitches: 98 (98)
           - ScannerThreadsTotalWallClockTime: 1m11s
             - MaterializeTupleTime(*): 1m9s
             - ScannerThreadsSysTime: 1.999ms
             - ScannerThreadsUserTime: 240.962ms
           - ScannerThreadsVoluntaryContextSwitches: 168 (168)
           - TotalRawHdfsReadTime(*): 904.843ms
           - TotalReadThroughput: 432.45 KB/sec
      Instance 8f4fe258fc27f5de:86cdd2f9cdda16b1 (host=myhdpnodell02.mydomain:22000):(Total: 30s861ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:3/190.70 MB 
        MemoryUsage(500.000ms): 0, 36.76 MB, 36.95 MB, 37.04 MB, 37.04 MB, 31.93 MB, 31.80 MB, 31.68 MB, 31.68 MB, 31.68 MB, 31.53 MB, 31.28 MB, 31.15 MB, 31.03 MB, 30.84 MB, 30.74 MB, 30.43 MB, 30.49 MB, 30.49 MB, 30.49 MB, 30.49 MB, 30.68 MB, 30.62 MB, 30.93 MB, 31.12 MB, 31.43 MB, 31.55 MB, 31.74 MB, 31.87 MB, 32.24 MB, 29.38 MB, 29.75 MB, 29.97 MB, 30.10 MB, 30.29 MB, 30.54 MB, 30.60 MB, 30.79 MB, 30.85 MB, 25.40 MB, 25.33 MB, 25.40 MB, 25.40 MB, 19.70 MB, 19.13 MB, 18.51 MB, 18.26 MB, 17.51 MB, 16.32 MB, 15.63 MB, 15.01 MB, 14.26 MB, 14.01 MB, 13.38 MB, 12.57 MB, 11.76 MB, 10.95 MB, 10.45 MB, 9.82 MB, 9.07 MB, 4.96 MB, 4.15 MB
        ThreadUsage(500.000ms): 1, 4, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 2.39 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 37.06 MB (38864072)
         - PerHostPeakMemUsage: 37.06 MB (38864072)
         - PrepareTime: 132.710ms
         - RowsProduced: 149.01K (149009)
         - TotalCpuTime: 43s111ms
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 30s501ms
         - TotalStorageWaitTime: 526.426ms
        BlockMgr:
           - BlockWritesOutstanding: 0 (0)
           - BlocksCreated: 0 (0)
           - BlocksRecycled: 0 (0)
           - BufferedPins: 0 (0)
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 19.20 GB (20615843840)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0.000ns
           - TotalEncryptionTime: 0.000ns
           - TotalIntegrityCheckTime: 0.000ns
           - TotalReadBlockTime: 0.000ns
        CodeGen:(Total: 329.923ms, non-child: 329.923ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 6.459ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.85 MB (1939948)
           - OptimizationTime: 191.801ms
           - PrepareTime: 130.606ms
        DataStreamSender (dst_id=1):(Total: 30s456ms, non-child: 30s456ms, % non-child: 100.00%)
           - BytesSent: 13.50 MB (14159217)
           - NetworkThroughput(*): 452.05 KB/sec
           - OverallThroughput: 454.00 KB/sec
           - PeakMemoryUsage: 1.06 KB (1088)
           - SerializeBatchTime: 89.237ms
           - ThriftTransmitTime(*): 30s587ms
           - UncompressedRowBatchSize: 25.81 MB (27063813)
        HDFS_SCAN_NODE (id=0):(Total: 401.126ms, non-child: 401.126ms, % non-child: 100.00%)
          ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 3
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:3/190.70 MB 
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 
          File Formats: PARQUET/SNAPPY:21 
          BytesRead(500.000ms): 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB, 12.23 MB
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 1.41 
           - BytesRead: 12.23 MB (12823663)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 21.396ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 7 (7)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 3 (3)
           - NumScannerThreadsStarted: 3 (3)
           - PeakMemoryUsage: 36.94 MB (38731912)
           - PerReadThreadRawHdfsThroughput: 17.87 MB/sec
           - RemoteScanRanges: 24 (24)
           - RowsRead: 444.62K (444623)
           - RowsReturned: 149.01K (149009)
           - RowsReturnedRate: 371.48 K/sec
           - ScanRangesComplete: 3 (3)
           - ScannerThreadsInvoluntaryContextSwitches: 14 (14)
           - ScannerThreadsTotalWallClockTime: 43s078ms
             - MaterializeTupleTime(*): 41s383ms
             - ScannerThreadsSysTime: 1.999ms
             - ScannerThreadsUserTime: 211.967ms
           - ScannerThreadsVoluntaryContextSwitches: 136 (136)
           - TotalRawHdfsReadTime(*): 684.405ms
           - TotalReadThroughput: 410.59 KB/sec
      Instance 8f4fe258fc27f5de:86cdd2f9cdda16b7 (host=myhdpnodell01.mydomain:22000):(Total: 30s422ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:4/241.49 MB 
        MemoryUsage(500.000ms): 9.06 KB, 34.93 MB, 31.77 MB, 31.77 MB, 31.77 MB, 31.65 MB, 31.56 MB, 31.31 MB, 30.90 MB, 30.61 MB, 30.49 MB, 30.49 MB, 30.49 MB, 30.49 MB, 30.49 MB, 30.43 MB, 30.49 MB, 30.43 MB, 30.43 MB, 30.55 MB, 30.36 MB, 27.44 MB, 27.69 MB, 28.06 MB, 28.56 MB, 28.56 MB, 28.94 MB, 29.44 MB, 29.63 MB, 30.00 MB, 30.19 MB, 30.60 MB, 30.72 MB, 30.85 MB, 30.79 MB, 30.95 MB, 30.88 MB, 30.95 MB, 30.82 MB, 30.88 MB, 30.82 MB, 30.82 MB, 25.62 MB, 20.08 MB, 19.08 MB, 18.14 MB, 17.45 MB, 17.14 MB, 16.52 MB, 16.20 MB, 15.83 MB, 14.58 MB, 13.58 MB, 12.95 MB, 12.27 MB, 11.39 MB, 10.89 MB, 10.02 MB, 9.27 MB, 7.70 MB, 7.08 MB
        ThreadUsage(500.000ms): 1, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 2.39 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 34.93 MB (36625424)
         - PerHostPeakMemUsage: 34.93 MB (36625424)
         - PrepareTime: 133.987ms
         - RowsProduced: 189.59K (189585)
         - TotalCpuTime: 42s799ms
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 30s024ms
         - TotalStorageWaitTime: 775.152ms
        BlockMgr:
           - BlockWritesOutstanding: 0 (0)
           - BlocksCreated: 0 (0)
           - BlocksRecycled: 0 (0)
           - BufferedPins: 0 (0)
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 19.20 GB (20615843840)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0.000ns
           - TotalEncryptionTime: 0.000ns
           - TotalIntegrityCheckTime: 0.000ns
           - TotalReadBlockTime: 0.000ns
        CodeGen:(Total: 332.134ms, non-child: 332.134ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 6.203ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.85 MB (1939948)
           - OptimizationTime: 193.006ms
           - PrepareTime: 131.395ms
        DataStreamSender (dst_id=1):(Total: 30s137ms, non-child: 30s137ms, % non-child: 100.00%)
           - BytesSent: 16.50 MB (17300730)
           - NetworkThroughput(*): 560.46 KB/sec
           - OverallThroughput: 560.61 KB/sec
           - PeakMemoryUsage: 1.06 KB (1088)
           - SerializeBatchTime: 111.342ms
           - ThriftTransmitTime(*): 30s145ms
           - UncompressedRowBatchSize: 32.86 MB (34453098)
        HDFS_SCAN_NODE (id=0):(Total: 416.770ms, non-child: 416.770ms, % non-child: 100.00%)
          ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 4
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:4/241.49 MB 
          Hdfs Read Thread Concurrency Bucket: 0:100% 1:0% 2:0% 3:0% 4:0% 5:0% 
          File Formats: PARQUET/SNAPPY:28 
          BytesRead(500.000ms): 0, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB, 15.50 MB
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 1.42 
           - BytesRead: 15.50 MB (16257106)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 26.584ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 7 (7)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 4 (4)
           - NumScannerThreadsStarted: 4 (4)
           - PeakMemoryUsage: 31.65 MB (33184224)
           - PerReadThreadRawHdfsThroughput: 18.93 MB/sec
           - RemoteScanRanges: 32 (32)
           - RowsRead: 562.51K (562515)
           - RowsReturned: 189.59K (189585)
           - RowsReturnedRate: 454.89 K/sec
           - ScanRangesComplete: 4 (4)
           - ScannerThreadsInvoluntaryContextSwitches: 93 (93)
           - ScannerThreadsTotalWallClockTime: 42s799ms
             - MaterializeTupleTime(*): 41s175ms
             - ScannerThreadsSysTime: 2.999ms
             - ScannerThreadsUserTime: 264.958ms
           - ScannerThreadsVoluntaryContextSwitches: 148 (148)
           - TotalRawHdfsReadTime(*): 819.102ms
           - TotalReadThroughput: 520.53 KB/sec
      Instance 8f4fe258fc27f5de:86cdd2f9cdda16b0 (host=myhdpnode10.mydomain:22000):(Total: 26s754ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:3/165.03 MB 
        MemoryUsage(500.000ms): 11.55 MB, 30.60 MB, 30.73 MB, 31.04 MB, 31.17 MB, 31.04 MB, 31.20 MB, 31.01 MB, 30.70 MB, 30.48 MB, 30.54 MB, 30.42 MB, 30.48 MB, 30.54 MB, 30.42 MB, 30.54 MB, 30.48 MB, 27.53 MB, 27.84 MB, 28.21 MB, 28.40 MB, 28.59 MB, 28.84 MB, 29.15 MB, 29.25 MB, 29.25 MB, 29.37 MB, 29.50 MB, 29.62 MB, 29.81 MB, 29.94 MB, 30.19 MB, 25.13 MB, 24.88 MB, 19.82 MB, 19.36 MB, 18.92 MB, 18.36 MB, 17.48 MB, 17.05 MB, 16.42 MB, 15.67 MB, 14.80 MB, 14.30 MB, 13.17 MB, 12.92 MB, 12.11 MB, 11.55 MB, 10.23 MB, 9.17 MB, 8.36 MB, 8.11 MB, 3.75 MB, 9.06 KB
        ThreadUsage(500.000ms): 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 2.28 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 31.23 MB (32746488)
         - PerHostPeakMemUsage: 31.23 MB (32746488)
         - PrepareTime: 135.319ms
         - RowsProduced: 129.26K (129257)
         - TotalCpuTime: 33s922ms
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 26s405ms
         - TotalStorageWaitTime: 530.926ms
        BlockMgr:
           - BlockWritesOutstanding: 0 (0)
           - BlocksCreated: 0 (0)
           - BlocksRecycled: 0 (0)
           - BufferedPins: 0 (0)
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 19.20 GB (20615843840)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0.000ns
           - TotalEncryptionTime: 0.000ns
           - TotalIntegrityCheckTime: 0.000ns
           - TotalReadBlockTime: 0.000ns
        CodeGen:(Total: 334.075ms, non-child: 334.075ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 6.403ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.85 MB (1939948)
           - OptimizationTime: 193.448ms
           - PrepareTime: 133.106ms
        DataStreamSender (dst_id=1):(Total: 26s295ms, non-child: 26s295ms, % non-child: 100.00%)
           - BytesSent: 11.71 MB (12283045)
           - NetworkThroughput(*): 452.98 KB/sec
           - OverallThroughput: 456.17 KB/sec
           - PeakMemoryUsage: 1.06 KB (1088)
           - SerializeBatchTime: 76.929ms
           - ThriftTransmitTime(*): 26s480ms
           - UncompressedRowBatchSize: 22.42 MB (23510755)
        HDFS_SCAN_NODE (id=0):(Total: 404.922ms, non-child: 404.922ms, % non-child: 100.00%)
          ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 3
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:3/165.03 MB 
          Hdfs Read Thread Concurrency Bucket: 0:98.11% 1:0% 2:0% 3:0% 4:0% 5:1.887% 
          File Formats: PARQUET/SNAPPY:21 
          BytesRead(500.000ms): 1.04 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB, 10.64 MB
           - AverageHdfsReadThreadConcurrency: 0.09 
           - AverageScannerThreadConcurrency: 1.30 
           - BytesRead: 10.64 MB (11152050)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 16.884ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 7 (7)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 3 (3)
           - NumScannerThreadsStarted: 3 (3)
           - PeakMemoryUsage: 31.08 MB (32585448)
           - PerReadThreadRawHdfsThroughput: 16.76 MB/sec
           - RemoteScanRanges: 24 (24)
           - RowsRead: 383.83K (383831)
           - RowsReturned: 129.26K (129257)
           - RowsReturnedRate: 319.21 K/sec
           - ScanRangesComplete: 3 (3)
           - ScannerThreadsInvoluntaryContextSwitches: 7 (7)
           - ScannerThreadsTotalWallClockTime: 33s903ms
             - MaterializeTupleTime(*): 32s682ms
             - ScannerThreadsSysTime: 0.000ns
             - ScannerThreadsUserTime: 183.971ms
           - ScannerThreadsVoluntaryContextSwitches: 114 (114)
           - TotalRawHdfsReadTime(*): 634.755ms
           - TotalReadThroughput: 410.97 KB/sec

Re: Impala Query Profile: SendersBlockedTimer

Explorer
      Instance 8f4fe258fc27f5de:86cdd2f9cdda16b6 (host=myhdpnode06.mydomain:22000):(Total: 21s282ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:3/153.01 MB 
        MemoryUsage(500.000ms): 19.91 MB, 31.00 MB, 31.00 MB, 31.07 MB, 31.25 MB, 31.44 MB, 31.44 MB, 31.50 MB, 31.50 MB, 31.69 MB, 31.69 MB, 23.71 MB, 23.46 MB, 23.46 MB, 23.83 MB, 23.90 MB, 24.02 MB, 24.32 MB, 18.89 MB, 18.52 MB, 17.83 MB, 17.58 MB, 17.27 MB, 16.80 MB, 16.37 MB, 16.12 MB, 15.18 MB, 14.74 MB, 14.18 MB, 13.62 MB, 13.37 MB, 12.87 MB, 11.99 MB, 11.55 MB, 10.93 MB, 10.24 MB, 6.44 MB, 5.87 MB, 5.37 MB, 4.81 MB, 4.06 MB, 3.75 MB, 3.25 MB
        ThreadUsage(500.000ms): 4, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 2, 2, 2, 2, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.72 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 31.80 MB (33343176)
         - PerHostPeakMemUsage: 31.80 MB (33343176)
         - PrepareTime: 135.339ms
         - RowsProduced: 120.39K (120386)
         - TotalCpuTime: 15s042ms
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 20s928ms
         - TotalStorageWaitTime: 442.491ms
        BlockMgr:
           - BlockWritesOutstanding: 0 (0)
           - BlocksCreated: 0 (0)
           - BlocksRecycled: 0 (0)
           - BufferedPins: 0 (0)
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 19.20 GB (20615843840)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0.000ns
           - TotalEncryptionTime: 0.000ns
           - TotalIntegrityCheckTime: 0.000ns
           - TotalReadBlockTime: 0.000ns
        CodeGen:(Total: 336.420ms, non-child: 336.420ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 6.587ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.85 MB (1939948)
           - OptimizationTime: 195.562ms
           - PrepareTime: 133.166ms
        DataStreamSender (dst_id=1):(Total: 21s006ms, non-child: 21s006ms, % non-child: 100.00%)
           - BytesSent: 11.09 MB (11633871)
           - NetworkThroughput(*): 540.85 KB/sec
           - OverallThroughput: 540.84 KB/sec
           - PeakMemoryUsage: 1.06 KB (1088)
           - SerializeBatchTime: 75.945ms
           - ThriftTransmitTime(*): 21s006ms
           - UncompressedRowBatchSize: 20.90 MB (21916861)
        HDFS_SCAN_NODE (id=0):(Total: 409.603ms, non-child: 409.603ms, % non-child: 100.00%)
          ExecOption: Expr Evaluation Codegen Enabled, Codegen enabled: 0 out of 3
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): -1:3/153.01 MB 
          Hdfs Read Thread Concurrency Bucket: 0:97.67% 1:0% 2:0% 3:0% 4:2.326% 5:0% 
          File Formats: PARQUET/SNAPPY:21 
          BytesRead(500.000ms): 4.43 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB, 9.89 MB
           - AverageHdfsReadThreadConcurrency: 0.09 
           - AverageScannerThreadConcurrency: 0.72 
           - BytesRead: 9.89 MB (10371800)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 15.437ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 7 (7)
           - NumDisksAccessed: 1 (1)
           - NumRowGroups: 3 (3)
           - NumScannerThreadsStarted: 3 (3)
           - PeakMemoryUsage: 31.56 MB (33097672)
           - PerReadThreadRawHdfsThroughput: 17.48 MB/sec
           - RemoteScanRanges: 24 (24)
           - RowsRead: 355.52K (355525)
           - RowsReturned: 120.39K (120386)
           - RowsReturnedRate: 293.91 K/sec
           - ScanRangesComplete: 3 (3)
           - ScannerThreadsInvoluntaryContextSwitches: 89 (89)
           - ScannerThreadsTotalWallClockTime: 14s927ms
             - MaterializeTupleTime(*): 14s024ms
             - ScannerThreadsSysTime: 2.998ms
             - ScannerThreadsUserTime: 170.973ms
           - ScannerThreadsVoluntaryContextSwitches: 72 (72)
           - TotalRawHdfsReadTime(*): 565.877ms
           - TotalReadThroughput: 471.10 KB/sec

Re: Impala Query Profile: SendersBlockedTimer

Cloudera Employee

RowMaterializationTimer - This counter tracks the time taken to convert rows sent back to the client into the ASCII format that the Beeswax API demands. 

SendersBlockedTimer - the wall-clock time that senders are waiting for the receive (upstream operator) buffer to be emptied. 

Impala processes data in batch, by default, the batch size = 1024.

 

If you look at the beginning of query profile

Query Timeline: 39s394ms
- Start execution: 30.989us (30.989us)
- Planning finished: 29.209ms (29.178ms)
- Ready to start 9 remote fragments: 29.964ms (755.205us)
- All 9 remote fragments started: 33.367ms (3.402ms)
- Rows available: 483.564ms (450.197ms)
- First row fetched: 483.630ms (66.052us)
ImpalaServer:
- ClientFetchWaitTimer: 36s067ms
- RowMaterializationTimer: 2s544ms

 

Total query time is ~39sec, the first batch took ~500ms. most of time are client waiting time. Obviously the client is not continue fetching data.

Impala uses pull model, only if client keeps asking for more data, then it keeps working to get next batch. Since client did not ask for more, the receiver buffer is not emptied, and sender will wait.

 

Did you use hue to run the query?

If you run the query in Impala-shell, it will fetch all data, no waiting. that will give you better idea how long the query takes. but if the query returns large amount of rows, the RowMaterializationTimer could be long.

you can use impala-shell -B -q "query" > /dev/null to discard the result  or write them to a file.