Thanks for your promt response. The full profile is too long to post once, so will post it in multi-threads. The first fragment is as follows:   Query Runtime Profile: Query (id=704b7e71011e9573:13c1490afe0019aa): Summary: Session ID: a64bfccabfc44c7a:4f426c2ace530499 Session Type: BEESWAX Start Time: 2016-04-25 20:01:42.047887000 End Time: 2016-04-25 20:03:58.618135000 Query Type: QUERY Query State: FINISHED Query Status: OK Impala Version: impalad version 2.2.0-cdh5.4.8 RELEASE (build 137d99e9f751c454a0c79f3b00302938e4984f9c) User: testuser@HADOOP.COM Connected User: testuser@HADOOP.COM Delegated User: Network Address: 192.168.1.45:57184 Default Db: default Sql Statement: select * from ps.test_tab where col_a='538938396' and no='6' Coordinator: hadoop13:22000 Plan: ---------------- Estimated Per-Host Requirements: Memory=4.73GB VCores=1 WARNING: The following tables are missing relevant table and/or column statistics. ps.test_tab F01:PLAN FRAGMENT [UNPARTITIONED] 01:EXCHANGE [UNPARTITIONED] hosts=29 per-host-mem=unavailable tuple-ids=0 row-size=622B cardinality=unavailable F00:PLAN FRAGMENT [RANDOM] DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=01, UNPARTITIONED] 00:SCAN HDFS [ps.test_tab, RANDOM] partitions=1/13 files=243 size=24.85GB predicates: col_a = '538938396' table stats: unavailable columns missing stats: * hosts=29 per-host-mem=4.73GB tuple-ids=0 row-size=622B cardinality=unavailable ---------------- Estimated Per-Host Mem: 5075107840 Estimated Per-Host VCores: 1 Tables Missing Stats: ps.test_tab Request Pool: default-pool ExecSummary: Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail ---------------------------------------------------------------------------------------------------------------------- 01:EXCHANGE 1 9.925us 9.925us 0 -1 0 -1.00 B UNPARTITIONED 00:SCAN HDFS 16 23s428ms 2m15s 0 -1 1.36 GB 4.73 GB ps.test_tab Planner Timeline: 3.794ms - Analysis finished: 1.114ms (1.114ms) - Equivalence classes computed: 1.306ms (191.564us) - Single node plan created: 2.322ms (1.16ms) - Distributed plan created: 2.807ms (484.899us) - Planning finished: 3.794ms (987.364us) Query Timeline: 2m16s - Start execution: 109.651us (109.651us) - Planning finished: 5.947ms (5.837ms) - Ready to start remote fragments: 7.248ms (1.301ms) - Remote fragments started: 329.94ms (321.845ms) - Rows available: 2m15s (2m15s) - First row fetched: 2m16s (723.75ms) - Unregister query: 2m16s (2.442ms) ImpalaServer: - ClientFetchWaitTimer: 724.375ms - RowMaterializationTimer: 0ns Execution Profile 704b7e71011e9573:13c1490afe0019aa:(Total: 2m15s, non-child: 0ns, % non-child: 0.00%) Fragment start latencies: count: 16, last: 0.319323ns, min: 0.004591ns, max: 0.319323ns, mean: 0.036861ns, stddev: 0.080221ns Per Node Peak Memory Usage: hadoop32:22000(557.23 MB) hadoop6:22000(1.07 GB) hadoop16:22000(824.00 MB) hadoop15:22000(1.15 GB) hadoop14:22000(774.47 MB) hadoop13:22000(488.44 MB) hadoop12:22000(175.64 MB) hadoop7:22000(1.36 GB) hadoop30:22000(968.32 MB) hadoop11:22000(480.09 MB) hadoop8:22000(556.99 MB) hadoop31:22000(678.14 MB) hadoop10:22000(931.50 MB) hadoop4:22000(652.39 MB) hadoop9:22000(700.22 MB) hadoop5:22000(459.63 MB) - FinalizationTimer: 0ns Coordinator Fragment F01:(Total: 2m15s, non-child: 250.809us, % non-child: 0.00%) MemoryUsage(4s000ms): 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB, 8.00 KB ThreadUsage(500.0ms): 1 - AverageThreadTokens: 1.00 - PeakMemoryUsage: 8.00 KB (8192) - PerHostPeakMemUsage: 0 - PrepareTime: 34.243us - RowsProduced: 0 (0) - TotalCpuTime: 1s046ms - TotalNetworkReceiveTime: 2m15s - TotalNetworkSendTime: 0ns - TotalStorageWaitTime: 0ns BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns EXCHANGE_NODE (id=1):(Total: 2m15s, non-child: 9.925us, % non-child: 0.00%) BytesReceived(4s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 - BytesReceived: 0 - ConvertRowBatchTime: 1.906us - DeserializeRowBatchTimer: 0ns - FirstBatchArrivalWaitTime: 2m15s - PeakMemoryUsage: 0 - RowsReturned: 0 (0) - RowsReturnedRate: 0 - SendersBlockedTimer: 0ns - SendersBlockedTotalTimer(*): 0ns Averaged Fragment F00:(Total: 23s460ms, non-child: 31.950ms, % non-child: 0.14%) split sizes: min: 619.60 MB, max: 4.61 GB, avg: 1.55 GB, stddev: 1.10 GB completion times: min:7s229ms max:2m15s mean: 23s454ms stddev:30s248ms execution rates: min:27.75 MB/sec max:156.48 MB/sec mean:87.51 MB/sec stddev:34.03 MB/sec num instances: 16 - AverageThreadTokens: 7.04 - PeakMemoryUsage: 745.07 MB (781258467) - PerHostPeakMemUsage: 745.07 MB (781258979) - PrepareTime: 351.778us - RowsProduced: 0 (0) - TotalCpuTime: 1m57s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 123ns - TotalStorageWaitTime: 1m12s BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 22.240us, non-child: 22.240us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 23s428ms, non-child: 23s428ms, % non-child: 100.00%) - AverageHdfsReadThreadConcurrency: 6.21 - AverageScannerThreadConcurrency: 6.04 - BytesRead: 1.55 GB (1668901870) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.46 GB (1565486679) - BytesReadRemoteUnexpected: 98.62 MB (103415191) - BytesReadShortCircuit: 1.46 GB (1565486679) - DecompressionTime: 267.18ms - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 7 (7) - NumScannerThreadsStarted: 10 (10) - PeakMemoryUsage: 745.05 MB (781246179) - PerReadThreadRawHdfsThroughput: 14.41 MB/sec - RemoteScanRanges: 56 (56) - RowsRead: 22.57M (22567007) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 15 (15) - ScannerThreadsInvoluntaryContextSwitches: 1.17K (1166) - ScannerThreadsTotalWallClockTime: 1m52s - MaterializeTupleTime(*): 38s478ms - ScannerThreadsSysTime: 151.6ms - ScannerThreadsUserTime: 38s641ms - ScannerThreadsVoluntaryContextSwitches: 800 (800) - TotalRawHdfsReadTime(*): 1m49s - TotalReadThroughput: 88.05 MB/sec Fragment F00: Instance 704b7e71011e9573:13c1490afe0019b2 (host=hadoop12:22000):(Total: 2m15s, non-child: 1.479ms, % non-child: 0.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:7/519.57 MB 1:1/146.98 MB 2:3/278.56 MB 3:6/558.73 MB 4:3/324.40 MB 5:1/121.36 MB 6:4/381.73 MB 7:3/391.78 MB 8:3/249.89 MB 10:4/461.08 MB 11:3/334.88 MB MemoryUsage(4s000ms): 79.95 MB, 86.50 MB, 93.83 MB, 98.06 MB, 99.87 MB, 63.14 MB, 65.26 MB, 85.37 MB, 61.75 MB, 109.14 MB, 85.33 MB, 84.14 MB, 61.62 MB, 61.50 MB, 107.47 MB, 39.42 MB, 66.96 MB, 104.71 MB, 102.27 MB, 100.25 MB, 50.51 MB, 92.83 MB, 93.34 MB, 51.29 MB, 102.11 MB, 79.02 MB, 100.86 MB, 94.76 MB, 52.75 MB, 102.00 MB, 61.42 MB, 105.48 MB, 54.05 MB ThreadUsage(4s000ms): 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2 - AverageThreadTokens: 2.00 - PeakMemoryUsage: 175.64 MB (184170048) - PerHostPeakMemUsage: 175.64 MB (184170048) - PrepareTime: 280.49us - RowsProduced: 0 (0) - TotalCpuTime: 3m27s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 229ns - TotalStorageWaitTime: 1m3s BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 14.736us, non-child: 14.736us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 2m15s, non-child: 2m15s, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:7/519.57 MB 1:1/146.98 MB 2:3/278.56 MB 3:6/558.73 MB 4:3/324.40 MB 5:1/121.36 MB 6:4/381.73 MB 7:3/391.78 MB 8:3/249.89 MB 10:4/461.08 MB 11:3/334.88 MB Hdfs Read Thread Concurrency Bucket: 0:64.94% 1:3.69% 2:1.476% 3:1.107% 4:2.214% 5:6.642% 6:19.93% 7:0% 8:0% 9:0% 10:0% 11:0% 12:0% 13:0% 14:0% 15:0% File Formats: PARQUET/NONE:2090 ExecOption: Codegen enabled: 0 out of 38 BytesRead(4s000ms): 64.87 MB, 181.39 MB, 313.01 MB, 441.49 MB, 589.11 MB, 703.53 MB, 759.85 MB, 858.54 MB, 953.43 MB, 1.07 GB, 1.20 GB, 1.29 GB, 1.37 GB, 1.48 GB, 1.62 GB, 1.67 GB, 1.71 GB, 1.80 GB, 1.93 GB, 2.05 GB, 2.13 GB, 2.25 GB, 2.38 GB, 2.47 GB, 2.62 GB, 2.74 GB, 2.85 GB, 2.97 GB, 3.08 GB, 3.22 GB, 3.32 GB, 3.45 GB, 3.54 GB - AverageHdfsReadThreadConcurrency: 1.72 - AverageScannerThreadConcurrency: 1.00 - BytesRead: 3.68 GB (3955651251) - BytesReadDataNodeCache: 0 - BytesReadLocal: 3.32 GB (3566509512) - BytesReadRemoteUnexpected: 371.11 MB (389141739) - BytesReadShortCircuit: 3.32 GB (3566509512) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 11 (11) - NumScannerThreadsStarted: 1 (1) - PeakMemoryUsage: 175.63 MB (184157760) - PerReadThreadRawHdfsThroughput: 16.33 MB/sec - RemoteScanRanges: 168 (168) - RowsRead: 51.40M (51395362) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 38 (38) - ScannerThreadsInvoluntaryContextSwitches: 767 (767) - ScannerThreadsTotalWallClockTime: 2m15s - MaterializeTupleTime(*): 1m11s - ScannerThreadsSysTime: 152.9ms - ScannerThreadsUserTime: 1m11s - ScannerThreadsVoluntaryContextSwitches: 1.38K (1379) - TotalRawHdfsReadTime(*): 3m51s - TotalReadThroughput: 27.84 MB/sec Instance 704b7e71011e9573:13c1490afe0019af (host=hadoop15:22000):(Total: 41s242ms, non-child: 49.114ms, % non-child: 0.12%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:3/364.32 MB 1:4/372.05 MB 2:3/289.97 MB 3:4/639.33 MB 4:4/493.65 MB 5:3/143.62 MB 6:2/154.68 MB 7:6/589.98 MB 8:4/516.45 MB 9:6/712.48 MB 10:6/445.67 MB MemoryUsage(1s000ms): 230.80 MB, 474.91 MB, 635.97 MB, 912.24 MB, 1.11 GB, 982.84 MB, 762.34 MB, 576.20 MB, 477.20 MB, 637.89 MB, 900.22 MB, 978.83 MB, 1.04 GB, 1.07 GB, 1.05 GB, 905.77 MB, 825.52 MB, 867.11 MB, 841.12 MB, 954.98 MB, 860.56 MB, 830.51 MB, 881.14 MB, 821.94 MB, 703.41 MB, 637.89 MB, 672.36 MB, 574.61 MB, 574.55 MB, 446.31 MB, 507.56 MB, 527.58 MB, 424.05 MB, 447.66 MB, 342.97 MB, 263.54 MB, 273.37 MB, 241.94 MB, 248.92 MB, 205.48 MB, 136.53 MB ThreadUsage(1s000ms): 15, 15, 15, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, 16, 16, 15, 14, 13, 12, 12, 11, 10, 9, 8, 8, 6, 5, 5, 5, 5, 4, 3, 3, 3, 3, 2, 2 - AverageThreadTokens: 11.13 - PeakMemoryUsage: 1.15 GB (1235100464) - PerHostPeakMemUsage: 1.15 GB (1235100464) - PrepareTime: 450.3us - RowsProduced: 0 (0) - TotalCpuTime: 6m58s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 97ns - TotalStorageWaitTime: 4m40s BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 23.286us, non-child: 23.286us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 41s193ms, non-child: 41s193ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:3/364.32 MB 1:4/372.05 MB 2:3/289.97 MB 3:4/639.33 MB 4:4/493.65 MB 5:3/143.62 MB 6:2/154.68 MB 7:6/589.98 MB 8:4/516.45 MB 9:6/712.48 MB 10:6/445.67 MB Hdfs Read Thread Concurrency Bucket: 0:24.39% 1:9.756% 2:3.659% 3:2.439% 4:6.098% 5:2.439% 6:8.537% 7:2.439% 8:3.659% 9:1.22% 10:0% 11:2.439% 12:1.22% 13:1.22% 14:3.659% 15:26.83% File Formats: PARQUET/NONE:2310 PARQUET/SNAPPY:165 ExecOption: Codegen enabled: 0 out of 45 BytesRead(1s000ms): 103.87 MB, 321.67 MB, 498.15 MB, 838.14 MB, 1.10 GB, 1.16 GB, 1.27 GB, 1.41 GB, 1.51 GB, 1.76 GB, 2.12 GB, 2.30 GB, 2.42 GB, 2.60 GB, 2.80 GB, 2.93 GB, 3.09 GB, 3.31 GB, 3.49 GB, 3.68 GB, 3.82 GB, 3.91 GB, 4.04 GB, 4.10 GB, 4.15 GB, 4.18 GB, 4.24 GB, 4.33 GB, 4.36 GB, 4.47 GB, 4.55 GB, 4.55 GB, 4.55 GB, 4.56 GB, 4.57 GB, 4.59 GB, 4.61 GB, 4.61 GB, 4.62 GB, 4.62 GB, 4.62 GB - AverageHdfsReadThreadConcurrency: 7.91 - AverageScannerThreadConcurrency: 10.13 - BytesRead: 4.62 GB (4955876955) - BytesReadDataNodeCache: 0 - BytesReadLocal: 4.30 GB (4617848579) - BytesReadRemoteUnexpected: 322.37 MB (338028376) - BytesReadShortCircuit: 4.30 GB (4617848579) - DecompressionTime: 2s446ms - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 11 (11) - NumScannerThreadsStarted: 15 (15) - PeakMemoryUsage: 1.15 GB (1235088176) - PerReadThreadRawHdfsThroughput: 14.59 MB/sec - RemoteScanRanges: 168 (168) - RowsRead: 76.26M (76259233) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 45 (45) - ScannerThreadsInvoluntaryContextSwitches: 6.97K (6970) - ScannerThreadsTotalWallClockTime: 6m58s - MaterializeTupleTime(*): 2m14s - ScannerThreadsSysTime: 668.37ms - ScannerThreadsUserTime: 2m16s - ScannerThreadsVoluntaryContextSwitches: 2.85K (2846) - TotalRawHdfsReadTime(*): 5m24s - TotalReadThroughput: 115.28 MB/sec Instance 704b7e71011e9573:13c1490afe0019ba (host=hadoop10:22000):(Total: 30s962ms, non-child: 315.670ms, % non-child: 1.02%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/121.46 MB 2:2/228.69 MB 3:1/98.85 MB 4:1/96.97 MB 5:1/144.57 MB 7:1/133.90 MB 9:4/518.27 MB 10:4/287.31 MB MemoryUsage(500.0ms): 0, 107.28 MB, 184.44 MB, 325.77 MB, 591.49 MB, 658.19 MB, 696.13 MB, 656.14 MB, 757.04 MB, 795.58 MB, 920.26 MB, 817.65 MB, 502.89 MB, 534.70 MB, 387.47 MB, 391.78 MB, 360.46 MB, 301.41 MB, 243.39 MB, 275.44 MB, 298.12 MB, 322.89 MB, 335.99 MB, 338.51 MB, 338.51 MB, 346.52 MB, 346.52 MB, 362.53 MB, 253.40 MB, 258.90 MB, 275.57 MB, 291.52 MB, 293.15 MB, 294.54 MB, 293.25 MB, 365.11 MB, 343.14 MB, 296.18 MB, 341.48 MB, 291.62 MB, 286.98 MB, 296.68 MB, 296.68 MB, 296.68 MB, 296.69 MB, 296.69 MB, 298.19 MB, 300.23 MB, 292.76 MB, 335.76 MB, 323.65 MB, 352.95 MB, 311.42 MB, 319.02 MB, 190.79 MB, 173.40 MB, 181.94 MB, 179.34 MB, 211.58 MB, 139.69 MB, 155.11 MB, 187.90 MB ThreadUsage(500.0ms): 1, 15, 15, 15, 15, 16, 16, 14, 13, 13, 13, 13, 9, 9, 7, 7, 7, 6, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 3, 3, 2, 2, 2, 2, 2, 2, 2, 2 - AverageThreadTokens: 6.10 - PeakMemoryUsage: 931.50 MB (976747384) - PerHostPeakMemUsage: 931.50 MB (976747384) - PrepareTime: 212.472us - RowsProduced: 0 (0) - TotalCpuTime: 2m39s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 230ns - TotalStorageWaitTime: 1m49s BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 35.319us, non-child: 35.319us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 30s646ms, non-child: 30s646ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 1:1/121.46 MB 2:2/228.69 MB 3:1/98.85 MB 4:1/96.97 MB 5:1/144.57 MB 7:1/133.90 MB 9:4/518.27 MB 10:4/287.31 MB Hdfs Read Thread Concurrency Bucket: 0:42.62% 1:13.11% 2:4.918% 3:6.557% 4:13.11% 5:3.279% 6:0% 7:0% 8:1.639% 9:0% 10:0% 11:0% 12:1.639% 13:0% 14:1.639% 15:11.48% File Formats: PARQUET/NONE:770 PARQUET/SNAPPY:55 ExecOption: Codegen enabled: 0 out of 15 BytesRead(500.0ms): 0, 2.10 MB, 76.28 MB, 200.65 MB, 422.32 MB, 447.62 MB, 536.97 MB, 652.68 MB, 805.66 MB, 922.02 MB, 1.05 GB, 1.07 GB, 1.09 GB, 1.16 GB, 1.19 GB, 1.20 GB, 1.22 GB, 1.24 GB, 1.25 GB, 1.26 GB, 1.26 GB, 1.26 GB, 1.26 GB, 1.27 GB, 1.27 GB, 1.27 GB, 1.27 GB, 1.27 GB, 1.28 GB, 1.28 GB, 1.29 GB, 1.31 GB, 1.31 GB, 1.31 GB, 1.34 GB, 1.38 GB, 1.39 GB, 1.41 GB, 1.42 GB, 1.42 GB, 1.42 GB, 1.44 GB, 1.44 GB, 1.44 GB, 1.44 GB, 1.44 GB, 1.44 GB, 1.45 GB, 1.46 GB, 1.53 GB, 1.56 GB, 1.56 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB, 1.59 GB - AverageHdfsReadThreadConcurrency: 3.70 - AverageScannerThreadConcurrency: 5.18 - BytesRead: 1.59 GB (1710640190) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.55 GB (1667758557) - BytesReadRemoteUnexpected: 40.90 MB (42881633) - BytesReadShortCircuit: 1.55 GB (1667758557) - DecompressionTime: 1s826ms - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 8 (8) - NumScannerThreadsStarted: 15 (15) - PeakMemoryUsage: 931.49 MB (976735096) - PerReadThreadRawHdfsThroughput: 14.56 MB/sec - RemoteScanRanges: 56 (56) - RowsRead: 26.93M (26931028) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 15 (15) - ScannerThreadsInvoluntaryContextSwitches: 979 (979) - ScannerThreadsTotalWallClockTime: 2m39s - MaterializeTupleTime(*): 44s978ms - ScannerThreadsSysTime: 288.13ms - ScannerThreadsUserTime: 45s830ms - ScannerThreadsVoluntaryContextSwitches: 1.72K (1723) - TotalRawHdfsReadTime(*): 1m52s - TotalReadThroughput: 52.63 MB/sec Instance 704b7e71011e9573:13c1490afe0019b8 (host=hadoop9:22000):(Total: 21s516ms, non-child: 1.800ms, % non-child: 0.01%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/132.06 MB 11:1/54.86 MB 2:1/72.90 MB 3:1/45.58 MB 4:3/390.45 MB 5:4/367.45 MB 7:2/177.69 MB 8:1/63.92 MB 9:1/139.10 MB 10:2/89.20 MB MemoryUsage(500.0ms): 5.24 MB, 97.25 MB, 220.33 MB, 435.04 MB, 488.75 MB, 678.23 MB, 699.41 MB, 661.17 MB, 583.29 MB, 453.71 MB, 374.29 MB, 415.89 MB, 438.55 MB, 423.82 MB, 491.63 MB, 542.10 MB, 549.99 MB, 568.94 MB, 553.48 MB, 509.58 MB, 529.10 MB, 506.61 MB, 459.41 MB, 415.77 MB, 501.36 MB, 547.47 MB, 569.50 MB, 524.37 MB, 424.03 MB, 317.78 MB, 294.50 MB, 312.50 MB, 250.79 MB, 251.13 MB, 267.86 MB, 277.08 MB, 280.64 MB, 297.34 MB, 298.61 MB, 341.80 MB, 294.50 MB, 178.31 MB, 89.47 MB, 65.70 MB ThreadUsage(500.0ms): 18, 18, 18, 18, 17, 17, 17, 17, 17, 15, 14, 14, 14, 13, 13, 13, 12, 12, 12, 11, 11, 10, 9, 8, 8, 8, 8, 8, 7, 6, 6, 6, 5, 5, 5, 5, 5, 5, 5, 4, 4, 3, 2, 2 - AverageThreadTokens: 10.11 - PeakMemoryUsage: 700.22 MB (734232408) - PerHostPeakMemUsage: 700.22 MB (734232408) - PrepareTime: 317.552us - RowsProduced: 0 (0) - TotalCpuTime: 3m16s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 107ns - TotalStorageWaitTime: 2m37s BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 24.317us, non-child: 24.317us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 21s514ms, non-child: 21s514ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/132.06 MB 11:1/54.86 MB 2:1/72.90 MB 3:1/45.58 MB 4:3/390.45 MB 5:4/367.45 MB 7:2/177.69 MB 8:1/63.92 MB 9:1/139.10 MB 10:2/89.20 MB Hdfs Read Thread Concurrency Bucket: 0:31.82% 1:9.091% 2:9.091% 3:9.091% 4:6.818% 5:6.818% 6:2.273% 7:6.818% 8:2.273% 9:0% 10:0% 11:0% 12:0% 13:6.818% 14:2.273% 15:6.818% File Formats: PARQUET/NONE:935 ExecOption: Codegen enabled: 0 out of 17 BytesRead(500.0ms): 0, 18.88 MB, 87.98 MB, 249.89 MB, 378.11 MB, 584.27 MB, 622.83 MB, 655.74 MB, 660.83 MB, 669.58 MB, 680.35 MB, 702.98 MB, 711.95 MB, 737.54 MB, 770.92 MB, 847.56 MB, 886.72 MB, 967.54 MB, 998.15 MB, 1005.62 MB, 1.01 GB, 1.02 GB, 1.04 GB, 1.07 GB, 1.15 GB, 1.23 GB, 1.31 GB, 1.31 GB, 1.31 GB, 1.31 GB, 1.31 GB, 1.32 GB, 1.32 GB, 1.33 GB, 1.33 GB, 1.35 GB, 1.36 GB, 1.38 GB, 1.38 GB, 1.50 GB, 1.50 GB, 1.50 GB, 1.50 GB, 1.50 GB - AverageHdfsReadThreadConcurrency: 4.52 - AverageScannerThreadConcurrency: 9.11 - BytesRead: 1.50 GB (1609316761) - BytesReadDataNodeCache: 0 - BytesReadLocal: 1.32 GB (1418197209) - BytesReadRemoteUnexpected: 182.27 MB (191119552) - BytesReadShortCircuit: 1.32 GB (1418197209) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 10 (10) - NumScannerThreadsStarted: 17 (17) - PeakMemoryUsage: 700.21 MB (734220120) - PerReadThreadRawHdfsThroughput: 14.42 MB/sec - RemoteScanRanges: 112 (112) - RowsRead: 21.32M (21316737) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 17 (17) - ScannerThreadsInvoluntaryContextSwitches: 1.95K (1948) - ScannerThreadsTotalWallClockTime: 3m16s - MaterializeTupleTime(*): 37s777ms - ScannerThreadsSysTime: 240.8ms - ScannerThreadsUserTime: 37s614ms - ScannerThreadsVoluntaryContextSwitches: 1.19K (1190) - TotalRawHdfsReadTime(*): 1m46s - TotalReadThroughput: 69.76 MB/sec Instance 704b7e71011e9573:13c1490afe0019b3 (host=hadoop7:22000):(Total: 18s713ms, non-child: 1.174ms, % non-child: 0.01%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/117.90 MB 11:2/186.31 MB 1:3/213.89 MB 3:3/396.93 MB 4:2/229.78 MB 6:3/343.63 MB 7:1/140.95 MB 8:3/347.39 MB 9:1/59.54 MB 10:5/636.14 MB MemoryUsage(500.0ms): 36.09 MB, 171.64 MB, 279.90 MB, 330.08 MB, 482.92 MB, 557.69 MB, 631.92 MB, 683.42 MB, 813.40 MB, 856.23 MB, 890.94 MB, 991.39 MB, 1.06 GB, 1.13 GB, 1.34 GB, 1.24 GB, 1.11 GB, 1016.00 MB, 1.13 GB, 941.57 MB, 769.10 MB, 502.54 MB, 446.45 MB, 346.56 MB, 245.62 MB, 190.13 MB, 236.46 MB, 247.23 MB, 260.49 MB, 264.49 MB, 270.19 MB, 273.72 MB, 239.61 MB, 243.18 MB, 211.42 MB, 87.91 MB, 80.00 MB ThreadUsage(500.0ms): 15, 15, 15, 15, 15, 16, 16, 16, 16, 18, 18, 18, 18, 18, 18, 18, 18, 16, 14, 13, 11, 7, 7, 6, 5, 4, 4, 4, 4, 4, 4, 4, 3, 3, 3, 2, 2 - AverageThreadTokens: 10.89 - PeakMemoryUsage: 1.36 GB (1465641912) - PerHostPeakMemUsage: 1.36 GB (1465641912) - PrepareTime: 254.381us - RowsProduced: 0 (0) - TotalCpuTime: 3m6s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 93ns - TotalStorageWaitTime: 2m6s BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 14.645us, non-child: 14.645us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0 HDFS_SCAN_NODE (id=0):(Total: 18s712ms, non-child: 18s712ms, % non-child: 100.00%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:1/117.90 MB 11:2/186.31 MB 1:3/213.89 MB 3:3/396.93 MB 4:2/229.78 MB 6:3/343.63 MB 7:1/140.95 MB 8:3/347.39 MB 9:1/59.54 MB 10:5/636.14 MB Hdfs Read Thread Concurrency Bucket: 0:24.32% 1:2.703% 2:5.405% 3:2.703% 4:8.108% 5:8.108% 6:5.405% 7:2.703% 8:0% 9:0% 10:0% 11:0% 12:2.703% 13:0% 14:0% 15:37.84% File Formats: PARQUET/NONE:1320 ExecOption: Codegen enabled: 0 out of 24 BytesRead(500.0ms): 100.00 KB, 7.96 MB, 66.44 MB, 146.37 MB, 295.79 MB, 358.55 MB, 421.58 MB, 519.90 MB, 724.48 MB, 835.77 MB, 903.78 MB, 970.56 MB, 1.19 GB, 1.29 GB, 1.53 GB, 1.71 GB, 1.82 GB, 1.87 GB, 2.16 GB, 2.18 GB, 2.23 GB, 2.32 GB, 2.37 GB, 2.38 GB, 2.39 GB, 2.41 GB, 2.44 GB, 2.46 GB, 2.47 GB, 2.49 GB, 2.50 GB, 2.50 GB, 2.56 GB, 2.61 GB, 2.61 GB, 2.61 GB, 2.61 GB - AverageHdfsReadThreadConcurrency: 10.49 - AverageScannerThreadConcurrency: 9.89 - BytesRead: 2.61 GB (2804563939) - BytesReadDataNodeCache: 0 - BytesReadLocal: 2.43 GB (2612224351) - BytesReadRemoteUnexpected: 183.43 MB (192339588) - BytesReadShortCircuit: 2.43 GB (2612224351) - DecompressionTime: 0ns - MaxCompressedTextFileLength: 0 - NumColumns: 55 (55) - NumDisksAccessed: 10 (10) - NumScannerThreadsStarted: 17 (17) - PeakMemoryUsage: 1.36 GB (1465629624) - PerReadThreadRawHdfsThroughput: 13.32 MB/sec - RemoteScanRanges: 112 (112) - RowsRead: 35.75M (35745185) - RowsReturned: 0 (0) - RowsReturnedRate: 0 - ScanRangesComplete: 24 (24) - ScannerThreadsInvoluntaryContextSwitches: 1.75K (1752) - ScannerThreadsTotalWallClockTime: 3m6s - MaterializeTupleTime(*): 58s922ms - ScannerThreadsSysTime: 280.12ms - ScannerThreadsUserTime: 58s839ms - ScannerThreadsVoluntaryContextSwitches: 1.03K (1035) - TotalRawHdfsReadTime(*): 3m20s - TotalReadThroughput: 144.58 MB/sec Instance 704b7e71011e9573:13c1490afe0019b9 (host=hadoop4:22000):(Total: 15s643ms, non-child: 958.853us, % non-child: 0.01%) Hdfs split stats (<volume id>:<# splits>/<split lengths>): 11:1/129.56 MB 0:3/382.82 MB 3:3/252.99 MB 5:2/184.55 MB 7:3/310.95 MB 8:2/205.49 MB MemoryUsage(500.0ms): 75.78 MB, 192.04 MB, 316.14 MB, 428.84 MB, 503.28 MB, 480.35 MB, 627.36 MB, 530.13 MB, 502.11 MB, 442.27 MB, 354.66 MB, 473.23 MB, 489.97 MB, 535.28 MB, 442.60 MB, 402.28 MB, 395.24 MB, 404.04 MB, 416.96 MB, 442.66 MB, 452.68 MB, 481.41 MB, 471.95 MB, 424.20 MB, 494.05 MB, 471.53 MB, 523.33 MB, 399.42 MB, 272.36 MB, 183.12 MB, 120.26 MB ThreadUsage(500.0ms): 15, 15, 15, 15, 15, 14, 14, 12, 12, 12, 11, 10, 10, 10, 9, 8, 8, 7, 7, 7, 7, 7, 7, 6, 6, 6, 6, 5, 4, 3, 3 - AverageThreadTokens: 9.23 - PeakMemoryUsage: 652.39 MB (684083360) - PerHostPeakMemUsage: 652.39 MB (684083360) - PrepareTime: 450.221us - RowsProduced: 0 (0) - TotalCpuTime: 2m11s - TotalNetworkReceiveTime: 0ns - TotalNetworkSendTime: 104ns - TotalStorageWaitTime: 1m35s BlockMgr: - BlockWritesOutstanding: 0 (0) - BlocksCreated: 0 (0) - BlocksRecycled: 0 (0) - BufferedPins: 0 (0) - BytesWritten: 0 - MaxBlockSize: 8.00 MB (8388608) - MemoryLimit: 161.34 GB (173237125120) - PeakMemoryUsage: 0 - TotalBufferWaitTime: 0ns - TotalEncryptionTime: 0ns - TotalIntegrityCheckTime: 0ns - TotalReadBlockTime: 0ns DataStreamSender (dst_id=1):(Total: 22.22us, non-child: 22.22us, % non-child: 100.00%) - BytesSent: 0 - NetworkThroughput(*): 0.00 /sec - OverallThroughput: 0.00 /sec - PeakMemoryUsage: 4.00 KB (4096) - SerializeBatchTime: 0ns - ThriftTransmitTime(*): 0ns - UncompressedRowBatchSize: 0     ... View more

The problem is solved after upgrading CM. Thanks again. ... View more

Ok, i see, thanks again. ... View more

Thanks for your prompt reply. Aside from these patches that CDH5.1.0 has contained, how to upgrade the newer patches of HBase? must upgrade the newer CDH versions? ... View more

Ok, thank you, i will attempt to upgrade CM. ... View more

I have upgraded CDH from 5.0.1 to 5.1.0, also found Hbase becomes 0.98.1 version. I indeed need Hbase0.98.5, but don't find the parcels against it, so, how to upgrade it? or how to know that the current 0.98.1 has contained the patches about 0.98.5? ... View more

Thanks, i will try it out. ... View more