Reply
Highlighted
New Contributor
Posts: 3
Registered: ‎04-27-2018

Qlikview query Impala

Hello

We are testing que query impala from Qlikview throught CDH Odbc Impala Driver.

We have a table with 300M of records.

 

In QlikView we have configured direct query to consult the data directly throught impala using odbc connection.

We define an expresion using "aggr" function, this function is similar to group by.

 

We can see that when we execute in qlikview the aggr expresion it is generated in impala the next query:

 

 

SELECT `t21tmp_d2`.`jornada` FROM `peaje`.`t21tmp_d2` `t21tmp_d2`

 

this query is terribly slow.

the table t21tmp_d2 has arround 300M of records, it is stored in parquet files and it is partitioned by Year, Month and jornada.

 

this query last after 10 minutes or more, in the query summary we can see how it is reading all 300M of records slowly  in the 00 Scan Hdfs step and increasing the 01 Exchange Step

Operator       #Hosts   Avg Time   Max Time   #Rows  Est. #Rows  Peak Mem  Est. Peak Mem  Detail                    
--------------------------------------------------------------------------------------------------------------------
01:EXCHANGE         1  980.412ms  980.412ms  30.57M     337.75M  19.96 MB              0  UNPARTITIONED             
00:SCAN HDFS        3   83.986ms   85.133ms  32.36M     337.75M   6.00 MB        1.00 MB  peaje.t21tmp_d2 t21tmp_d2


and the query finish when the value of the #Rows column reach the value of Est. #Rows colum.

 

The Text Plain of the query is

Max Per-Host Resource Reservation: Memory=0B
Per-Host Resource Estimates: Memory=10.00MB

F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
|  Per-Host Resources: mem-estimate=0B mem-reservation=0B
PLAN-ROOT SINK
|  mem-estimate=0B mem-reservation=0B
|
01:EXCHANGE [UNPARTITIONED]
|  mem-estimate=0B mem-reservation=0B
|  tuple-ids=0 row-size=15B cardinality=337747134
|
F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
Per-Host Resources: mem-estimate=1.00MB mem-reservation=0B
00:SCAN HDFS [peaje.t21tmp_d2 t21tmp_d2, RANDOM]
   partitions=181/181 files=181 size=14.35GB
   stored statistics:
     table: rows=337747134 size=14.35GB
     partitions: 181/181 rows=337747134
     columns: all
   extrapolated-rows=disabled
   mem-estimate=1.00MB mem-reservation=0B
   tuple-ids=0 row-size=15B cardinality=337747134

 

the profile:

Query (id=94d49000a222a5c:735a206f00000000):
  Summary:
    Session ID: a54285fbaa8a00bc:f16cfc834c3e4796
    Session Type: HIVESERVER2
    HiveServer2 Protocol Version: V6
    Start Time: 2018-07-15 09:15:21.777688000
    End Time: 
    Query Type: QUERY
    Query State: FINISHED
    Query Status: OK
    Impala Version: impalad version 2.12.0-cdh5.15.0 RELEASE (build 23f574543323301846b41fa5433690df32efe085)
    User: P16280
    Connected User: P16280
    Delegated User: 
    Network Address: 10.2.96.33:50952
    Default Db: peaje
    Sql Statement: SELECT `t21tmp_d2`.`jornada` FROM `peaje`.`t21tmp_d2` `t21tmp_d2` 
    Coordinator: dallas20.metromadrid.net:22000
    Query Options (set by configuration): 
    Query Options (set by configuration and planner): MT_DOP=0
    Plan: 
----------------
Max Per-Host Resource Reservation: Memory=0B
Per-Host Resource Estimates: Memory=10.00MB

F01:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
|  Per-Host Resources: mem-estimate=0B mem-reservation=0B
PLAN-ROOT SINK
|  mem-estimate=0B mem-reservation=0B
|
01:EXCHANGE [UNPARTITIONED]
|  mem-estimate=0B mem-reservation=0B
|  tuple-ids=0 row-size=15B cardinality=337747134
|
F00:PLAN FRAGMENT [RANDOM] hosts=3 instances=3
Per-Host Resources: mem-estimate=1.00MB mem-reservation=0B
00:SCAN HDFS [peaje.t21tmp_d2 t21tmp_d2, RANDOM]
   partitions=181/181 files=181 size=14.35GB
   stored statistics:
     table: rows=337747134 size=14.35GB
     partitions: 181/181 rows=337747134
     columns: all
   extrapolated-rows=disabled
   mem-estimate=1.00MB mem-reservation=0B
   tuple-ids=0 row-size=15B cardinality=337747134
----------------
    Estimated Per-Host Mem: 10485760
    Per Host Min Reservation: dallas20.metromadrid.net:22000(0) dallas21.metromadrid.net:22000(0) dallas22.metromadrid.net:22000(0) 
    Request Pool: root.P16280
    Admission result: Admitted immediately
    Query Compilation: 5.272ms
       - Metadata of all 1 tables cached: 221.221us (221.221us)
       - Analysis finished: 447.346us (226.125us)
       - Value transfer graph computed: 476.309us (28.963us)
       - Single node plan created: 2.632ms (2.155ms)
       - Runtime filters computed: 2.673ms (40.954us)
       - Distributed plan created: 2.695ms (22.458us)
       - Lineage info computed: 2.723ms (27.830us)
       - Planning finished: 5.272ms (2.549ms)
    Query Timeline: 3m52s
       - Query submitted: 39.805us (39.805us)
       - Planning finished: 9.955ms (9.916ms)
       - Submit for admission: 11.969ms (2.013ms)
       - Completed admission: 12.044ms (74.919us)
       - Ready to start on 3 backends: 15.465ms (3.420ms)
       - All 3 execution backends (4 fragment instances) started: 22.720ms (7.255ms)
       - Rows available: 54.642ms (31.922ms)
       - First row fetched: 60.542ms (5.899ms)
     - ComputeScanRangeAssignmentTimer: 280.888us
  ImpalaServer:
     - ClientFetchWaitTimer: 3m36s
     - RowMaterializationTimer: 15s631ms
  Execution Profile 94d49000a222a5c:735a206f00000000:(Total: 15s580ms, non-child: 0.000ns, % non-child: 0.00%)
    Number of filters: 0
    Filter routing table: 
 ID  Src. Node  Tgt. Node(s)  Target type  Partition filter  Pending (Expected)  First arrived  Completed   Enabled
-------------------------------------------------------------------------------------------------------------------

    Backend startup latencies: Count: 3, min / max: 4ms / 5ms, 25th %-ile: 4ms, 50th %-ile: 5ms, 75th %-ile: 5ms, 90th %-ile: 5ms, 95th %-ile: 5ms, 99.9th %-ile: 5ms
     - FiltersReceived: 0 (0)
     - FinalizationTimer: 0.000ns
     - NumBackends: 3 (3)
     - NumFragmentInstances: 4 (4)
     - NumFragments: 2 (2)
    Averaged Fragment F01:
    Coordinator Fragment F01:
      Instance 94d49000a222a5c:735a206f00000000 (host=dallas20.metromadrid.net:22000):(Total: 31.142ms, non-child: 0.000ns, % non-child: 0.00%)
        Fragment Instance Lifecycle Event Timeline: 37.649ms
           - Prepare Finished: 615.700us (615.700us)
           - Open Finished: 31.435ms (30.819ms)
           - First Batch Produced: 31.465ms (30.331us)
           - First Batch Sent: 37.649ms (6.183ms)
        MemoryUsage(4s000ms): 19.97 MB, 19.97 MB, 19.97 MB, 19.96 MB, 19.96 MB, 19.97 MB, 19.97 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.97 MB, 19.97 MB, 19.97 MB, 19.96 MB, 19.97 MB, 19.97 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.97 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.97 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.97 MB, 19.97 MB, 19.96 MB, 19.97 MB, 19.97 MB, 19.96 MB, 19.96 MB, 19.97 MB, 19.96 MB, 19.97 MB, 19.96 MB, 19.96 MB, 19.97 MB, 19.97 MB, 19.97 MB, 19.96 MB, 19.96 MB, 19.97 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.96 MB, 19.96 MB
         - AverageThreadTokens: 0.00 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 19.97 MB (20935028)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 22.21 MB (23289974)
         - RowsProduced: 83.54M (83539968)
         - TotalNetworkReceiveTime: 371.948ms
         - TotalNetworkSendTime: 0.000ns
         - TotalStorageWaitTime: 0.000ns
         - TotalThreadsInvoluntaryContextSwitches: 0 (0)
         - TotalThreadsTotalWallClockTime: 30.810ms
           - TotalThreadsSysTime: 0.000ns
           - TotalThreadsUserTime: 16.163ms
         - TotalThreadsVoluntaryContextSwitches: 7 (7)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 0.000ns
             - ExecTreeExecTime: 2s725ms
           - OpenTime: 30.812ms
             - ExecTreeOpenTime: 14.622ms
           - PrepareTime: 293.401us
             - ExecTreePrepareTime: 80.684us
        PLAN_ROOT_SINK:
           - PeakMemoryUsage: 0
        EXCHANGE_NODE (id=1):(Total: 2s657ms, non-child: 2s642ms, % non-child: 99.45%)
           - ConvertRowBatchTime: 2s197ms
           - PeakMemoryUsage: 19.96 MB (20926836)
           - RowsReturned: 83.54M (83539968)
           - RowsReturnedRate: 31.44 M/sec
          Buffer pool:
             - AllocTime: 139.437ms
             - CumulativeAllocationBytes: 1.26 GB (1357529088)
             - CumulativeAllocations: 165.71K (165714)
             - PeakReservation: 19.95 MB (20914176)
             - PeakUnpinnedBytes: 0
             - PeakUsedReservation: 19.95 MB (20914176)
             - ReadIoBytes: 0
             - ReadIoOps: 0 (0)
             - ReadIoWaitTime: 0.000ns
             - WriteIoBytes: 0
             - WriteIoOps: 0 (0)
             - WriteIoWaitTime: 0.000ns
          Dequeue:
            BytesDequeued(4s000ms): 5.05 MB, 12.46 MB, 21.10 MB, 30.84 MB, 40.49 MB, 51.51 MB, 62.37 MB, 73.59 MB, 84.66 MB, 95.84 MB, 106.88 MB, 117.82 MB, 129.29 MB, 140.63 MB, 151.73 MB, 163.23 MB, 174.35 MB, 185.61 MB, 197.25 MB, 208.48 MB, 219.94 MB, 231.48 MB, 242.52 MB, 253.59 MB, 264.67 MB, 275.24 MB, 286.05 MB, 296.85 MB, 307.92 MB, 319.17 MB, 330.12 MB, 340.89 MB, 351.94 MB, 362.73 MB, 373.65 MB, 384.89 MB, 396.02 MB, 407.67 MB, 419.32 MB, 430.70 MB, 442.36 MB, 453.63 MB, 464.44 MB, 475.55 MB, 486.62 MB, 497.90 MB, 509.05 MB, 519.85 MB, 531.31 MB, 542.63 MB, 553.70 MB, 565.20 MB, 576.87 MB, 588.24 MB, 599.96 MB, 611.42 MB, 622.87 MB, 634.40 MB
             - FirstBatchWaitTime: 14.615ms
             - TotalBytesDequeued: 639.46 MB (670522458)
             - TotalGetBatchTime: 349.277ms
               - DataWaitTime: 14.616ms
          Enqueue:
            BytesReceived(4s000ms): 7.55 MB, 11.26 MB, 15.59 MB, 20.47 MB, 25.30 MB, 30.82 MB, 36.27 MB, 41.89 MB, 47.44 MB, 53.04 MB, 58.57 MB, 64.05 MB, 69.80 MB, 75.48 MB, 81.04 MB, 86.80 MB, 92.38 MB, 98.02 MB, 103.84 MB, 109.47 MB, 115.22 MB, 121.00 MB, 126.53 MB, 132.08 MB, 137.63 MB, 142.93 MB, 148.34 MB, 153.75 MB, 159.30 MB, 164.93 MB, 170.42 MB, 175.81 MB, 181.35 MB, 186.76 MB, 192.23 MB, 197.86 MB, 203.44 MB, 209.27 MB, 215.11 MB, 220.81 MB, 226.65 MB, 232.30 MB, 237.72 MB, 243.29 MB, 248.83 MB, 254.48 MB, 260.07 MB, 265.48 MB, 271.22 MB, 276.89 MB, 282.44 MB, 288.20 MB, 294.05 MB, 299.75 MB, 305.62 MB, 311.36 MB, 317.10 MB, 322.87 MB
            DeferredQueueSize(4s000ms): 3, 3, 3, 2, 2, 3, 3, 2, 2, 2, 2, 2, 3, 3, 3, 2, 3, 3, 2, 2, 2, 2, 2, 2, 3, 2, 2, 2, 3, 2, 2, 2, 2, 2, 2, 3, 2, 2, 3, 3, 2, 2, 3, 2, 3, 2, 2, 3, 3, 3, 2, 2, 3, 2, 2, 2, 2, 2
             - DispatchTime: (Avg: 40.891us ; Min: 4.452us ; Max: 2.425ms ; Number of samples: 82860)
             - DeserializeRowBatchTime: 1s416ms
             - TotalBatchesEnqueued: 82.86K (82857)
             - TotalBatchesReceived: 82.86K (82860)
             - TotalBytesReceived: 325.41 MB (341217480)
             - TotalEarlySenders: 0 (0)
             - TotalEosReceived: 0 (0)
             - TotalHasDeferredRPCsTime: 0.000ns
             - TotalRPCsDeferred: 81.56K (81565)
        CodeGen:(Total: 16.118ms, non-child: 16.118ms, % non-child: 100.00%)
           - CodegenTime: 0.000ns
           - CompileTime: 0.000ns
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 0 (0)
           - NumInstructions: 0 (0)
           - OptimizationTime: 0.000ns
           - PeakMemoryUsage: 0
           - PrepareTime: 15.623ms
    Averaged Fragment F00:
    Fragment F00:
      Instance 94d49000a222a5c:735a206f00000001 (host=dallas20.metromadrid.net:22000):(Total: 29.577ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 3:5/440.21 MB 1:2/99.12 MB 14:1/60.23 MB 6:2/176.08 MB 13:2/137.34 MB 8:6/492.91 MB 4:6/488.49 MB 2:4/338.51 MB 0:4/299.73 MB 12:7/541.10 MB 10:6/539.35 MB 9:3/230.94 MB 11:6/441.64 MB 5:3/225.81 MB 7:5/374.77 MB 
        Fragment Instance Lifecycle Event Timeline: 34.021ms
           - Prepare Finished: 1.376ms (1.376ms)
           - Open Finished: 29.977ms (28.601ms)
           - First Batch Produced: 33.983ms (4.005ms)
           - First Batch Sent: 34.021ms (37.371us)
        MemoryUsage(4s000ms): 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB
        ThreadUsage(4s000ms): 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57
         - AverageThreadTokens: 57.00 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 5.84 MB (6119170)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 22.21 MB (23289974)
         - RowsProduced: 28.00M (27996160)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 3m46s
         - TotalStorageWaitTime: 33.547ms
         - TotalThreadsInvoluntaryContextSwitches: 0 (0)
         - TotalThreadsTotalWallClockTime: 28.610ms
           - TotalThreadsSysTime: 1.966ms
           - TotalThreadsUserTime: 26.499ms
         - TotalThreadsVoluntaryContextSwitches: 9 (9)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 0.000ns
             - ExecTreeExecTime: 238.723ms
           - OpenTime: 28.613ms
             - ExecTreeOpenTime: 61.765us
           - PrepareTime: 901.555us
             - ExecTreePrepareTime: 499.306us
        KrpcDataStreamSender (dst_id=1):(Total: 3m47s, non-child: 1s202ms, % non-child: 0.53%)
          BytesSent(4s000ms): 3.28 MB, 4.52 MB, 5.96 MB, 7.59 MB, 9.20 MB, 11.04 MB, 12.85 MB, 14.73 MB, 16.58 MB, 18.45 MB, 20.29 MB, 22.12 MB, 24.03 MB, 25.93 MB, 27.78 MB, 29.70 MB, 31.56 MB, 33.44 MB, 35.38 MB, 37.26 MB, 39.17 MB, 41.10 MB, 42.94 MB, 44.79 MB, 46.64 MB, 48.41 MB, 50.21 MB, 52.02 MB, 53.87 MB, 55.75 MB, 57.57 MB, 59.37 MB, 61.22 MB, 63.02 MB, 64.84 MB, 66.72 MB, 68.58 MB, 70.53 MB, 72.47 MB, 74.37 MB, 76.32 MB, 78.20 MB, 80.01 MB, 81.86 MB, 83.71 MB, 85.59 MB, 87.46 MB, 89.26 MB, 91.17 MB, 93.07 MB, 94.91 MB, 96.83 MB, 98.78 MB, 100.68 MB, 102.64 MB, 104.55 MB, 106.47 MB
           - EosSent: 0 (0)
           - NetworkThroughput: 23.84 MB/sec
           - PeakMemoryUsage: 7.52 KB (7704)
           - RowsSent: 28.00M (27995136)
           - RpcFailure: 0 (0)
           - RpcRetry: 0 (0)
           - SerializeBatchTime: 612.604ms
           - TotalBytesSent: 107.36 MB (112577884)
           - UncompressedRowBatchSize: 214.30 MB (224707460)
        HDFS_SCAN_NODE (id=0):(Total: 196.622ms, non-child: 196.622ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 3:5/440.21 MB 1:2/99.12 MB 14:1/60.23 MB 6:2/176.08 MB 13:2/137.34 MB 8:6/492.91 MB 4:6/488.49 MB 2:4/338.51 MB 0:4/299.73 MB 12:7/541.10 MB 10:6/539.35 MB 9:3/230.94 MB 11:6/441.64 MB 5:3/225.81 MB 7:5/374.77 MB 
          ExecOption: PARQUET Codegen Enabled
          BytesRead(4s000ms): 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB
           - FooterProcessingTime: (Avg: 949.403us ; Min: 373.127us ; Max: 1.738ms ; Number of samples: 56)
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 56.00 
           - BytesRead: 5.47 MB (5734400)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandle**bleep**Count: 0 (0)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumColumns: 1 (1)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 0 (0)
           - NumRowGroups: 0 (0)
           - NumScannerThreadsStarted: 56 (56)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 5.82 MB (6103040)
           - PerReadThreadRawHdfsThroughput: 495.03 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 0.000ns
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 6.98B (6979653632)
           - RowsReturned: 28.00M (27996160)
           - RowsReturnedRate: 142.39 M/sec
           - ScanRangesComplete: 0 (0)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 0.000ns
             - MaterializeTupleTime(*): 0.000ns
             - ScannerThreadsSysTime: 0.000ns
             - ScannerThreadsUserTime: 0.000ns
           - ScannerThreadsVoluntaryContextSwitches: 0 (0)
           - TotalRawHdfsOpenFileTime(*): 481.422us
           - TotalRawHdfsReadTime(*): 11.047ms
           - TotalReadThroughput: 24.56 KB/sec
        CodeGen:(Total: 28.222ms, non-child: 28.222ms, % non-child: 100.00%)
           - CodegenTime: 480.026us
           - CompileTime: 4.087ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 6.828ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 16.848ms
      Instance 94d49000a222a5c:735a206f00000003 (host=dallas21.metromadrid.net:22000):(Total: 26.556ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 3:5/335.70 MB 7:3/183.09 MB 1:3/261.11 MB 2:4/225.30 MB 11:3/180.60 MB 8:2/192.45 MB 12:3/252.58 MB 14:9/664.13 MB 13:5/439.53 MB 9:4/317.54 MB 10:3/301.14 MB 6:5/413.60 MB 0:6/441.23 MB 4:3/274.77 MB 5:6/402.00 MB 
        Fragment Instance Lifecycle Event Timeline: 30.775ms
           - Prepare Finished: 1.233ms (1.233ms)
           - Open Finished: 26.881ms (25.648ms)
           - First Batch Produced: 30.732ms (3.850ms)
           - First Batch Sent: 30.775ms (43.467us)
        MemoryUsage(4s000ms): 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB, 2.24 MB
        ThreadUsage(4s000ms): 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57, 57
         - AverageThreadTokens: 57.00 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 6.01 MB (6303490)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 6.01 MB (6303490)
         - RowsProduced: 27.83M (27833344)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 3m47s
         - TotalStorageWaitTime: 22.094ms
         - TotalThreadsInvoluntaryContextSwitches: 0 (0)
         - TotalThreadsTotalWallClockTime: 25.645ms
           - TotalThreadsSysTime: 966.000us
           - TotalThreadsUserTime: 24.050ms
         - TotalThreadsVoluntaryContextSwitches: 0 (0)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 0.000ns
             - ExecTreeExecTime: 255.479ms
           - OpenTime: 25.647ms
             - ExecTreeOpenTime: 79.767us
           - PrepareTime: 846.347us
             - ExecTreePrepareTime: 453.427us
        KrpcDataStreamSender (dst_id=1):(Total: 3m48s, non-child: 1s325ms, % non-child: 0.58%)
          BytesSent(4s000ms): 2.16 MB, 3.39 MB, 4.84 MB, 6.46 MB, 8.07 MB, 9.91 MB, 11.73 MB, 13.60 MB, 15.45 MB, 17.32 MB, 19.16 MB, 20.99 MB, 22.91 MB, 24.80 MB, 26.66 MB, 28.58 MB, 30.43 MB, 32.32 MB, 34.26 MB, 36.13 MB, 38.05 MB, 39.97 MB, 41.82 MB, 43.67 MB, 45.52 MB, 47.28 MB, 49.09 MB, 50.89 MB, 52.74 MB, 54.62 MB, 56.45 MB, 58.25 MB, 60.09 MB, 61.90 MB, 63.72 MB, 65.60 MB, 67.46 MB, 69.40 MB, 71.35 MB, 73.25 MB, 75.19 MB, 77.08 MB, 78.88 MB, 80.74 MB, 82.59 MB, 84.47 MB, 86.33 MB, 88.14 MB, 90.05 MB, 91.94 MB, 93.79 MB, 95.71 MB, 97.66 MB, 99.56 MB, 101.52 MB, 103.43 MB, 105.34 MB
           - EosSent: 0 (0)
           - NetworkThroughput: 15.34 MB/sec
           - PeakMemoryUsage: 7.52 KB (7704)
           - RowsSent: 27.83M (27832320)
           - RpcFailure: 0 (0)
           - RpcRetry: 0 (0)
           - SerializeBatchTime: 705.609ms
           - TotalBytesSent: 106.74 MB (111923122)
           - UncompressedRowBatchSize: 213.05 MB (223400639)
        HDFS_SCAN_NODE (id=0):(Total: 205.120ms, non-child: 205.120ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 3:5/335.70 MB 7:3/183.09 MB 1:3/261.11 MB 2:4/225.30 MB 11:3/180.60 MB 8:2/192.45 MB 12:3/252.58 MB 14:9/664.13 MB 13:5/439.53 MB 9:4/317.54 MB 10:3/301.14 MB 6:5/413.60 MB 0:6/441.23 MB 4:3/274.77 MB 5:6/402.00 MB 
          ExecOption: PARQUET Codegen Enabled
          BytesRead(4s000ms): 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB, 5.47 MB
           - FooterProcessingTime: (Avg: 880.238us ; Min: 390.969us ; Max: 1.831ms ; Number of samples: 56)
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 56.00 
           - BytesRead: 5.47 MB (5734400)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandle**bleep**Count: 0 (0)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumColumns: 1 (1)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 0 (0)
           - NumRowGroups: 0 (0)
           - NumScannerThreadsStarted: 56 (56)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 6.00 MB (6287360)
           - PerReadThreadRawHdfsThroughput: 510.51 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 0.000ns
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 6.90B (6899208192)
           - RowsReturned: 27.83M (27833344)
           - RowsReturnedRate: 135.69 M/sec
           - ScanRangesComplete: 0 (0)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 0.000ns
             - MaterializeTupleTime(*): 0.000ns
             - ScannerThreadsSysTime: 0.000ns
             - ScannerThreadsUserTime: 0.000ns
           - ScannerThreadsVoluntaryContextSwitches: 0 (0)
           - TotalRawHdfsOpenFileTime(*): 756.176us
           - TotalRawHdfsReadTime(*): 10.712ms
           - TotalReadThroughput: 24.45 KB/sec
        CodeGen:(Total: 25.272ms, non-child: 25.272ms, % non-child: 100.00%)
           - CodegenTime: 391.123us
           - CompileTime: 4.080ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 6.347ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 14.345ms
      Instance 94d49000a222a5c:735a206f00000002 (host=dallas22.metromadrid.net:22000):(Total: 27.038ms, non-child: 0.000ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 12:2/194.71 MB 3:2/202.11 MB 11:3/269.69 MB 0:3/240.29 MB 2:4/339.27 MB 13:4/345.24 MB 10:3/288.28 MB 9:5/443.29 MB 14:2/194.92 MB 7:4/358.52 MB 4:3/231.01 MB 6:6/596.06 MB 1:7/584.24 MB 5:3/288.02 MB 8:4/347.43 MB 
        Fragment Instance Lifecycle Event Timeline: 31.340ms
           - Prepare Finished: 1.107ms (1.107ms)
           - Open Finished: 27.287ms (26.179ms)
           - First Batch Produced: 31.275ms (3.988ms)
           - First Batch Sent: 31.340ms (64.878us)
        MemoryUsage(4s000ms): 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB, 2.23 MB
        ThreadUsage(4s000ms): 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56, 56
         - AverageThreadTokens: 56.00 
         - BloomFilterBytes: 0
         - PeakMemoryUsage: 5.02 MB (5263106)
         - PeakReservation: 0
         - PeakUsedReservation: 0
         - PerHostPeakMemUsage: 5.02 MB (5263106)
         - RowsProduced: 27.69M (27685888)
         - TotalNetworkReceiveTime: 0.000ns
         - TotalNetworkSendTime: 3m46s
         - TotalStorageWaitTime: 31.167ms
         - TotalThreadsInvoluntaryContextSwitches: 0 (0)
         - TotalThreadsTotalWallClockTime: 26.176ms
           - TotalThreadsSysTime: 1.950ms
           - TotalThreadsUserTime: 23.431ms
         - TotalThreadsVoluntaryContextSwitches: 0 (0)
        Buffer pool:
           - AllocTime: 0.000ns
           - CumulativeAllocationBytes: 0
           - CumulativeAllocations: 0 (0)
           - PeakReservation: 0
           - PeakUnpinnedBytes: 0
           - PeakUsedReservation: 0
           - ReadIoBytes: 0
           - ReadIoOps: 0 (0)
           - ReadIoWaitTime: 0.000ns
           - ReservationLimit: 0
           - WriteIoBytes: 0
           - WriteIoOps: 0 (0)
           - WriteIoWaitTime: 0.000ns
        Fragment Instance Lifecycle Timings:
           - ExecTime: 0.000ns
             - ExecTreeExecTime: 259.458ms
           - OpenTime: 26.179ms
             - ExecTreeOpenTime: 75.959us
           - PrepareTime: 821.105us
             - ExecTreePrepareTime: 428.111us
        KrpcDataStreamSender (dst_id=1):(Total: 3m47s, non-child: 1s332ms, % non-child: 0.59%)
          BytesSent(4s000ms): 1.97 MB, 3.27 MB, 4.63 MB, 6.31 MB, 7.88 MB, 9.72 MB, 11.54 MB, 13.40 MB, 15.25 MB, 17.12 MB, 18.96 MB, 20.79 MB, 22.70 MB, 24.59 MB, 26.45 MB, 28.37 MB, 30.23 MB, 32.11 MB, 34.05 MB, 35.93 MB, 37.84 MB, 39.77 MB, 41.62 MB, 43.47 MB, 45.31 MB, 47.09 MB, 48.90 MB, 50.69 MB, 52.54 MB, 54.42 MB, 56.25 MB, 58.05 MB, 59.89 MB, 61.69 MB, 63.52 MB, 65.39 MB, 67.25 MB, 69.20 MB, 71.14 MB, 73.04 MB, 74.99 MB, 76.87 MB, 78.69 MB, 80.54 MB, 82.38 MB, 84.27 MB, 86.13 MB, 87.93 MB, 89.84 MB, 91.74 MB, 93.59 MB, 95.51 MB, 97.45 MB, 99.35 MB, 101.31 MB, 103.22 MB, 105.14 MB
           - EosSent: 0 (0)
           - NetworkThroughput: 12.77 MB/sec
           - PeakMemoryUsage: 7.52 KB (7704)
           - RowsSent: 27.68M (27684864)
           - RpcFailure: 0 (0)
           - RpcRetry: 0 (0)
           - SerializeBatchTime: 690.282ms
           - TotalBytesSent: 106.17 MB (111330130)
           - UncompressedRowBatchSize: 211.92 MB (222217103)
        HDFS_SCAN_NODE (id=0):(Total: 213.310ms, non-child: 213.310ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 12:2/194.71 MB 3:2/202.11 MB 11:3/269.69 MB 0:3/240.29 MB 2:4/339.27 MB 13:4/345.24 MB 10:3/288.28 MB 9:5/443.29 MB 14:2/194.92 MB 7:4/358.52 MB 4:3/231.01 MB 6:6/596.06 MB 1:7/584.24 MB 5:3/288.02 MB 8:4/347.43 MB 
          ExecOption: PARQUET Codegen Enabled
          BytesRead(4s000ms): 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB, 5.37 MB
           - FooterProcessingTime: (Avg: 1.029ms ; Min: 440.621us ; Max: 1.982ms ; Number of samples: 55)
           - AverageHdfsReadThreadConcurrency: 0.00 
           - AverageScannerThreadConcurrency: 55.00 
           - BytesRead: 5.37 MB (5632000)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadRemoteUnexpected: 0
           - BytesReadShortCircuit: 0
           - CachedFileHandle**bleep**Count: 0 (0)
           - CachedFileHandlesMissCount: 0 (0)
           - CollectionItemsRead: 0 (0)
           - DecompressionTime: 0.000ns
           - MaxCompressedTextFileLength: 0
           - NumColumns: 1 (1)
           - NumDictFilteredRowGroups: 0 (0)
           - NumDisksAccessed: 0 (0)
           - NumRowGroups: 0 (0)
           - NumScannerThreadsStarted: 55 (55)
           - NumScannersWithNoReads: 0 (0)
           - NumStatsFilteredRowGroups: 0 (0)
           - PeakMemoryUsage: 5.00 MB (5246976)
           - PerReadThreadRawHdfsThroughput: 390.88 MB/sec
           - RemoteScanRanges: 0 (0)
           - RowBatchQueueGetWaitTime: 0.000ns
           - RowBatchQueuePutWaitTime: 0.000ns
           - RowsRead: 6.95B (6950144000)
           - RowsReturned: 27.69M (27685888)
           - RowsReturnedRate: 129.79 M/sec
           - ScanRangesComplete: 0 (0)
           - ScannerThreadsInvoluntaryContextSwitches: 0 (0)
           - ScannerThreadsTotalWallClockTime: 0.000ns
             - MaterializeTupleTime(*): 0.000ns
             - ScannerThreadsSysTime: 0.000ns
             - ScannerThreadsUserTime: 0.000ns
           - ScannerThreadsVoluntaryContextSwitches: 0 (0)
           - TotalRawHdfsOpenFileTime(*): 1.141ms
           - TotalRawHdfsReadTime(*): 13.741ms
           - TotalReadThroughput: 24.12 KB/sec
        CodeGen:(Total: 25.823ms, non-child: 25.823ms, % non-child: 100.00%)
           - CodegenTime: 404.617us
           - CompileTime: 4.113ms
           - LoadTime: 0.000ns
           - ModuleBitcodeSize: 1.95 MB (2039944)
           - NumFunctions: 7 (7)
           - NumInstructions: 105 (105)
           - OptimizationTime: 6.504ms
           - PeakMemoryUsage: 52.50 KB (53760)
           - PrepareTime: 14.701ms



 

anybody know why this query run so slow?

anybody could help me?

 

anybody has experience in consulting impala data from Qlikview?

anybody know if there is a better way to consult imapala data interactly by a bussines user?

 

Thanks in advance for your help

thanks

regards

Announcements