Reply
New Contributor
Posts: 5
Registered: ‎11-07-2017

long codegen time while codegen disabled

 I have a query,tooks a long time on codegen:

      CodeGen:(Total: 32m22s, non-child: 32m22s, % non-child: 100.00%)
         - CodegenTime: 0ns
         - CompileTime: 53.143ms
         - LoadTime: 58.680us
         - ModuleFileSize: 1.96 MB (2054956)
         - OptimizationTime: 32m22s
         - PrepareTime: 157.700ms

but from the profile ,we can see that codegen is diabled for this query:

ExecOption: Codegen enabled: 0 out of 1

 

can anyone help to firgure out a way to bypass.

 

 

Query (id=a84436eacc99694c:4870859f6ca7d094):
  Summary:
    Session ID: 9245a5d4802eafa6:2d08911d4ea694b2
    Session Type: HIVESERVER2
    HiveServer2 Protocol Version: V6
    Start Time: 2017-11-07 16:29:13.181284000
    End Time: 2017-11-07 17:01:36.300620000
    Query Type: DML
    Query State: FINISHED
    Query Status: OK
    Impala Version: impalad version 2.1.0-cdh5 RELEASE (build e48c2b48c53ea9601b8f47a39373aa8-PetaBase)
    User: 
    Connected User: 
    Delegated User: 
    Network Address: ::ffff:141.1.9.32:45361
    Default Db: db_np
    Sql Statement: INSERT INTO `db_np`.`t_ccbq_all_change` SELECT `v_msccbq`.`c_bh`, `v_msccbq`.`c_ajbh`, `v_msccbq`.`c_lx`, `v_msccbq`.`c_bqfdf`, `v_msccbq`.`c_sqdlr`, `v_msccbq`.`c_dlrlxfs`, `v_msccbq`.`c_sqrq`, `v_msccbq`.`n_sqje`, `v_msccbq`.`c_sqcc`, `v_msccbq`.`c_cdrq`, `v_msccbq`.`c_cdjg`, `v_msccbq`.`c_cdssd`, `v_msccbq`.`c_zs`, `v_msccbq`.`c_tzpsrq`, `v_msccbq`.`c_tzps`, `v_msccbq`.`c_yzpsrq`, `v_msccbq`.`c_yzps`, `v_msccbq`.`c_xwnr`, `v_msccbq`.`c_zxrq`, `v_msccbq`.`n_sjzxje`, `v_msccbq`.`c_jcbqrq`, `v_msccbq`.`c_zjbh`, `v_msccbq`.`n_xh`, `v_msccbq`.`n_bqqx`, `v_msccbq`.`c_sqxbrq`, `v_msccbq`.`n_sqbqf`, `v_msccbq`.`c_pmggrq`, `v_msccbq`.`c_pmggqj`, `v_msccbq`.`c_yszt`, `v_msccbq`.`c_ssbqcdws`, `v_msccbq`.`c_sfcszsfpt`, `v_msccbq`.`c_jfzt`, `v_msccbq`.`c_ssbqsqly`, `v_msccbq`.`c_jcbqsqly`, `v_msccbq`.`c_sqjcsj`, `v_msccbq`.`c_fyid`, `v_msccbq`.`c_fy`, `v_msccbq`.`n_fydm`, `v_msccbq`.`n_nply` FROM `db_np`.`v_msccbq`
    Coordinator: master:22000
    Plan: 
----------------
Estimated Per-Host Requirements: Memory=320.01MB VCores=9
WARNING: The following tables are missing relevant table and/or column statistics.
db_np.t_msccbq

F02:PLAN FRAGMENT [HASH(n_dm)]
  WRITE TO HDFS [db_np.t_ccbq_all_change, OVERWRITE=false]
  |  partitions=1
  |  hosts=3 per-host-mem=6.96KB
  |
  16:HASH JOIN [LEFT OUTER JOIN, BROADCAST]
  |  hash predicates: ccbq.n_jbfy = fy.n_fybs
  |  other predicates: fy.c_fyid IS NOT NULL
  |  hosts=3 per-host-mem=1.02KB
  |  tuple-ids=1N,0,3N,5N,7N,9N,11N,13N,14N row-size=1019B cardinality=21
  |
  |--25:EXCHANGE [BROADCAST]
  |     hosts=3 per-host-mem=0B
  |     tuple-ids=14 row-size=73B cardinality=13
  |
  15:HASH JOIN [LEFT OUTER JOIN, BROADCAST]
  |  hash predicates: ccbq.n_jfzt = jfzt.n_dm
  |  hosts=3 per-host-mem=128B
  |  tuple-ids=1N,0,3N,5N,7N,9N,11N,13N row-size=945B cardinality=21
  |
  |--24:EXCHANGE [BROADCAST]
  |     hosts=3 per-host-mem=0B
  |     tuple-ids=13 row-size=29B cardinality=4
  |
  14:HASH JOIN [LEFT OUTER JOIN, BROADCAST]
  |  hash predicates: ccbq.n_jcbqsqly = n_dm
  |  hosts=3 per-host-mem=1.82KB
  |  tuple-ids=1N,0,3N,5N,7N,9N,11N row-size=916B cardinality=21
  |
  |--23:EXCHANGE [BROADCAST]
  |     hosts=3 per-host-mem=0B
  |     tuple-ids=11 row-size=81B cardinality=21
  |
  13:HASH JOIN [LEFT OUTER JOIN, BROADCAST]
  |  hash predicates: ccbq.n_ssbqsqly = n_dm
  |  hosts=3 per-host-mem=1.82KB
  |  tuple-ids=1N,0,3N,5N,7N,9N row-size=836B cardinality=21
  |
  |--22:EXCHANGE [BROADCAST]
  |     hosts=3 per-host-mem=0B
  |     tuple-ids=9 row-size=81B cardinality=21
  |
  12:HASH JOIN [LEFT OUTER JOIN, BROADCAST]
  |  hash predicates: ccbq.n_yszt = n_dm
  |  hosts=3 per-host-mem=1.82KB
  |  tuple-ids=1N,0,3N,5N,7N row-size=755B cardinality=21
  |
  |--21:EXCHANGE [BROADCAST]
  |     hosts=3 per-host-mem=0B
  |     tuple-ids=7 row-size=81B cardinality=21
  |
  11:HASH JOIN [LEFT OUTER JOIN, BROADCAST]
  |  hash predicates: ccbq.n_cdjg = n_dm
  |  hosts=3 per-host-mem=1.82KB
  |  tuple-ids=1N,0,3N,5N row-size=674B cardinality=21
  |
  |--20:EXCHANGE [BROADCAST]
  |     hosts=3 per-host-mem=0B
  |     tuple-ids=5 row-size=81B cardinality=21
  |
  10:HASH JOIN [LEFT OUTER JOIN, BROADCAST]
  |  hash predicates: ccbq.n_lx = n_dm
  |  hosts=3 per-host-mem=1.82KB
  |  tuple-ids=1N,0,3N row-size=593B cardinality=21
  |
  |--19:EXCHANGE [BROADCAST]
  |     hosts=3 per-host-mem=0B
  |     tuple-ids=3 row-size=81B cardinality=21
  |
  09:HASH JOIN [RIGHT OUTER JOIN, PARTITIONED]
  |  hash predicates: n_dm = ccbq.n_bqfdf
  |  hosts=3 per-host-mem=0B
  |  tuple-ids=1N,0 row-size=513B cardinality=21
  |
  |--18:EXCHANGE [HASH(ccbq.n_bqfdf)]
  |     hosts=1 per-host-mem=0B
  |     tuple-ids=0 row-size=432B cardinality=0
  |
  17:EXCHANGE [HASH(n_dm)]
     hosts=3 per-host-mem=0B
     tuple-ids=1 row-size=81B cardinality=21

F09:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=25, BROADCAST]
  08:SCAN HDFS [db_np.d_fy fy, RANDOM]
     partitions=1/1 files=1 size=14.28KB
     predicates: fy.c_fyid IS NOT NULL
     table stats: 131 rows total
     column stats: all
     hosts=3 per-host-mem=48.00MB
     tuple-ids=14 row-size=73B cardinality=13

F08:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=24, BROADCAST]
  07:SCAN HDFS [db_np.d_jfzt jfzt, RANDOM]
     partitions=1/1 files=1 size=728B
     table stats: 4 rows total
     column stats: all
     hosts=3 per-host-mem=32.00MB
     tuple-ids=13 row-size=29B cardinality=4

F07:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=23, BROADCAST]
  06:SCAN HDFS [db_np.d_wd, RANDOM]
     partitions=1/1 files=1 size=89.39KB
     predicates: c_wdflmc = '收录来源'
     table stats: 9427 rows total
     column stats: all
     hosts=3 per-host-mem=48.00MB
     tuple-ids=11 row-size=81B cardinality=21

F06:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=22, BROADCAST]
  05:SCAN HDFS [db_np.d_wd, RANDOM]
     partitions=1/1 files=1 size=89.39KB
     predicates: c_wdflmc = '收录来源'
     table stats: 9427 rows total
     column stats: all
     hosts=3 per-host-mem=48.00MB
     tuple-ids=9 row-size=81B cardinality=21

F05:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=21, BROADCAST]
  04:SCAN HDFS [db_np.d_wd, RANDOM]
     partitions=1/1 files=1 size=89.39KB
     predicates: c_wdflmc = '移送处理结果'
     table stats: 9427 rows total
     column stats: all
     hosts=3 per-host-mem=48.00MB
     tuple-ids=7 row-size=81B cardinality=21

F04:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=20, BROADCAST]
  03:SCAN HDFS [db_np.d_wd, RANDOM]
     partitions=1/1 files=1 size=89.39KB
     predicates: c_wdflmc = '保全裁定结果'
     table stats: 9427 rows total
     column stats: all
     hosts=3 per-host-mem=48.00MB
     tuple-ids=5 row-size=81B cardinality=21

F03:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=19, BROADCAST]
  02:SCAN HDFS [db_np.d_wd, RANDOM]
     partitions=1/1 files=1 size=89.39KB
     predicates: c_wdflmc = '诉讼保全、先予执行类型'
     table stats: 9427 rows total
     column stats: all
     hosts=3 per-host-mem=48.00MB
     tuple-ids=3 row-size=81B cardinality=21

F01:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=18, HASH(ccbq.n_bqfdf)]
  00:SCAN HDFS [db_np.t_msccbq ccbq, RANDOM]
     partitions=1/1 files=1 size=0B
     table stats: 0 rows total
     column stats: all
     hosts=1 per-host-mem=0B
     tuple-ids=0 row-size=432B cardinality=0

F00:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F02, EXCHANGE=17, HASH(n_dm)]
  01:SCAN HDFS [db_np.d_wd, RANDOM]
     partitions=1/1 files=1 size=89.39KB
     predicates: c_wdflmc = '保全发动方'
     table stats: 9427 rows total
     column stats: all
     hosts=3 per-host-mem=48.00MB
     tuple-ids=1 row-size=81B cardinality=21
----------------
    Estimated Per-Host Mem: 335554820
    Estimated Per-Host VCores: 9
    Tables Missing Stats: db_np.t_msccbq
    Request Pool: default-pool
    ExecSummary: 
Operator          #Hosts  Avg Time  Max Time  #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail                        
-------------------------------------------------------------------------------------------------------------------------
16:HASH JOIN           1   7.673ms   7.673ms      0          21    6.08 MB        1.02 KB  LEFT OUTER JOIN, BROADCAST    
|--25:EXCHANGE         1  10.693us  10.693us    131          13          0              0  BROADCAST                     
|  08:SCAN HDFS        1   2.314ms   2.314ms    131          13   92.00 KB       48.00 MB  db_np.d_fy fy                 
15:HASH JOIN           1   6.702ms   6.702ms      0          21    4.07 MB       128.00 B  LEFT OUTER JOIN, BROADCAST    
|--24:EXCHANGE         1  10.159us  10.159us      4           4          0              0  BROADCAST                     
|  07:SCAN HDFS        1   1.876ms   1.876ms      4           4   47.00 KB       32.00 MB  db_np.d_jfzt jfzt             
14:HASH JOIN           1   6.700ms   6.700ms      0          21    4.06 MB        1.82 KB  LEFT OUTER JOIN, BROADCAST    
|--23:EXCHANGE         1   9.723us   9.723us      6          21          0              0  BROADCAST                     
|  06:SCAN HDFS        1    4.32ms    4.32ms      6          21  411.36 KB       48.00 MB  db_np.d_wd                    
13:HASH JOIN           1   6.707ms   6.707ms      0          21    4.05 MB        1.82 KB  LEFT OUTER JOIN, BROADCAST    
|--22:EXCHANGE         1   8.757us   8.757us      6          21          0              0  BROADCAST                     
|  05:SCAN HDFS        1   5.297ms   5.297ms      6          21  411.36 KB       48.00 MB  db_np.d_wd                    
12:HASH JOIN           1   6.974ms   6.974ms      0          21    4.04 MB        1.82 KB  LEFT OUTER JOIN, BROADCAST    
|--21:EXCHANGE         1   6.983us   6.983us      5          21          0              0  BROADCAST                     
|  04:SCAN HDFS        1   4.941ms   4.941ms      5          21  411.36 KB       48.00 MB  db_np.d_wd                    
11:HASH JOIN           1   7.131ms   7.131ms      0          21    4.04 MB        1.82 KB  LEFT OUTER JOIN, BROADCAST    
|--20:EXCHANGE         1    6.65us    6.65us      6          21          0              0  BROADCAST                     
|  03:SCAN HDFS        1   6.124ms   6.124ms      6          21  411.36 KB       48.00 MB  db_np.d_wd                    
10:HASH JOIN           1   6.161ms   6.161ms      0          21    4.03 MB        1.82 KB  LEFT OUTER JOIN, BROADCAST    
|--19:EXCHANGE         1   6.905us   6.905us      4          21          0              0  BROADCAST                     
|  02:SCAN HDFS        1   5.730ms   5.730ms      4          21  411.36 KB       48.00 MB  db_np.d_wd                    
09:HASH JOIN           1   9.653ms   9.653ms      0          21    4.02 MB              0  RIGHT OUTER JOIN, PARTITIONED 
|--18:EXCHANGE         1   3.480us   3.480us      0           0          0              0  HASH(ccbq.n_bqfdf)            
|  00:SCAN HDFS        1  76.574us  76.574us      0           0          0              0  db_np.t_msccbq ccbq           
17:EXCHANGE            1   8.877us   8.877us      2          21          0              0  HASH(n_dm)                    
01:SCAN HDFS           1   5.184ms   5.184ms      2          21  411.36 KB       48.00 MB  db_np.d_wd                    
    Query Timeline: 32m23s
       - Start execution: 34.463us (34.463us)
       - Planning finished: 14.539ms (14.505ms)
       - Ready to start remote fragments: 15.80ms (540.719us)
       - Remote fragments started: 191.228ms (176.148ms)
       - DML data written: 32m22s (32m22s)
       - DML Metastore update finished: 32m23s (155.94ms)
       - Request finished: 32m23s (259.744us)
       - Unregister query: 32m23s (1.465ms)
  ImpalaServer:
     - ClientFetchWaitTimer: 622.87us
     - MetastoreUpdateTimer: 155.342ms
     - RowMaterializationTimer: 0ns
  Execution Profile a84436eacc99694c:4870859f6ca7d094:(Total: 32m22s, non-child: 0ns, % non-child: 0.00%)
    Fragment start latencies:  count: 10, last: 0.00122889, min: 0.000756666, max: 0.164792, mean: 0.0174591, stddev: 0.0491115
    Insert Stats: 

 

 

Fragment F02:
      Instance a84436eacc99694c:4870859f6ca7d095 (host=slave1:22000):(Total: 187.927ms, non-child: 0ns, % non-child: 0.00%)
        MemoryUsage(32s000ms): 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB, 401.88 KB
        ThreadUsage(32s000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.00 
         - PeakMemoryUsage: 27.73 MB (29081648)
         - PerHostPeakMemUsage: 27.73 MB (29081648)
         - PrepareTime: 397.116us
         - RowsProduced: 0
         - TotalCpuTime: 32m22s
         - TotalNetworkReceiveTime: 21.425us
         - TotalNetworkSendTime: 0ns
         - TotalStorageWaitTime: 0ns
        BlockMgr:
           - BlockWritesOutstanding: 0
           - BlocksCreated: 437
           - BlocksRecycled: 138
           - BufferedPins: 0
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 85.69 GB (92008636416)
           - PeakMemoryUsage: 3.02 KB (3088)
           - TotalBufferWaitTime: 0ns
           - TotalEncryptionTime: 0ns
           - TotalIntegrityCheckTime: 0ns
           - TotalReadBlockTime: 0ns
        CodeGen:(Total: 32m22s, non-child: 32m22s, % non-child: 100.00%)
           - CodegenTime: 0ns
           - CompileTime: 53.143ms
           - LoadTime: 58.680us
           - ModuleFileSize: 1.96 MB (2054956)
           - OptimizationTime: 32m22s
           - PrepareTime: 157.700ms
        HdfsTableSink:(Total: 179.796ms, non-child: 179.796ms, % non-child: 100.00%)
           - BytesWritten: 3.23 KB (3304)
           - CompressTimer: 0ns
           - EncodeTimer: 426ns
           - FilesCreated: 1
           - FinalizePartitionFileTimer: 9.651ms
           - HdfsWriteTimer: 691.84us
           - PartitionsCreated: 1
           - PeakMemoryUsage: 7.94 MB (8323072)
           - RowsInserted: 0
           - TmpFileCreateTimer: 4.561ms
        HASH_JOIN_NODE (id=16):(Total: 8.457ms, non-child: 7.673ms, % non-child: 90.73%)
          ExecOption: Join Build-Side Prepared Asynchronously
           - BuildPartitionTime: 27.903us
           - BuildRows: 131
           - BuildRowsPartitioned: 131
           - BuildTime: 28.181us
           - GetNewBlockTime: 5.261ms
           - HashBuckets: 324
           - LargestPartitionPercent: 4
           - MaxPartitionLevel: 0
           - NumRepartitions: 0
           - PartitionsCreated: 32
           - PeakMemoryUsage: 6.08 MB (6375968)
           - PinTime: 0ns
           - ProbeRows: 0
           - ProbeRowsPartitioned: 0
           - ProbeTime: 363ns
           - RowsReturned: 0
           - RowsReturnedRate: 0
           - SpilledPartitions: 0
           - UnpinTime: 1.822us
          EXCHANGE_NODE (id=25):(Total: 10.693us, non-child: 10.693us, % non-child: 100.00%)
            BytesReceived(32s000ms): 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB, 5.62 KB
             - BytesReceived: 5.62 KB (5750)
             - ConvertRowBatchTime: 3.869us
             - DeserializeRowBatchTimer: 19.974us
             - FirstBatchArrivalWaitTime: 0ns
             - PeakMemoryUsage: 0
             - RowsReturned: 131
             - RowsReturnedRate: 12.25 M/sec
             - SendersBlockedTimer: 0ns
             - SendersBlockedTotalTimer(*): 0ns
        HASH_JOIN_NODE (id=15):(Total: 8.70ms, non-child: 6.702ms, % non-child: 83.05%)
          ExecOption: Join Build-Side Prepared Asynchronously
           - BuildPartitionTime: 6.380us
           - BuildRows: 4
           - BuildRowsPartitioned: 4
           - BuildTime: 3.630us
           - GetNewBlockTime: 4.971ms
           - HashBuckets: 8
           - LargestPartitionPercent: 25
           - MaxPartitionLevel: 0
           - NumRepartitions: 0
           - PartitionsCreated: 32
           - PeakMemoryUsage: 4.07 MB (4263936)
           - PinTime: 0ns
           - ProbeRows: 0
           - ProbeRowsPartitioned: 0
           - ProbeTime: 230ns
           - RowsReturned: 0
           - RowsReturnedRate: 0
           - SpilledPartitions: 0
           - UnpinTime: 1.714us
          EXCHANGE_NODE (id=24):(Total: 10.159us, non-child: 10.159us, % non-child: 100.00%)
            BytesReceived(32s000ms): 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B, 123.00 B
             - BytesReceived: 123.00 B (123)
             - ConvertRowBatchTime: 1.42us
             - DeserializeRowBatchTimer: 8.361us
             - FirstBatchArrivalWaitTime: 0ns
             - PeakMemoryUsage: 0
             - RowsReturned: 4
             - RowsReturnedRate: 393.74 K/sec
             - SendersBlockedTimer: 0ns
             - SendersBlockedTotalTimer(*): 0ns
        HASH_JOIN_NODE (id=14):(Total: 7.879ms, non-child: 6.700ms, % non-child: 85.04%)
          ExecOption: Join Build-Side Prepared Asynchronously
           - BuildPartitionTime: 5.395us
           - BuildRows: 6
           - BuildRowsPartitioned: 6
           - BuildTime: 8.404us
           - GetNewBlockTime: 4.288ms
           - HashBuckets: 12
           - LargestPartitionPercent: 16
           - MaxPartitionLevel: 0
           - NumRepartitions: 0
           - PartitionsCreated: 32
           - PeakMemoryUsage: 4.06 MB (4255744)
           - PinTime: 0ns
           - ProbeRows: 0
           - ProbeRowsPartitioned: 0
           - ProbeTime: 323ns
           - RowsReturned: 0
           - RowsReturnedRate: 0
           - SpilledPartitions: 0
           - UnpinTime: 1.713us
          EXCHANGE_NODE (id=23):(Total: 9.723us, non-child: 9.723us, % non-child: 100.00%)
            BytesReceived(32s000ms): 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B, 340.00 B
             - BytesReceived: 340.00 B (340)
             - ConvertRowBatchTime: 1.38us
             - DeserializeRowBatchTimer: 5.253us
             - FirstBatchArrivalWaitTime: 0ns
             - PeakMemoryUsage: 0
             - RowsReturned: 6
             - RowsReturnedRate: 617.09 K/sec
             - SendersBlockedTimer: 0ns
             - SendersBlockedTotalTimer(*): 0ns
        HASH_JOIN_NODE (id=13):(Total: 7.599ms, non-child: 6.707ms, % non-child: 88.27%)
          ExecOption: Join Build-Side Prepared Asynchronously
           - BuildPartitionTime: 6.126us
           - BuildRows: 6
           - BuildRowsPartitioned: 6
           - BuildTime: 6.76us
           - GetNewBlockTime: 5.91ms
           - HashBuckets: 12
           - LargestPartitionPercent: 16
           - MaxPartitionLevel: 0
           - NumRepartitions: 0
           - PartitionsCreated: 32
           - PeakMemoryUsage: 4.05 MB (4247552)
           - PinTime: 0ns
           - ProbeRows: 0
           - ProbeRowsPartitioned: 0
           - ProbeTime: 373ns
           - RowsReturned: 0
           - RowsReturnedRate: 0
           - SpilledPartitions: 0
           - UnpinTime: 1.734us
          EXCHANGE_NODE (id=22):(Total: 8.757us, non-child: 8.757us, % non-child: 100.00%)
            BytesReceived(32s000ms): 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B, 352.00 B
             - BytesReceived: 352.00 B (352)
             - ConvertRowBatchTime: 993ns
             - DeserializeRowBatchTimer: 7.459us
             - FirstBatchArrivalWaitTime: 0ns
             - PeakMemoryUsage: 0
             - RowsReturned: 6
             - RowsReturnedRate: 685.17 K/sec
             - SendersBlockedTimer: 0ns
             - SendersBlockedTotalTimer(*): 0ns
        HASH_JOIN_NODE (id=12):(Total: 7.342ms, non-child: 6.974ms, % non-child: 94.99%)
          ExecOption: Join Build-Side Prepared Asynchronously
           - BuildPartitionTime: 8.329us
           - BuildRows: 5
           - BuildRowsPartitioned: 5
           - BuildTime: 6.205us
           - GetNewBlockTime: 4.741ms
           - HashBuckets: 10
           - LargestPartitionPercent: 20
           - MaxPartitionLevel: 0
           - NumRepartitions: 0
           - PartitionsCreated: 32
           - PeakMemoryUsage: 4.04 MB (4239360)
           - PinTime: 0ns
           - ProbeRows: 0
           - ProbeRowsPartitioned: 0
           - ProbeTime: 278ns
           - RowsReturned: 0
           - RowsReturnedRate: 0
           - SpilledPartitions: 0
           - UnpinTime: 1.794us
          EXCHANGE_NODE (id=21):(Total: 6.983us, non-child: 6.983us, % non-child: 100.00%)
            BytesReceived(32s000ms): 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B, 262.00 B
             - BytesReceived: 262.00 B (262)
             - ConvertRowBatchTime: 855ns
             - DeserializeRowBatchTimer: 4.621us
             - FirstBatchArrivalWaitTime: 0ns
             - PeakMemoryUsage: 0
             - RowsReturned: 5
             - RowsReturnedRate: 716.02 K/sec
             - SendersBlockedTimer: 0ns
             - SendersBlockedTotalTimer(*): 0ns
        HASH_JOIN_NODE (id=11):(Total: 6.823ms, non-child: 7.131ms, % non-child: 100.00%)
          ExecOption: Join Build-Side Prepared Asynchronously
           - BuildPartitionTime: 6.3us
           - BuildRows: 6
           - BuildRowsPartitioned: 6
           - BuildTime: 8.29us
           - GetNewBlockTime: 4.755ms
           - HashBuckets: 12
           - LargestPartitionPercent: 16
           - MaxPartitionLevel: 0
           - NumRepartitions: 0
           - PartitionsCreated: 32
           - PeakMemoryUsage: 4.04 MB (4231168)
           - PinTime: 0ns
           - ProbeRows: 0
           - ProbeRowsPartitioned: 0
           - ProbeTime: 428ns
           - RowsReturned: 0
           - RowsReturnedRate: 0
           - SpilledPartitions: 0
           - UnpinTime: 1.773us
          EXCHANGE_NODE (id=20):(Total: 6.65us, non-child: 6.65us, % non-child: 100.00%)
            BytesReceived(32s000ms): 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B, 288.00 B
             - BytesReceived: 288.00 B (288)
             - ConvertRowBatchTime: 756ns
             - DeserializeRowBatchTimer: 4.728us
             - FirstBatchArrivalWaitTime: 0ns
             - PeakMemoryUsage: 0
             - RowsReturned: 6
             - RowsReturnedRate: 989.28 K/sec
             - SendersBlockedTimer: 0ns
             - SendersBlockedTotalTimer(*): 0ns
        HASH_JOIN_NODE (id=10):(Total: 5.837ms, non-child: 6.161ms, % non-child: 100.00%)
          ExecOption: Join Build-Side Prepared Asynchronously
           - BuildPartitionTime: 4.773us
           - BuildRows: 4
           - BuildRowsPartitioned: 4
           - BuildTime: 7.239us
           - GetNewBlockTime: 3.841ms
           - HashBuckets: 8
           - LargestPartitionPercent: 25
           - MaxPartitionLevel: 0
           - NumRepartitions: 0
           - PartitionsCreated: 32
           - PeakMemoryUsage: 4.03 MB (4222976)
           - PinTime: 0ns
           - ProbeRows: 0
           - ProbeRowsPartitioned: 0
           - ProbeTime: 1.429us
           - RowsReturned: 0
           - RowsReturnedRate: 0
           - SpilledPartitions: 0
           - UnpinTime: 1.581us
          EXCHANGE_NODE (id=19):(Total: 6.905us, non-child: 6.905us, % non-child: 100.00%)
            BytesReceived(32s000ms): 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B, 220.00 B
             - BytesReceived: 220.00 B (220)
             - ConvertRowBatchTime: 1.216us
             - DeserializeRowBatchTimer: 4.378us
             - FirstBatchArrivalWaitTime: 0ns
        
   

 

Cloudera Employee
Posts: 269
Registered: ‎07-29-2015

Re: long codegen time while codegen disabled

I can't see exactly where it is, because the profile is truncated, but the "codegen disabled" message is in a HDFS SCAN section of the profile. Codegen may be enabled for all the other operators in the query. In more recent versions of Impala, every operator that supports codegen will say whether codegen was enabled or disabled, so it's easier to see.

 

If possible I'd really recommend upgrading to a more recent release of Impala. Impala 2.10 in CDH5.13 has all of the improvements we've made so far (there are a lot of them). We've put a lot of effort into improving codegen time and I suspect in combination the changes will solve your problem.

 

As a workaround, you can disable codegen query-wide with the disable_codegen query option - https://www.cloudera.com/documentation/enterprise/latest/topics/impala_disable_codegen.html.

 

 

New Contributor
Posts: 5
Registered: ‎11-07-2017

Re: long codegen time while codegen disabled

here is the complete profile:

https://github.com/zuomingcai/vue-typescript-component/blob/master/profile_1.txt

 

it seems no operator is codegen enabled,but still long codegen optimization time got.

 

Cloudera Employee
Posts: 269
Registered: ‎07-29-2015

Re: long codegen time while codegen disabled

In Impala 2.1 the absence of a message saying "Codegen enabled" doesn't mean that codegen is not enabled. It is definitely enabled for some operators in that profile you sent me.

New Contributor
Posts: 5
Registered: ‎11-07-2017

Re: long codegen time while codegen disabled

I had run the same sql with codegen disabled and enabled seperately,
the flowlling is the profiles:

 

codegen disabled:

https://github.com/zuomingcai/vue-typescript-component/blob/master/profile(codegen%20disabled).txt

 

codegen enabled:

https://github.com/zuomingcai/vue-typescript-component/blob/master/profile(codegen%20enabled).txt

 

from above profiles,it seems that the previous query that took a long time on codegen optimization is codegen disabled.

is there any workaroud other than upgrade.

 

Cloudera Employee
Posts: 269
Registered: ‎07-29-2015

Re: long codegen time while codegen disabled

It looks like the codegen time is much lower, so it did have an effect.

 

I do still see one codegen entry in the profile. There were some cases in that version of Impala where codegen could be enabled for parts of the query even when disable_codegen=true. E.g. if you are using an LLVM IR UDF or if you are calling a function with > 9 argument.

 

I don't know of a better workaround unfortunately.

 

New Contributor
Posts: 5
Registered: ‎11-07-2017

Re: long codegen time while codegen disabled

thanks for your information,a lot of other customer uses our software and it runs well.so we think it may be triggered by the hardware and software that impala runs on.for this customer it's deployed on a clound server.do you think it is possible to bypass if we moved to a physical computer。what is the most possible factor that trigger this bug,cpu?

Highlighted
New Contributor
Posts: 5
Registered: ‎11-07-2017

Re: long codegen time while codegen disabled

[ Edited ]

a lot of other customers use our software and runs well,so we think it may be triggered by the hardware and software that impala runs on.for this customer it is deployed on a number of cloud servers.do you think it is possible to bypass if move to physical computers.what is the most possible factor that trigger this bug,cpu? can we avoid the long time codegen optimization if we switch to a newer impala version and disable codegen without changing hardware and software that impala runs on.I need some information to decide which option to choose.

Announcements