Support Questions
Find answers, ask questions, and share your expertise

[CDH 5.2.1] Impala Memory limit exceeded even with additional run arguments and computed stats

[CDH 5.2.1] Impala Memory limit exceeded even with additional run arguments and computed stats

Explorer

Hey there,

I've encounter this problem in CDH 5.2.1 and just can't get through it...

I've got 2 node cluster: VM on pc with 22GB RAM and 8GB RAM on notebook.

I've set impala deamon on pc and trying to run this query:

 

insert overwrite table pq_q21_tmp1
select
l_orderkey, cast(count(distinct l_suppkey) as int), max(l_suppkey) as max_suppkey
from
pq_lineitem
group by l_orderkey;

 

pq_lineitem is stored as parquet file with 600037906 rows ~= 30GB of data (78GB of data in normal textfile storage type - with normal text file problem also occured)

 

And whatever value i'd put into max gb of ram available for impala it always crashes with Maximum limit exceeded...

 

I've followed tips from this topic: Backend 6:Memory Limit Exceeded" in impala 2 (chd 5.2) but same query is working fine in im , computed stats for tables, added those keys/values (through cloudera manager impala configuration) but it didn't change much.. maybe a little more percentage of query run but still failed with memory limit exception.

 

Below I attach query profile, plan... 

 

Please help cause i'm basically stuck on this one and can't move on. Query was run through Hue webUI and through impala-shell - same result.

 

Query profile:

Query (id=f43914357b78c71:b62946de4ba0a7a7):
  Summary:
    Session ID: af4602c0772ba9ac:c6fd8a20a683528f
    Session Type: BEESWAX
    Start Time: 2014-12-27 16:41:28.766666000
    End Time: 2014-12-27 16:43:55.378458000
    Query Type: DML
    Query State: EXCEPTION
    Query Status: Memory limit exceeded
    Impala Version: impalad version 2.0.1-cdh5 RELEASE (build cc09df07d93c79351052eef2f4e5baf52373dc33)
    User: root
    Connected User: root
    Delegated User: 
    Network Address: ::ffff:192.168.1.31:43608
    Default Db: test
    Sql Statement: insert overwrite table pq_q21_tmp1 select l_orderkey, cast(count(distinct l_suppkey) as int), max(l_suppkey) as max_suppkey from pq_lineitem group by l_orderkey
    Coordinator: h1.t.pl:22000
    Plan: 
----------------
Estimated Per-Host Requirements: Memory=14.75GB VCores=2

F01:PLAN FRAGMENT [HASH(l_orderkey)]
  WRITE TO HDFS [test.pq_q21_tmp1, OVERWRITE=true]
  |  partitions=1
  |  hosts=2 per-host-mem=849.58MB
  |
  02:AGGREGATE [FINALIZE]
  |  output: count(l_suppkey), max:merge(l_suppkey)
  |  group by: l_orderkey
  |  hosts=2 per-host-mem=934.53MB
  |  tuple-ids=2 row-size=16B cardinality=111355608
  |
  04:AGGREGATE
  |  output: max:merge(l_suppkey)
  |  group by: l_orderkey, l_suppkey
  |  hosts=2 per-host-mem=7.38GB
  |  tuple-ids=1 row-size=12B cardinality=600037906
  |
  03:EXCHANGE [HASH(l_orderkey)]
     hosts=2 per-host-mem=0B
     tuple-ids=1 row-size=12B cardinality=600037906

F00:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=03, HASH(l_orderkey)]
  01:AGGREGATE
  |  output: max(l_suppkey)
  |  group by: l_orderkey, l_suppkey
  |  hosts=2 per-host-mem=7.38GB
  |  tuple-ids=1 row-size=12B cardinality=600037906
  |
  00:SCAN HDFS [test.pq_lineitem, RANDOM]
     partitions=1/1 size=19.77GB
     table stats: 600037906 rows total
     column stats: all
     hosts=2 per-host-mem=176.00MB
     tuple-ids=0 row-size=8B cardinality=600037906
----------------
    Estimated Per-Host Mem: 15841000242
    Estimated Per-Host VCores: 2
    Admission result: Admitted immediately
    Request Pool: root.root
    ExecSummary: 
Operator       #Hosts   Avg Time   Max Time    #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail           
-------------------------------------------------------------------------------------------------------------
02:AGGREGATE        1    1.226ms    1.226ms        0     111.36M    8.01 MB      934.53 MB  FINALIZE         
04:AGGREGATE        1  121.516ms  121.516ms        0     600.04M    8.01 MB        7.38 GB                   
03:EXCHANGE         1        0ns        0ns        0     600.04M          0              0  HASH(l_orderkey) 
01:AGGREGATE        1        0ns        0ns        0     600.04M    7.48 GB        7.38 GB                   
00:SCAN HDFS        1   14s797ms   14s797ms  164.91M     600.04M  209.80 MB      176.00 MB  test.pq_lineitem 
    Query Timeline: 2m26s
       - Start execution: 56.538us (56.538us)
       - Planning finished: 18.846ms (18.789ms)
       - Submit for admission: 23.862ms (5.16ms)
       - Completed admission: 23.942ms (80.236us)
       - Ready to start remote fragments: 24.118ms (175.389us)
       - Remote fragments started: 279.496ms (255.378ms)
       - Request finished: 2m25s (2m25s)
       - Unregister query: 2m26s (1s001ms)
  ImpalaServer:
     - ClientFetchWaitTimer: 0ns
     - RowMaterializationTimer: 0ns
  Execution Profile f43914357b78c71:b62946de4ba0a7a7:(Total: 2m25s, non-child: 0ns, % non-child: 0.00%)
    Fragment start latencies:  count: 2, last: 0.13061, min: 0.124446, max: 0.13061, mean: 0.127528, stddev: 0.003082
    Per Node Peak Memory Usage: h1.t.pl:22000(7.67 GB) 
     - FinalizationTimer: 6.135ms
    Averaged Fragment F01:(Total: 123.907ms, non-child: 0ns, % non-child: 0.00%)
      split sizes:  min: 0, max: 0, avg: 0, stddev: 0
      completion times: min:2m25s  max:2m25s  mean: 2m25s  stddev:0ns
      execution rates: min:0.00 /sec  max:0.00 /sec  mean:0.00 /sec  stddev:0.00 /sec
      num instances: 1
       - AverageThreadTokens: 1.00 
       - PeakMemoryUsage: 16.02 MB (16801792)
       - PerHostPeakMemUsage: 7.67 GB (8234215720)
       - PrepareTime: 122.743ms
       - RowsProduced: 0
       - TotalCpuTime: 0ns
       - TotalNetworkReceiveTime: 0ns
       - TotalNetworkSendTime: 0ns
       - TotalStorageWaitTime: 0ns
      BlockMgr:
         - BlockWritesOutstanding: 0
         - BlocksCreated: 0
         - BlocksRecycled: 0
         - BufferedPins: 0
         - BytesWritten: 0
         - MaxBlockSize: 8.00 MB (8388608)
         - MemoryLimit: 6.25 GB (6710886400)
         - PeakMemoryUsage: 0
         - TotalBufferWaitTime: 0ns
         - TotalEncryptionTime: 0ns
         - TotalIntegrityCheckTime: 0ns
         - TotalReadBlockTime: 0ns
      CodeGen:(Total: 246.807ms, non-child: 246.807ms, % non-child: 100.00%)
         - CodegenTime: 2.682ms
         - CompileTime: 126.886ms
         - LoadTime: 119.100ms
         - ModuleFileSize: 1.94 MB (2033316)
      HdfsTableSink:(Total: 97.349us, non-child: 97.349us, % non-child: 100.00%)
         - BytesWritten: 0
         - CompressTimer: 0ns
         - EncodeTimer: 0ns
         - HdfsWriteTimer: 0ns
         - PeakMemoryUsage: 0
         - RowsInserted: 0
      AGGREGATION_NODE (id=2):(Total: 122.743ms, non-child: 1.226ms, % non-child: 1.00%)
         - BuildBuckets: 0
         - BuildTime: 0ns
         - GetResultsTime: 0ns
         - LoadFactor: 0.00 
         - PeakMemoryUsage: 8.01 MB (8396800)
         - RowsReturned: 0
         - RowsReturnedRate: 0
      AGGREGATION_NODE (id=4):(Total: 121.516ms, non-child: 121.516ms, % non-child: 100.00%)
         - BuildBuckets: 0
         - BuildTime: 0ns
         - GetResultsTime: 0ns
         - LoadFactor: 0.00 
         - PeakMemoryUsage: 8.01 MB (8396800)
         - RowsReturned: 0
         - RowsReturnedRate: 0
      EXCHANGE_NODE (id=3):
         - BytesReceived: 0
         - ConvertRowBatchTime: 0ns
         - DeserializeRowBatchTimer: 0ns
         - FirstBatchArrivalWaitTime: 0ns
         - PeakMemoryUsage: 0
         - RowsReturned: 0
         - RowsReturnedRate: 0
         - SendersBlockedTimer: 0ns
         - SendersBlockedTotalTimer(*): 0ns
    Averaged Fragment F00:(Total: 129.986ms, non-child: 0ns, % non-child: 0.00%)
      split sizes:  min: 19.77 GB, max: 19.77 GB, avg: 19.77 GB, stddev: 0
      completion times: min:2m25s  max:2m25s  mean: 2m25s  stddev:0ns
      execution rates: min:139.27 MB/sec  max:139.27 MB/sec  mean:139.27 MB/sec  stddev:0.00 /sec
      num instances: 1
       - AverageThreadTokens: 5.85 
       - PeakMemoryUsage: 7.65 GB (8217413928)
       - PerHostPeakMemUsage: 7.67 GB (8234215720)
       - PrepareTime: 129.621ms
       - RowsProduced: 0
       - TotalCpuTime: 2m3s
       - TotalNetworkReceiveTime: 0ns
       - TotalNetworkSendTime: 0ns
       - TotalStorageWaitTime: 1m17s
      CodeGen:(Total: 239.434ms, non-child: 239.434ms, % non-child: 100.00%)
         - CodegenTime: 2.342ms
         - CompileTime: 112.437ms
         - LoadTime: 125.891ms
         - ModuleFileSize: 1.94 MB (2033316)
      DataStreamSender (dst_id=3):(Total: 21.386us, non-child: 21.386us, % non-child: 100.00%)
         - BytesSent: 0
         - NetworkThroughput(*): 0.00 /sec
         - OverallThroughput: 0.00 /sec
         - PeakMemoryUsage: 8.00 KB (8192)
         - SerializeBatchTime: 0ns
         - ThriftTransmitTime(*): 0ns
         - UncompressedRowBatchSize: 0
      AGGREGATION_NODE (id=1):(Total: 129.620ms, non-child: 0ns, % non-child: 0.00%)
         - BuildBuckets: 134.22M (134217728)
         - BuildTime: 2m7s
         - GetResultsTime: 0ns
         - LoadFactor: 0.71 
         - PeakMemoryUsage: 7.48 GB (8036380664)
         - RowsReturned: 0
         - RowsReturnedRate: 0
      HDFS_SCAN_NODE (id=0):(Total: 14s797ms, non-child: 14s797ms, % non-child: 100.00%)
         - AverageHdfsReadThreadConcurrency: 0.47 
         - AverageScannerThreadConcurrency: 4.87 
         - BytesRead: 934.26 MB (979646329)
         - BytesReadDataNodeCache: 0
         - BytesReadLocal: 0
         - BytesReadShortCircuit: 0
         - DecompressionTime: 1s585ms
         - MaxCompressedTextFileLength: 0
         - NumColumns: 0
         - NumDisksAccessed: 0
         - NumScannerThreadsStarted: 5
         - PeakMemoryUsage: 209.80 MB (219993744)
         - PerReadThreadRawHdfsThroughput: 14.08 MB/sec
         - RowsRead: 164.94M (164937416)
         - RowsReturned: 164.91M (164912840)
         - RowsReturnedRate: 11.14 M/sec
         - ScanRangesComplete: 20
         - ScannerThreadsInvoluntaryContextSwitches: 68
         - ScannerThreadsTotalWallClockTime: 2m3s
           - MaterializeTupleTime(*): 9m22s
           - ScannerThreadsSysTime: 190.970ms
           - ScannerThreadsUserTime: 2s610ms
         - ScannerThreadsVoluntaryContextSwitches: 42.21K (42211)
         - TotalRawHdfsReadTime(*): 1m6s
         - TotalReadThroughput: 6.53 MB/sec
    Fragment F01:
      Instance f43914357b78c71:b62946de4ba0a7a8 (host=h1.t.pl:22000):(Total: 123.907ms, non-child: 0ns, % non-child: 0.00%)
        MemoryUsage(4s000ms): 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB, 16.02 MB
        ThreadUsage(4s000ms): 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
         - AverageThreadTokens: 1.00 
         - PeakMemoryUsage: 16.02 MB (16801792)
         - PerHostPeakMemUsage: 7.67 GB (8234215720)
         - PrepareTime: 122.743ms
         - RowsProduced: 0
         - TotalCpuTime: 0ns
         - TotalNetworkReceiveTime: 0ns
         - TotalNetworkSendTime: 0ns
         - TotalStorageWaitTime: 0ns
        BlockMgr:
           - BlockWritesOutstanding: 0
           - BlocksCreated: 0
           - BlocksRecycled: 0
           - BufferedPins: 0
           - BytesWritten: 0
           - MaxBlockSize: 8.00 MB (8388608)
           - MemoryLimit: 6.25 GB (6710886400)
           - PeakMemoryUsage: 0
           - TotalBufferWaitTime: 0ns
           - TotalEncryptionTime: 0ns
           - TotalIntegrityCheckTime: 0ns
           - TotalReadBlockTime: 0ns
        CodeGen:(Total: 246.807ms, non-child: 246.807ms, % non-child: 100.00%)
           - CodegenTime: 2.682ms
           - CompileTime: 126.886ms
           - LoadTime: 119.100ms
           - ModuleFileSize: 1.94 MB (2033316)
        HdfsTableSink:(Total: 97.349us, non-child: 97.349us, % non-child: 100.00%)
           - BytesWritten: 0
           - CompressTimer: 0ns
           - EncodeTimer: 0ns
           - HdfsWriteTimer: 0ns
           - PeakMemoryUsage: 0
           - RowsInserted: 0
        AGGREGATION_NODE (id=2):(Total: 122.743ms, non-child: 1.226ms, % non-child: 1.00%)
          ExecOption: Codegen Enabled
           - BuildBuckets: 0
           - BuildTime: 0ns
           - GetResultsTime: 0ns
           - LoadFactor: 0.00 
           - PeakMemoryUsage: 8.01 MB (8396800)
           - RowsReturned: 0
           - RowsReturnedRate: 0
        AGGREGATION_NODE (id=4):(Total: 121.516ms, non-child: 121.516ms, % non-child: 100.00%)
          ExecOption: Codegen Enabled
           - BuildBuckets: 0
           - BuildTime: 0ns
           - GetResultsTime: 0ns
           - LoadFactor: 0.00 
           - PeakMemoryUsage: 8.01 MB (8396800)
           - RowsReturned: 0
           - RowsReturnedRate: 0
        EXCHANGE_NODE (id=3):
          BytesReceived(4s000ms): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
           - BytesReceived: 0
           - ConvertRowBatchTime: 0ns
           - DeserializeRowBatchTimer: 0ns
           - FirstBatchArrivalWaitTime: 0ns
           - PeakMemoryUsage: 0
           - RowsReturned: 0
           - RowsReturnedRate: 0
           - SendersBlockedTimer: 0ns
           - SendersBlockedTotalTimer(*): 0ns
    Fragment F00:
      Instance f43914357b78c71:b62946de4ba0a7a9 (host=h1.t.pl:22000):(Total: 129.986ms, non-child: 0ns, % non-child: 0.00%)
        Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:81/19.77 GB 
        MemoryUsage(4s000ms): 82.58 MB, 422.07 MB, 847.14 MB, 1.04 GB, 1.45 GB, 1.87 GB, 1.85 GB, 2.35 GB, 2.36 GB, 3.20 GB, 3.67 GB, 3.66 GB, 3.64 GB, 3.60 GB, 3.73 GB, 4.63 GB, 4.64 GB, 4.66 GB, 4.68 GB, 4.67 GB, 5.05 GB, 5.24 GB, 5.24 GB, 5.26 GB, 5.38 GB, 5.60 GB, 5.60 GB, 6.08 GB, 6.18 GB, 6.15 GB, 6.50 GB, 7.18 GB, 7.35 GB, 7.55 GB, 7.63 GB
        ThreadUsage(4s000ms): 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 5, 5, 5, 5
         - AverageThreadTokens: 5.85 
         - PeakMemoryUsage: 7.65 GB (8217413928)
         - PerHostPeakMemUsage: 7.67 GB (8234215720)
         - PrepareTime: 129.621ms
         - RowsProduced: 0
         - TotalCpuTime: 2m3s
         - TotalNetworkReceiveTime: 0ns
         - TotalNetworkSendTime: 0ns
         - TotalStorageWaitTime: 1m17s
        CodeGen:(Total: 239.434ms, non-child: 239.434ms, % non-child: 100.00%)
           - CodegenTime: 2.342ms
           - CompileTime: 112.437ms
           - LoadTime: 125.891ms
           - ModuleFileSize: 1.94 MB (2033316)
        DataStreamSender (dst_id=3):(Total: 21.386us, non-child: 21.386us, % non-child: 100.00%)
           - BytesSent: 0
           - NetworkThroughput(*): 0.00 /sec
           - OverallThroughput: 0.00 /sec
           - PeakMemoryUsage: 8.00 KB (8192)
           - SerializeBatchTime: 0ns
           - ThriftTransmitTime(*): 0ns
           - UncompressedRowBatchSize: 0
        AGGREGATION_NODE (id=1):(Total: 129.620ms, non-child: 0ns, % non-child: 0.00%)
          ExecOption: Codegen Enabled
           - BuildBuckets: 134.22M (134217728)
           - BuildTime: 2m7s
           - GetResultsTime: 0ns
           - LoadFactor: 0.71 
           - PeakMemoryUsage: 7.48 GB (8036380664)
           - RowsReturned: 0
           - RowsReturnedRate: 0
        HDFS_SCAN_NODE (id=0):(Total: 14s797ms, non-child: 14s797ms, % non-child: 100.00%)
          Hdfs split stats (<volume id>:<# splits>/<split lengths>): 0:81/19.77 GB 
          BytesRead(4s000ms): 14.32 MB, 55.39 MB, 97.72 MB, 138.81 MB, 180.06 MB, 199.61 MB, 214.24 MB, 254.39 MB, 295.34 MB, 336.43 MB, 370.89 MB, 375.39 MB, 375.39 MB, 379.83 MB, 415.98 MB, 455.89 MB, 492.41 MB, 529.74 MB, 567.61 MB, 575.00 MB, 575.00 MB, 575.00 MB, 575.00 MB, 575.00 MB, 576.04 MB, 607.60 MB, 648.34 MB, 714.33 MB, 774.59 MB, 774.59 MB, 774.59 MB, 789.56 MB, 854.42 MB, 929.27 MB, 934.26 MB
           - AverageHdfsReadThreadConcurrency: 0.47 
           - AverageScannerThreadConcurrency: 4.87 
           - BytesRead: 934.26 MB (979646329)
           - BytesReadDataNodeCache: 0
           - BytesReadLocal: 0
           - BytesReadShortCircuit: 0
           - DecompressionTime: 1s585ms
           - MaxCompressedTextFileLength: 0
           - NumColumns: 0
           - NumDisksAccessed: 0
           - NumScannerThreadsStarted: 5
           - PeakMemoryUsage: 209.80 MB (219993744)
           - PerReadThreadRawHdfsThroughput: 14.08 MB/sec
           - RowsRead: 164.94M (164937416)
           - RowsReturned: 164.91M (164912840)
           - RowsReturnedRate: 11.14 M/sec
           - ScanRangesComplete: 20
           - ScannerThreadsInvoluntaryContextSwitches: 68
           - ScannerThreadsTotalWallClockTime: 2m3s
             - MaterializeTupleTime(*): 9m22s
             - ScannerThreadsSysTime: 190.970ms
             - ScannerThreadsUserTime: 2s610ms
           - ScannerThreadsVoluntaryContextSwitches: 42.21K (42211)
           - TotalRawHdfsReadTime(*): 1m6s
           - TotalReadThroughput: 6.53 MB/sec

 

Query summary:

Operator       #Hosts   Avg Time   Max Time    #Rows  Est. #Rows   Peak Mem  Est. Peak Mem  Detail           
-------------------------------------------------------------------------------------------------------------
02:AGGREGATE        1    1.226ms    1.226ms        0     111.36M    8.01 MB      934.53 MB  FINALIZE         
04:AGGREGATE        1  121.516ms  121.516ms        0     600.04M    8.01 MB        7.38 GB                   
03:EXCHANGE         1        0ns        0ns        0     600.04M          0              0  HASH(l_orderkey) 
01:AGGREGATE        1        0ns        0ns        0     600.04M    7.48 GB        7.38 GB                   
00:SCAN HDFS        1   14s797ms   14s797ms  164.91M     600.04M  209.80 MB      176.00 MB  test.pq_lineitem 

 Plan:

----------------
Estimated Per-Host Requirements: Memory=14.75GB VCores=2

F01:PLAN FRAGMENT [HASH(l_orderkey)]
  WRITE TO HDFS [test.pq_q21_tmp1, OVERWRITE=true]
  |  partitions=1
  |  hosts=2 per-host-mem=849.58MB
  |
  02:AGGREGATE [FINALIZE]
  |  output: count(l_suppkey), max:merge(l_suppkey)
  |  group by: l_orderkey
  |  hosts=2 per-host-mem=934.53MB
  |  tuple-ids=2 row-size=16B cardinality=111355608
  |
  04:AGGREGATE
  |  output: max:merge(l_suppkey)
  |  group by: l_orderkey, l_suppkey
  |  hosts=2 per-host-mem=7.38GB
  |  tuple-ids=1 row-size=12B cardinality=600037906
  |
  03:EXCHANGE [HASH(l_orderkey)]
     hosts=2 per-host-mem=0B
     tuple-ids=1 row-size=12B cardinality=600037906

F00:PLAN FRAGMENT [RANDOM]
  DATASTREAM SINK [FRAGMENT=F01, EXCHANGE=03, HASH(l_orderkey)]
  01:AGGREGATE
  |  output: max(l_suppkey)
  |  group by: l_orderkey, l_suppkey
  |  hosts=2 per-host-mem=7.38GB
  |  tuple-ids=1 row-size=12B cardinality=600037906
  |
  00:SCAN HDFS [test.pq_lineitem, RANDOM]
     partitions=1/1 size=19.77GB
     table stats: 600037906 rows total
     column stats: all
     hosts=2 per-host-mem=176.00MB
     tuple-ids=0 row-size=8B cardinality=600037906
----------------

 

Deamon logs:

Log file created at: 2014/12/27 16:59:50
Running on machine: h1.t.pl
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I1227 16:59:50.748847 36027 authentication.cc:990] Internal communication is not authenticated
I1227 16:59:50.748942 36027 authentication.cc:1011] External communication is not authenticated
I1227 16:59:50.749084 36027 init.cc:135] impalad version 2.0.1-cdh5 RELEASE (build cc09df07d93c79351052eef2f4e5baf52373dc33)
Built on Wed, 19 Nov 2014 11:20:16 PST
I1227 16:59:50.749090 36027 init.cc:136] Using hostname: h1.t.pl
I1227 16:59:50.749586 36027 logging.cc:100] Flags (see also /varz are on debug webserver):
--catalog_service_port=26000
--load_catalog_in_background=true
--num_metadata_loading_threads=16
--sentry_config=
--disable_optimization_passes=false
--dump_ir=false
--opt_module=
--print_llvm_ir_instruction_count=false
--unopt_module=
--abort_on_config_error=true
--be_port=22000
--be_principal=
--compact_catalog_topic=false
--disable_mem_pools=false
--enable_process_lifetime_heap_profiling=false
--heap_profile_dir=
--hostname=h1.t.pl
--keytab_file=
--krb5_conf=
--krb5_debug_file=
--mem_limit=8388608000
--principal=
--log_filename=impalad
--data_source_batch_size=1024
--exchg_node_buffer_size_bytes=10485760
--enable_partitioned_aggregation=false
--enable_partitioned_hash_join=false
--enable_probe_side_filtering=true
--skip_lzo_version_check=false
--max_row_batches=0
--enable_phj_probe_side_filtering=true
--enable_ldap_auth=false
--kerberos_reinit_interval=60
--ldap_allow_anonymous_binds=false
--ldap_baseDN=
--ldap_bind_pattern=
--ldap_ca_certificate=
--ldap_domain=
--ldap_manual_config=false
--ldap_passwords_in_clear_ok=false
--ldap_tls=false
--ldap_uri=
--sasl_path=/usr/lib/sasl2:/usr/lib64/sasl2:/usr/local/lib/sasl2:/usr/lib/x86_64-linux-gnu/sasl2
--rpc_cnxn_attempts=10
--rpc_cnxn_retry_interval_ms=2000
--disk_spill_encryption=false
--insert_inherit_permissions=false
--max_free_io_buffers=128
--min_buffer_size=1024
--num_disks=0
--num_threads_per_disk=0
--read_size=8388608
--catalog_service_host=h1.t.pl
--cgroup_hierarchy_path=
--enable_rm=false
--enable_webserver=true
--llama_addresses=
--llama_callback_port=28000
--llama_host=
--llama_max_request_attempts=5
--llama_port=15000
--llama_registration_timeout_secs=30
--llama_registration_wait_secs=3
--num_hdfs_worker_threads=16
--resource_broker_cnxn_attempts=1
--resource_broker_cnxn_retry_interval_ms=3000
--resource_broker_recv_timeout=0
--resource_broker_send_timeout=0
--staging_cgroup=impala_staging
--state_store_host=h1.t.pl
--state_store_subscriber_port=23000
--use_statestore=true
--local_library_dir=/var/lib/impala/udfs
--serialize_batch=false
--status_report_interval=5
--num_threads_per_core=3
--scratch_dirs=/impala/impalad
--queue_wait_timeout_ms=60000
--default_pool_max_queued=50
--default_pool_max_requests=20
--default_pool_mem_limit=
--disable_pool_max_requests=false
--disable_pool_mem_limits=false
--fair_scheduler_allocation_path=/var/run/cloudera-scm-agent/process/368-impala-IMPALAD/impala-conf/fair-scheduler.xml
--llama_site_path=/var/run/cloudera-scm-agent/process/368-impala-IMPALAD/impala-conf/llama-site.xml
--authorization_policy_file=
--authorization_policy_provider_class=org.apache.sentry.provider.common.HadoopGroupResourceAuthorizationProvider
--authorized_proxy_user_config=
--load_catalog_at_startup=false
--server_name=
--abort_on_failed_audit_event=false
--audit_event_log_dir=/var/log/impalad/audit
--be_service_threads=64
--beeswax_port=21000
--cancellation_thread_pool_size=5
--default_query_options=
--fe_service_threads=64
--hs2_port=21050
--idle_query_timeout=0
--idle_session_timeout=0
--local_nodemanager_url=
--log_mem_usage_interval=0
--log_query_to_file=true
--max_audit_event_log_file_size=5000
--max_profile_log_file_size=5000
--max_result_cache_size=100000
--profile_log_dir=
--query_log_size=25
--ssl_client_ca_certificate=
--ssl_private_key=
--ssl_server_certificate=
--max_vcore_oversubscription_ratio=2.5
--rm_always_use_defaults=false
--rm_default_cpu_vcores=2
--rm_default_memory=4G
--disable_admission_control=false
--require_username=false
--statestore_subscriber_cnxn_attempts=10
--statestore_subscriber_cnxn_retry_interval_ms=3000
--statestore_subscriber_timeout_seconds=30
--state_store_port=24000
--statestore_heartbeat_frequency_ms=5INFO1227 16:59:53.875000 main org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.AllocationFileLoaderService] Loading allocation file /var/run/cloudera-scm-agent/process/368-impala-IMPALAD/impala-conf/fair-scheduler.xml
INFO1227 16:59:53.909000 FileWatchThread(/var/run/cloudera-scm-agent/process/368-impala-IMPALAD/impala-conf/llama-site.xml)-0 com.cloudera.impala.util.RequestPoolService] Loading Llama configuration: /var/run/cloudera-scm-agent/process/368-impala-IMPALAD/impala-conf/llama-site.xml
00
--statestore_max_missed_heartbeats=10
--statestore_num_heartbeat_threads=10
--statestore_suspect_heartbeats=5
--force_lowercase_usernames=false
--num_cores=0
--web_log_bytes=1048576
--non_impala_java_vlog=0
--periodic_counter_update_period_ms=500
--enable_webserver_doc_root=true
--webserver_authentication_domain=
--webserver_certificate_file=
--webserver_doc_root=/opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/lib/impala
--webserver_interface=
--webserver_password_file=
--webserver_port=25000
--flagfile=/var/run/cloudera-scm-agent/process/368-impala-IMPALAD/impala-conf/impalad_flags
--fromenv=
--tryfromenv=
--undefok=
--tab_completion_columns=80
--tab_completion_word=
--help=false
--helpfull=false
--helpmatch=
--helpon=
--helppackage=false
--helpshort=false
--helpxml=false
--version=false
--alsologtoemail=
--alsologtostderr=false
--drop_log_memory=true
--log_backtrace_at=
--log_dir=/var/log/impalad
--log_link=
--log_prefix=true
--logbuflevel=0
--logbufsecs=30
--logbufvlevel=1
--logemaillevel=999
--logmailer=/bin/mail
--logtostderr=false
--max_log_size=200
--minloglevel=0
--stderrthreshold=2
--stop_logging_if_full_disk=false
--symbolize_stacktrace=true
--v=1
--vmodule=
I1227 16:59:50.749635 36027 init.cc:141] Cpu Info:
  Model: Intel(R) Core(TM) i5-4670 CPU @ 3.40GHz
  Cores: 4
  L1 Cache: 32.00 KB
  L2 Cache: 256.00 KB
  L3 Cache: 6.00 MB
  Hardware Supports:
    ssse3
    sse4_1
    sse4_2
    popcnt
I1227 16:59:50.749641 36027 init.cc:142] Disk Info: 
  Num disks 2: 
    sda (rotational=true)
    dm- (rotational=true)

I1227 16:59:50.749646 36027 init.cc:143] Physical Memory: 21.49 GB
I1227 16:59:50.749650 36027 init.cc:144] OS version: Linux version 2.6.32-504.el6.x86_64 (mockbuild@c6b9.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Wed Oct 15 04:27:16 UTC 2014
I1227 16:59:50.749651 36027 init.cc:145] Process ID: 36027
I1227 16:59:53.509343 36027 hbase-table-scanner.cc:155] Detected HBase version >= 0.95.2
I1227 16:59:53.918553 36027 GlogAppender.java:123] Logging initialized. Impala: VLOG, All other: INFO
I1227 16:59:53.921885 36027 JniFrontend.java:128] Authorization is 'DISABLED'.
I1227 16:59:53.922027 36027 JniFrontend.java:130] Java Version Info: Java(TM) SE Runtime Environment (1.7.0_67-b01)
W1227 16:59:55.045609 36027 HiveConf.java:1491] DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
I1227 16:59:55.514428 36027 impala-server.cc:1205] Default query options:TQueryOptions {
  01: abort_on_error (bool) = false,
  02: max_errors (i32) = 0,
  03: disable_codegen (bool) = false,
  04: batch_size (i32) = 0,
  05: num_nodes (i32) = 0,
  06: max_scan_range_length (i64) = 0,
  07: num_scanner_threads (i32) = 0,
  08: max_io_buffers (i32) = 0,
  09: allow_unsupported_formats (bool) = false,
  10: default_order_by_limit (i64) = -1,
  11: debug_action (string) = "",
  12: mem_limit (i64) = 0,
  13: abort_on_default_limit_exceeded (bool) = false,
  15: hbase_caching (i32) = 0,
  16: hbase_cache_blocks (bool) = false,
  17: parquet_file_size (i64) = 0,
  18: explain_level (i32) = 1,
  19: sync_ddl (bool) = false,
  23: disable_cached_reads (bool) = false,
  24: disable_outermost_topn (bool) = false,
  25: rm_initial_mem (i64) = 0,
  26: query_timeout_s (i32) = 0,
  28: appx_count_distinct (bool) = false,
  29: disable_unsafe_spills (bool) = false,
}
I1227 16:59:57.295456 36027 tmp-file-mgr.cc:77] Created the following scratch dirs:/impala/impalad/impala-scratch
I1227 16:59:57.295583 36027 simple-logger.cc:72] Logging to: /var/log/impalad/profiles//impala_profile_log_1.0-1419695997295
I1227 16:59:57.307965 36027 simple-logger.cc:72] Logging to: /var/log/impalad/audit/impala_audit_event_log_1.0-1419695997295
I1227 16:59:57.436043 36027 impala-server.cc:1919] Impala Beeswax Service listening on 21000
I1227 16:59:57.437436 36027 impala-server.cc:1941] Impala HiveServer2 Service listening on 21050
I1227 16:59:57.438791 36027 impala-server.cc:1953] ImpalaInternalService listening on 22000
I1227 16:59:57.439273 36027 thrift-server.cc:387] ThriftServer 'backend' started on port: 22000
I1227 16:59:57.439291 36027 exec-env.cc:277] Starting global services
I1227 16:59:57.464712 36027 exec-env.cc:338] Using global memory limit: 7.81 GB
I1227 16:59:57.470160 36027 webserver.cc:154] Starting webserver on 0.0.0.0:25000
I1227 16:59:57.470199 36027 webserver.cc:168] Document root: /opt/cloudera/parcels/CDH-5.2.1-1.cdh5.2.1.p0.12/lib/impala
I1227 16:59:57.470710 36027 webserver.cc:234] Webserver started
I1227 16:59:57.470743 36027 simple-scheduler.cc:180] Starting simple scheduler
I1227 16:59:57.470974 36027 simple-scheduler.cc:231] Simple-scheduler using 192.168.1.31 as IP address
I1227 16:59:57.470999 36027 statestore-subscriber.cc:178] Starting statestore subscriber
I1227 16:59:57.480949 36027 thrift-server.cc:387] ThriftServer 'StatestoreSubscriber' started on port: 23000
I1227 16:59:57.480978 36027 statestore-subscriber.cc:189] Registering with statestore
I1227 16:59:57.481036 36027 client-cache.cc:107] CreateClient(): creating new client for h1.t.pl:24000
I1227 16:59:57.491168 36027 statestore-subscriber.cc:163] Subscriber registration ID: 914c7033a1f737c0:9f956385526c2086
I1227 16:59:57.491200 36027 statestore-subscriber.cc:193] statestore registration successful
I1227 16:59:57.626232 36027 thrift-server.cc:387] ThriftServer 'beeswax-frontend' started on port: 21000
I1227 16:59:57.731696 36027 thrift-server.cc:387] ThriftServer 'hiveserver2-frontend' started on port: 21050
I1227 16:59:57.731714 36027 impalad-main.cc:89] Impala has started.
W1227 16:59:58.059953 36249 HiveConf.java:1491] DEPRECATED: Configuration property hive.metastore.local no longer has any effect. Make sure to provide a valid value for hive.metastore.uris if you are connecting to a remote metastore.
I1227 16:59:58.619810 36249 simple-scheduler.cc:340] Registering local backend with statestore

 

2 REPLIES 2

Re: [CDH 5.2.1] Impala Memory limit exceeded even with additional run arguments and computed stats

Cloudera Employee

Having nodes of widely different sizes and capabilities is usually not a good idea, because Impala will divide the work roughly equally. So even if your query did succeed, the 22 GB node would finish its half of the work very quickly, then would sit and wait while the 8 GB node did its half of the work always at the edge of running out of memory. Things would probably work just fine with 2 nodes @ 22 GB each, or even a single 22 GB node, but having an 8 GB one means one node will always be on the edge of running out of memory. For example, when Impala encounters a low memory condition and decides to start using a scratch workspace on disk, it expects roughly 300 MB of memory to be still free to get ready for writing to disk.

 

How many different l_orderkey values are there?  GROUP BY on a column that is unique or almost unique is a memory-intensive operation.  Writing to a Parquet table is a memory-intensive operation.  The first order of business is probably more memory for that 8 GB node to bring it up to a reasonable level.

 

John

Re: [CDH 5.2.1] Impala Memory limit exceeded even with additional run arguments and computed stats

Explorer

OH, I didn't know that... well it's kind'a design flaw as for me to have each node with exactly same parameters to run impala smoothly.

Gonna try it out with equal machines.

Thanks.

 

Regards,

Kuba