Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

mem_limit issue

mem_limit issue

Hello, 

 

We have a impala tables with 14Millilon records per table. When I tried to join the tables , it failed with memory limit exceeded. Based on impala document, I changed the mem_limit to 64G and gradually increased to 3096G, but no luck. Please throw some light for the memory limit issue.

 

[ip-172-30-1-57.ec2.internal:21000] > select count(*) from tab3_nfs_new;

Query: select count(*) from tab3_nfs_new

+----------+

| count(*) |

+----------+

| 14216336 |

+----------+

Fetched 1 row(s) in 8.29s

[ip-172-30-1-57.ec2.internal:21000] > select count(*) from tab4_nfs_new;

Query: select count(*) from tab4_nfs_new

+----------+

| count(*) |

+----------+

| 14987634 |

+----------+

Fetched 1 row(s) in 11.46s

[ip-172-30-1-57.ec2.internal:21000] >

ip-172-30-1-57.ec2.internal:21000] > set mem_limit=64g;

MEM_LIMIT set to 64g

[ip-172-30-1-57.ec2.internal:21000] > select count(*) from tab3_nfs_new join tab4_nfs_new;

Query: select count(*) from tab3_nfs_new join tab4_nfs_new

WARNINGS:

Memory limit exceeded

 

 

 

Memory Limit Exceeded

Query(b44dde4886ccafd2:cfb6b9ffcdc4e3a6) Limit: Limit=64.00 GB Consumption=80.77 MB

  Fragment b44dde4886ccafd2:cfb6b9ffcdc4e3a7: Consumption=12.00 KB

    AGGREGATION_NODE (id=6): Consumption=4.00 KB

    EXCHANGE_NODE (id=5): Consumption=0

    DataStreamRecvr: Consumption=0

  Block Manager: Limit=156.00 MB Consumption=0

  Fragment b44dde4886ccafd2:cfb6b9ffcdc4e3a8: Consumption=64.62 MB

    AGGREGATION_NODE (id=3): Consumption=4.00 KB

    NESTED_LOOP_JOIN_NODE (id=2): Consumption=64.06 MB

    HDFS_SCAN_NODE (id=0): Consumption=0

    EXCHANGE_NODE (id=4): Consumption=0

    DataStreamRecvr: Consumption=544.00 KB

    DataStreamSender: Consumption=16.00 KB

  Fragment b44dde4886ccafd2:cfb6b9ffcdc4e3a9: Consumption=16.14 MB

    HDFS_SCAN_NODE (id=1): Consumption=16.01 MB

    DataStreamSender: Consumption=128.00 KB

WARNING: The following tables are missing relevant table and/or column statistics.

nfs_db.tab3_nfs_new,nfs_db.tab4_nfs_new

 

[ip-172-30-1-57.ec2.internal:21000] >

 

regards, 

osk

1 REPLY 1
Highlighted

Re: mem_limit issue

Cloudera Employee
==>

WARNING: The following tables are missing relevant table and/or column statistics.
nfs_db.tab3_nfs_new,nfs_db.tab4_nfs_new

==>

Please execute COMPUTE STATS for the above tables and see if that helps to get the query running.
Don't have an account?
Coming from Hortonworks? Activate your account here