Member since
07-16-2018
8
Posts
0
Kudos Received
0
Solutions
10-23-2019
11:23 AM
CDH 5.15.0 CentOS 6.10 final Hey, My internal team members have been using an enterprise CM environment installed on a cluster with more-than-adequate hardware (used to be customer-facing and handle multiple large queries at once), but they have been complaining about their scripts and queries failing inconsistently due to out-of-memory errors. This is occurring with both users with memory limits as well as users with free access to the entire cluster's resources. An example output is shown below. Is this a known issue with the current CDH version? The reason I'm raising this concern is because this cluster used to run smoothly under much heavier query load and query concurrency, and now it seems to be a roll of the dice every time a non-tiny query is run. Memory limit exceeded: Error occurred on backend <hostname> by fragment b84dc213ea94e53d:a98ab78000000ad
Memory left in process limit: 125.63 GB
Memory left in query limit: -130.89 KB
Query(b84dc213ea94e53d:a98ab7800000000): memory limit exceeded. Limit=1.00 GB Reservation=441.88 MB ReservationLimit=819.20 MB OtherMemory=582.25 MB Total=1.00 GB Peak=1.00 GB
Unclaimed reservations: Reservation=112.00 MB OtherMemory=0 Total=112.00 MB Peak=237.75 MB
Fragment b84dc213ea94e53d:a98ab7800000141: Reservation=0 OtherMemory=57.64 KB Total=57.64 KB Peak=1.57 MB
AGGREGATION_NODE (id=49): Total=42.12 KB Peak=42.12 KB
Exprs: Total=42.12 KB Peak=42.12 KB
EXCHANGE_NODE (id=48): Reservation=0 OtherMemory=0 Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=50): Total=424.00 B Peak=424.00 B
CodeGen: Total=7.10 KB Peak=1.52 MB
Fragment b84dc213ea94e53d:a98ab7800000122: Reservation=0 OtherMemory=10.32 MB Total=10.32 MB Peak=14.39 MB
AGGREGATION_NODE (id=32): Total=42.12 KB Peak=42.12 KB
Exprs: Total=42.12 KB Peak=42.12 KB
HASH_JOIN_NODE (id=31): Total=142.25 KB Peak=142.25 KB
Exprs: Total=31.12 KB Peak=31.12 KB
Hash Join Builder (join_node_id=31): Total=31.12 KB Peak=31.12 KB
Hash Join Builder (join_node_id=31) Exprs: Total=31.12 KB Peak=31.12 KB
EXCHANGE_NODE (id=46): Reservation=0 OtherMemory=10.09 MB Total=10.09 MB Peak=10.09 MB
DataStreamRecvr: Total=10.09 MB Peak=10.09 MB
EXCHANGE_NODE (id=47): Reservation=0 OtherMemory=0 Total=0 Peak=0
DataStreamRecvr: Total=0 Peak=0
DataStreamSender (dst_id=48): Total=12.11 KB Peak=12.11 KB
CodeGen: Total=31.34 KB Peak=4.59 MB
Fragment b84dc213ea94e53d:a98ab780000006b: Reservation=34.00 MB OtherMemory=17.89 MB Total=51.89 MB Peak=51.89 MB
HASH_JOIN_NODE (id=30): Reservation=34.00 MB OtherMemory=2.60 MB Total=36.60 MB Peak=36.60 MB
Exprs: Total=43.12 KB Peak=43.12 KB
Hash Join Builder (join_node_id=30): Total=39.12 KB Peak=63.12 KB
Hash Join Builder (join_node_id=30) Exprs: Total=39.12 KB Peak=39.12 KB
EXCHANGE_NODE (id=37): Reservation=0 OtherMemory=10.04 MB Total=10.04 MB Peak=10.04 MB
DataStreamRecvr: Total=10.04 MB Peak=10.04 MB
EXCHANGE_NODE (id=38): Reservation=0 OtherMemory=0 Total=0 Peak=1.20 MB
DataStreamRecvr: Total=0 Peak=1.20 MB
DataStreamSender (dst_id=46): Total=2.85 MB Peak=3.61 MB
CodeGen: Total=11.39 KB Peak=1.51 MB
Fragment b84dc213ea94e53d:a98ab7800000034: Reservation=1.94 MB OtherMemory=409.38 MB Total=411.32 MB Peak=411.32 MB
HASH_JOIN_NODE (id=29): Reservation=1.94 MB OtherMemory=6.95 MB Total=8.89 MB Peak=12.12 MB
Exprs: Total=21.12 KB Peak=21.12 KB
Hash Join Builder (join_node_id=29): Total=21.12 KB Peak=45.12 KB
Hash Join Builder (join_node_id=29) Exprs: Total=21.12 KB Peak=21.12 KB
HDFS_SCAN_NODE (id=0): Total=393.15 MB Peak=393.15 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=35): Reservation=0 OtherMemory=0 Total=0 Peak=4.02 KB
DataStreamRecvr: Total=0 Peak=4.02 KB
DataStreamSender (dst_id=37): Total=3.03 MB Peak=6.07 MB
DataStreamSender (dst_id=37) Exprs: Total=4.00 KB Peak=4.00 KB
CodeGen: Total=12.10 KB Peak=1.76 MB
Fragment b84dc213ea94e53d:a98ab780000001f: Reservation=0 OtherMemory=0 Total=0 Peak=3.51 MB
HASH_JOIN_NODE (id=6): Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
Hash Join Builder (join_node_id=6): Total=0 Peak=37.12 KB
HDFS_SCAN_NODE (id=5): Total=0 Peak=326.00 KB
EXCHANGE_NODE (id=34): Reservation=0 OtherMemory=0 Total=0 Peak=4.02 KB
DataStreamRecvr: Total=0 Peak=4.02 KB
DataStreamSender (dst_id=35): Total=0 Peak=177.28 KB
CodeGen: Total=0 Peak=1.53 MB
Fragment b84dc213ea94e53d:a98ab7800000056: Reservation=0 OtherMemory=0 Total=0 Peak=22.23 MB
SELECT_NODE (id=11): Total=0 Peak=1.02 MB
ANALYTIC_EVAL_NODE (id=10): Reservation=0 OtherMemory=0 Total=0 Peak=5.54 MB
ANALYTIC_EVAL_NODE (id=9): Reservation=0 OtherMemory=0 Total=0 Peak=4.53 MB
SORT_NODE (id=8): Reservation=0 OtherMemory=0 Total=0 Peak=12.12 MB
EXCHANGE_NODE (id=36): Reservation=0 OtherMemory=0 Total=0 Peak=2.04 MB
DataStreamRecvr: Total=0 Peak=2.04 MB
DataStreamSender (dst_id=38): Total=0 Peak=1.02 MB
CodeGen: Total=0 Peak=1.13 MB
Fragment b84dc213ea94e53d:a98ab780000004a: Reservation=0 OtherMemory=0 Total=0 Peak=144.13 KB
HDFS_SCAN_NODE (id=7): Total=0 Peak=109.00 KB
DataStreamSender (dst_id=36): Total=0 Peak=30.91 KB
CodeGen: Total=0 Peak=52.50 KB
Fragment b84dc213ea94e53d:a98ab7800000103: Reservation=258.00 MB OtherMemory=1.68 MB Total=259.68 MB Peak=259.68 MB
SELECT_NODE (id=28): Total=4.00 KB Peak=4.00 KB
Exprs: Total=4.00 KB Peak=4.00 KB
ANALYTIC_EVAL_NODE (id=27): Total=4.00 KB Peak=4.00 KB
Exprs: Total=4.00 KB Peak=4.00 KB
ANALYTIC_EVAL_NODE (id=26): Total=4.00 KB Peak=4.00 KB
Exprs: Total=4.00 KB Peak=4.00 KB
SORT_NODE (id=25): Reservation=258.00 MB OtherMemory=293.67 KB Total=258.29 MB Peak=258.29 MB
EXCHANGE_NODE (id=45): Reservation=0 OtherMemory=1.33 MB Total=1.33 MB Peak=10.01 MB
DataStreamRecvr: Total=1.35 MB Peak=10.01 MB
DataStreamSender (dst_id=47): Total=49.41 KB Peak=49.41 KB
CodeGen: Total=3.51 KB Peak=1.03 MB
Fragment b84dc213ea94e53d:a98ab78000000e4: Reservation=34.00 MB OtherMemory=2.40 MB Total=36.40 MB Peak=44.01 MB
HASH_JOIN_NODE (id=24): Reservation=34.00 MB OtherMemory=355.95 KB Total=34.35 MB Peak=34.35 MB
Exprs: Total=43.12 KB Peak=43.12 KB
Hash Join Builder (join_node_id=24): Total=39.12 KB Peak=55.12 KB
Hash Join Builder (join_node_id=24) Exprs: Total=39.12 KB Peak=39.12 KB
EXCHANGE_NODE (id=43): Reservation=0 OtherMemory=1.12 MB Total=1.12 MB Peak=8.75 MB
DataStreamRecvr: Total=1.12 MB Peak=8.75 MB
EXCHANGE_NODE (id=44): Reservation=0 OtherMemory=0 Total=0 Peak=821.20 KB
DataStreamRecvr: Total=0 Peak=821.20 KB
DataStreamSender (dst_id=45): Total=669.34 KB Peak=789.34 KB
DataStreamSender (dst_id=45) Exprs: Total=8.00 KB Peak=8.00 KB
CodeGen: Total=11.46 KB Peak=1.53 MB
Fragment b84dc213ea94e53d:a98ab78000000ad: Reservation=1.94 MB OtherMemory=140.59 MB Total=142.53 MB Peak=178.54 MB
HASH_JOIN_NODE (id=23): Reservation=1.94 MB OtherMemory=1.12 MB Total=3.05 MB Peak=4.28 MB
Exprs: Total=21.12 KB Peak=21.12 KB
Hash Join Builder (join_node_id=23): Total=21.12 KB Peak=45.12 KB
Hash Join Builder (join_node_id=23) Exprs: Total=21.12 KB Peak=21.12 KB
HDFS_SCAN_NODE (id=12): Total=137.83 MB Peak=174.28 MB
Exprs: Total=4.00 KB Peak=4.00 KB
EXCHANGE_NODE (id=41): Reservation=0 OtherMemory=0 Total=0 Peak=4.02 KB
DataStreamRecvr: Total=0 Peak=4.02 KB
DataStreamSender (dst_id=43): Total=643.78 KB Peak=971.78 KB
DataStreamSender (dst_id=43) Exprs: Total=4.00 KB Peak=4.00 KB
CodeGen: Total=12.06 KB Peak=1.73 MB
Fragment b84dc213ea94e53d:a98ab7800000098: Reservation=0 OtherMemory=0 Total=0 Peak=3.40 MB
HASH_JOIN_NODE (id=18): Reservation=0 OtherMemory=0 Total=0 Peak=2.02 MB
Hash Join Builder (join_node_id=18): Total=0 Peak=37.12 KB
HDFS_SCAN_NODE (id=17): Total=0 Peak=210.00 KB
EXCHANGE_NODE (id=40): Reservation=0 OtherMemory=0 Total=0 Peak=4.02 KB
DataStreamRecvr: Total=0 Peak=4.02 KB
DataStreamSender (dst_id=41): Total=0 Peak=177.28 KB
CodeGen: Total=0 Peak=1.53 MB
Fragment b84dc213ea94e53d:a98ab78000000cf: Reservation=0 OtherMemory=0 Total=0 Peak=18.08 MB
SELECT_NODE (id=22): Total=0 Peak=528.00 KB
ANALYTIC_EVAL_NODE (id=21): Reservation=0 OtherMemory=0 Total=0 Peak=4.53 MB
SORT_NODE (id=20): Reservation=0 OtherMemory=0 Total=0 Peak=12.10 MB
EXCHANGE_NODE (id=42): Reservation=0 OtherMemory=0 Total=0 Peak=1.37 MB
DataStreamRecvr: Total=0 Peak=1.37 MB
DataStreamSender (dst_id=44): Total=0 Peak=1.52 MB
CodeGen: Total=0 Peak=876.00 KB
Fragment b84dc213ea94e53d:a98ab78000000c3: Reservation=0 OtherMemory=0 Total=0 Peak=126.65 KB
HDFS_SCAN_NODE (id=19): Total=0 Peak=81.02 KB
DataStreamSender (dst_id=42): Total=0 Peak=41.41 KB
CodeGen: Total=0 Peak=52.50 KB
... View more
Labels:
- Labels:
-
Apache Impala
10-17-2018
04:50 PM
This didn't work. I tested a query with 10,250 characters after making the change and restarting the service, but the query was still truncated at the exact same spot. Are there any alternatives or other ways to fix this?
... View more
09-07-2018
01:32 PM
Thanks for the detailed response! I'm nearing the end of my internship, but my team is going to look into this. We'll be experimenting it outside of production at first to be safe, but overall it looks like a relatively simple fix. I'll update your response as the solution if it works.
... View more
09-05-2018
11:53 AM
I'd really appreciate a response on this. I understand if there's a hard limit on the size of the JSON files I'm extracting, but I'm wondering if it's something on my end. We would really like to access these queries for measuring performance.
... View more
08-23-2018
10:39 AM
I faced that same issue as well Tomas. This is more at the individual query object level though. We're trying to solve some performance issues and want to look at some of the most expensive queries, but we can't actually extract these really long queries to run performance testing with since they are cut off.
... View more
08-22-2018
10:21 AM
cm_api v19 python 2.7.15 I've noticed that some of my outputted query statements from utilizing the get_impala_queries() function in one of my Python scripts are cut off after reaching 10,000 characters in length. I don't want to post the actual query statements themselves since it is for my company, but I was curious if there are limitations set on the JSON files for these apiImpalaQuery objects. If so, is there any way to work around this? Steps to re-create situation: 1. call get_impala_queries() in Python script 2. access queries to obtain apiImpalaQuery objects 3. within each query, access the statement property 4. any query that is over 10,000 characters in length is cut-off, but ends with an added "..." to signify it isn't the end of the query
... View more
Labels:
- Labels:
-
Cloudera Manager
07-16-2018
11:08 AM
using CM_API v19 Hi, I'm trying to utilize the CM API with Python 2 to extract a large amount of Impala queries for analysis, but whenever I utilize the get_impala_queries() function to gather queries, I can not grab any queries further than about 11 days into the past. When I actually check the host webpage, there are definitely more queries that I want to extract that occurred earlier than 11 days ago. For example, when I run the script around now (7-16-18 16:35 UTC), I can not obtain any queries from dates earlier than 7-05-18 06:20 UTC.
... View more
Labels:
- Labels:
-
Apache Impala
-
Cloudera Manager