Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ambari agent memory leak or taking too much memory

Ambari agent memory leak or taking too much memory

Contributor

Ambari agent in HDP 2.2.0 seems to be taking about 22Gigs after running for a while at a customer site.

Is there a way to reduce or limit its memory usage

2 REPLIES 2
Highlighted

Re: Ambari agent memory leak or taking too much memory

Highlighted

Re: Ambari agent memory leak or taking too much memory

New Contributor

This is a known Ambari bug:

https://issues.apache.org/jira/browse/AMBARI-17539

This is resolved in Ambari Agent 2.4.0

As a work around you can modify the main.py

1. Stop Ambari agent

2. Backup file

mv /usr/lib/python2.6/site-packages/ambari_agent/main.py /tmp/main.py.backup

3. Edit main.py under this path : /usr/lib/python2.6/site-packages/ambari_agent/

Add the following at the beginning of the file:

def fix_subprocess_racecondition():

""" Subprocess in Python has race condition with enabling/disabling gc. Which may lead to turning off python garbage collector. This leads to a memory leak. This function monkey patches subprocess to fix the issue. !!! PLEASE NOTE THIS SHOULD BE CALLED BEFORE ANY OTHER INITIALIZATION was done to avoid already created links to subprocess or subprocess.gc or gc """

# monkey patching subprocess

import subprocess subprocess.gc.isenabled = lambda: True

# re-importing gc to have correct isenabled for non-subprocess contexts

import sys del sys.modules['gc']

import gc fix_subprocess_racecondition()

4. Start Ambari agent

Please do this on one host and monitor the memory usage. If memory usage looks OK, then replace main.py on all hosts.

Don't have an account?
Coming from Hortonworks? Activate your account here