Changes to /etc/cloudera-scm-agent/config.ini Not Always Respected

[#1]

CM 552

SCM Agent Version: 5.5.3

Centos5.11

We needed to relocate the agent outputs from /var/log and /var/lib because of space restrictions on the root mount.  Using config.ini, we were able to sucessfully move log_file to a new location.  

However, changes to the lib_dir property are wholly ignored and we are fairly sure we are not victims of a phat-phinger attack.  Could someone check on this?

Thanks!

Hugh

[#2 · Accepted solution]

Thanks for reporting this.

Upon investigating, there's a small bug with how we load/respect lib_dir in the agent. There's two ways to supply it, one via the CMF_AGENT_ARGS (--lib_dir=/path) inside /etc/default/cloudera-scm-agent and another via the line inside /etc/cloudera-scm-agent/config.ini.

The small bug is that for the UUID file we seem to look at the default path before we load up the config.

I'd have expected some form of error in your case vs. a blatant ignore, especially if /var/lib/cloudera-scm-agent no longer exists, so I'd first make sure you have lib_dir configured without the prefix # comment-marking keyword in your config.ini, and if that is so, apply the workaround to the bug (specify both in CMF_AGENT_ARGS and in config.ini).

The internal bug report is OPSAPS-32501

P.s. Make sure to copy over the uuid file before you erase /var/lib/cloudera-scm-agent. If you've lost it, no worries, you can create the uuid file under your lib_dir with the content matching the one shown on the CM -> Hosts -> (the specific host's page) -> "Host ID" field on top. Failure in keeping the same UUID will result in duplicated hosts (though that can be fixed trivially by fixing the uuid file and restarting agent yet again, then deleting the role-less duped host).

[#3]

Hi, Harsh!

At least I am not losing my mind 🙂  Ok. I will set this up in my lab and let you know what the outcome is.  Thank you for getting back to me on this!!

Regards!

-Hugh

[#4]

Quick follow-up:

I have confirmed that the combination of updating /etc/cloudera-scm-agent/config.ini AND adding --lib_dir= to /etc/defaults/cloudera-scm-agent successfully corrects the startup problem when relocating /var/lib/cloudera-scm-agent.  Here is what I did:

1. Stop agent:

   sudo service cloudera-scm-agent stop

2. Update /etc/cloudera-scm-agent/config.ini:

# Particularly, the agent's UUID is stored here.
# lib_dir=/var/lib/cloudera-scm-agent
lib_dir=/NEW_DIR_PATH

3. Update /etc/default/cloudera-scm-agent:

# Specify any command line arguments for the Cloudera SCM Agent here.
#

CMF_AGENT_ARGS="--lib_dir=/NEW_DIR_PATH "

4. Copy old content of /var/lib/coudera-scm-agent to new location /NEW_DIR_PATH:

sudo cp /var/lib/cloudera-scm-agent/* /NEW_DIR_PATH

5. Rename old content (in case we mess up):

sudo mv /var/lib/cloudera-scm-agent /var/lib/cloudera-scm-agent-del

6. Restart agent:

sudo /sbin/service cloudera-scm-agent start

7. If you are happy and you know it, delete the old lib:

sudo rm-rf /var/lib/cloudera-scm-agent-del

I am happy.  And I appreciate the really quick turn-around!!

-Hugh

[#5]

Hi all,

I can’t modify the default log folder for cloudera-scm-agent and supervisord (cmf/), I tried to modify the python file .../cmf/agent.py (for cmf agent manage by supervisord) and the /etc/cloudera-scm-agent/config.init whit my New_Log_Folder_Path but nothing change…

Any time that I restart the agents, I have the same issue, the cmf-agent (supervisord) and cloudera-scm-agent write their logs in the default log folder: /var/log/cloudera-scm-agent/… Below the process running when the agents are started:

ps -aux | grep cloudera

root       7257  0.7  0.0 1722792 46732 ?       Ssl  11:03   0:18 python2.7 /usr/lib64/cmf/agent/build/env/bin/cmf-agent --package_dir /usr/lib64/cmf/service --agent_dir /var/run/cloudera-scm-agent --lib_dir /var/lib/cloudera-scm-agent --logfile /var/log/cloudera-scm-agent/cloudera-scm-agent.log --daemon --comm_name cmf-agent --pidfile /var/run/cloudera-scm-agent/cloudera-scm-agent.pid --lib_dir=/var/lib/cloudera-scm-agent

root       7281  0.0  0.0 212548 12252 ?        S    11:03   0:00 python2.7 /usr/lib64/cmf/agent/build/env/bin/cmf-listener -l /var/log/cloudera-scm-agent/cmf_listener.log /run/cloudera-scm-agent/events

root      15086  0.0  0.0 112660   976 pts/0    S+   11:43   0:00 grep --color=auto cloudera

ps -aux | grep supervisord

root       7279  0.0  0.0 222276 13388 ?        Ss   11:03   0:01 /usr/lib64/cmf/agent/build/env/bin/python /usr/lib64/cmf/agent/build/env/bin/supervisord

root      15106  0.0  0.0 112660   976 pts/0    S+   11:43   0:00 grep --color=auto supervisord

Could you help me please to change this default log folder for supervisord and cloudera-scm-agent? If the services are started like root, could be a problem? I installed cloudera-manager-agent-5.12.2-1.cm5122.p0.12.el7.x86_64.rpm.

Many thanks in advance for the kind cooperation.

Kind regards,

Alexander

[#6]

To change the whole log directory you'll currently need to pass --logdir to
the agent as arguments, instead of via a config flag.

Edit your agent environment config file at /etc/default/cloudera-scm-agent
and ensure that the CMF_AGENT_ARGS env-var inside it carries the below:

--logdir=/your/custom/cloudera-scm/user/writable/directory/

Save, then restart the agent service.

P.s. Using symlinks will also work.

[#7]

Hi Harsh,

First of all, thanks a lot for you rapidly feedback, with your suggestions I have solved my problem with the logs files path. However I wish to ask you kindly one more thing, why I still see into the cloudera-scm-agent process parameters, two parameters for the log files?

root      52342      1  0 14:17 ?        00:00:01 python2.7 /usr/lib64/cmf/agent/build/env/bin/cmf-agent --package_dir /usr/lib64/cmf/service --agent_dir /var/run/cloudera-scm-agent --lib_dir /var/lib/cloudera-scm-agent --logfile /var/log/cloudera-scm-agent/cloudera-scm-agent.log (Old parameter) --daemon --comm_name cmf-agent --pidfile /var/run/cloudera-scm-agent/cloudera-scm-agent.pid --logdir=/var/log/cloudera/cloudera-scm-agent/cloudera-scm-agent.log (New parameter: Now, the agent write the logs in this new path as desired)

Instead for the cmf-listener process, all works fine:

root      50427  50421  0 14:08 ?        00:00:00 python2.7 /usr/lib64/cmf/agent/build/env/bin/cmf-listener -l /var/log/cloudera/cloudera-scm-agent/cmf_listener.log (New parameter) /run/cloudera-scm-agent/events

Is it possible to delete the "old parameter" from the cloudera-scm-agent process at start up?

I stopped/started all cloudera-scm-agent process after modify the file /etc/default/cloudera-scm-agent.

P.s: I don’t like so much use the symbolic links in these cases…😉

Many thanks!

Alex

[#8]

Hi all,

in order to illustrate to you a full overview about how change the all logs files related to Cloudera Manager Agents (v 5.12), I want to share with you guys my particular situation and how I solved it.

I wanted modify the default path logs for the following log files links with Cloudera Manager Agent:

- supervisord.out

- supervisord.log

(these are the supervisor's logs, this service start up with the Cloudera Manager Agent service the first time when the server start up or when we start up manually the cloudera-scm-agent service. If you stop the Cloudera Manager Agent service but the server remain up, this service remain up also).

- cmf_listener.log

(this is a cmf_listener service's logs, this service start up with the Cloudera Manager Agent service the first time when the server start up or when we start up manually the cloudera-scm-agent service. If you stop the Cloudera Manager Agent service but the server remain up, this service remain up also (managed by supervisord).

- cloudera-scm-agent.log

- cloudera-scm-agent.out

(These are the cloudera-scm-agent logs…)

Below the files that I've modified in order to set a new path for these logs with a dedicated file-system:

• /usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.2-py2.7.egg/lib/cmf/agent.py (set a new path for the logs: supervisord.out; cmf_listener.log and supervisord.log. Furthermore, check if the parameter for cloudera libraries agent is present in order to avoid unexpected errors with the start up of supervisord (default_lib_dir = '/var/lib/cloudera-scm-agent'))
• /usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.12.2-py2.7.egg/cmf/agent.py (set a new path for the logs: supervisord.out; cmf_listener.log and supervisord.log. Furthermore, check if the parameter for cloudera libraries agent is present in order to avoid unexpected errors with the start up of supervisord (default_lib_dir = '/var/lib/cloudera-scm-agent'))

• /etc/default/cloudera-scm-agent(set a new path for the cloudera agent logs as arguments: CMF_AGENT_ARGS="--logdir=/your/custom/cloudera-scm/user/writable/directory/" and set also CMF_AGENT_ARGS="--lib_dir=/var/lib/cloudera-scm-agent", in order to avoid unexpected errors with the start up of cloudera-scm-agent (As suggested by Harsh 😉).

• /etc/cloudera-scm-agent/config.ini (set a new path for the cloudera agent logs: log_file=/your/custom/cloudera-scm/user/writable/directory/ and set also lib_dir=/var/lib/cloudera-scm-agent, in order to avoid unexpected errors with the start up of cloudera-scm-agent)

• /etc/init.d/cloudera-scm-agent (set a new path for cloudera-scm-agent.out modifying the following parameter: AGENT_OUT=${CMF_VAR:-/var}/log/cloudera/$prog/$prog.out). In my case I left the logs into /var/log but I mounted a dedicated file-system on the folder /var/log/cloudera/ for all these logs.
  

If you want to stop all the processes related to Cloudera Manager Agent, you should perform the following steps:

• service cloudera-scm-agent stop (Stop Cloudera Manager Agent process)
• ps -eaf | grep cmf (Show the supervisord parent process)
  

root      77977      1  0 13:09 ?        00:00:00 /usr/lib64/cmf/agent/build/env/bin/python /usr/lib64/cmf/agent/build/env/bin/supervisord

root      77983  77977  0 13:09 ?        00:00:00 python2.7 /usr/lib64/cmf/agent/build/env/bin/cmf-listener -l /var/log/cloudera/cloudera-scm-agent/cmf_listener.log /run/cloudera-scm-agent/events

kill -15 77977 (Stop supervisord and cmf_listener processes). I used kill -15 because I didn't found another way to stop these processes…

I would like so much if anyone from Cloudera's Engineers can confirm/amend the procedure that I just described above...

Cheers!

Alex