Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cluster/Database setup: unable to connect to MySQL database

Cluster/Database setup: unable to connect to MySQL database

Explorer

I'm running through the Cloudera Manager Installation here and am setting up the external databases for Hive, the Activity Monitor and the Oozie Server (following directions herehere and here). I setup the databases in MySQL with 

 

create database amon;
create database rman;
create database metastore;
create database sentry;
create database nav;
create database navms;

CREATE USER 'amon'@'%' IDENTIFIED BY 'amon_password';
CREATE USER 'rman'@'%' IDENTIFIED BY 'rman_password';
CREATE USER 'hive'@'%' IDENTIFIED BY 'hive_password';
CREATE USER 'sentry'@'%' IDENTIFIED BY 'sentry_password';
CREATE USER 'nav'@'%' IDENTIFIED BY 'nav_password';
CREATE USER 'navms'@'%' IDENTIFIED BY 'navms_password';

grant all on amon.* TO 'amon'@'%';
grant all on rman.* TO 'rman'@'%';
grant all on metastore.* TO 'hive'@'%';
grant all on sentry.* TO 'sentry'@'%';
grant all on nav.* TO 'nav'@'%';
grant all on navms.* TO 'navms'@'%';

 I ran the scm_prepare_database script with

sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql cms root

 

In the Cloudera Manager installer I enter the following:

 

 

Everything is being run/setup on the same server (so MySQL, Cloudera Manager, Hive, Activity Monitor, Oozie Server, etc.). From the command line I'm able to access the above databases via

 

mysql -u hive -phive_password  metastore
mysql -u amon -pamon_password amon
mysql -u oozie -poozie  oozie

Here is the output from /var/log/cloudera-scm-server/cloudera-scm-server.log

 

2016-06-30 13:56:14,837 INFO 703190548@scm-web-1254:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command AmonTestDatabaseConnection BasicCmdArgs{args=[hadoop2.wwbp.org, mysql, hadoop2.wwbp.org, amon, amon, amon_password]}.  Host: DbHost{id=4, hostId=a8907bf3-baac-4e11-a9cd-68c90475ca6e, hostName=hadoop2.wwbp.org}
2016-06-30 13:56:14,841 INFO 703190548@scm-web-1254:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HiveTestDatabaseConnection BasicCmdArgs{args=[hadoop2.wwbp.org, mysql, hadoop2.wwbp.org, metastore, hive, hive_password]}.  Host: DbHost{id=4, hostId=a8907bf3-baac-4e11-a9cd-68c90475ca6e, hostName=hadoop2.wwbp.org}
2016-06-30 13:56:14,844 INFO 703190548@scm-web-1254:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command OozieTestDatabaseConnection BasicCmdArgs{args=[hadoop2.wwbp.org, mysql, hadoop2.wwbp.org, oozie, oozie, oozie]}.  Host: DbHost{id=4, hostId=a8907bf3-baac-4e11-a9cd-68c90475ca6e, hostName=hadoop2.wwbp.org}
2016-06-30 13:56:14,883 WARN 623697616@agentServer-250:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=60 name=null host=a8907bf3-baac-4e11-a9cd-68c90475ca6e/hadoop2.wwbp.org
2016-06-30 13:56:14,883 WARN 623697616@agentServer-250:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=61 name=null host=a8907bf3-baac-4e11-a9cd-68c90475ca6e/hadoop2.wwbp.org
2016-06-30 13:56:14,883 WARN 623697616@agentServer-250:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=62 name=null host=a8907bf3-baac-4e11-a9cd-68c90475ca6e/hadoop2.wwbp.org
2016-06-30 13:56:29,913 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'AmonTestDatabaseConnection'
2016-06-30 13:56:29,913 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: -1
2016-06-30 13:56:29,913 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Error found before invoking supervisord: Cant locate template for uri 'supervisee.conf'
2016-06-30 13:56:29,913 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 92(AmonTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Unexpected error. Unable to verify database connection.
2016-06-30 13:56:29,920 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'HiveTestDatabaseConnection'
2016-06-30 13:56:29,920 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: -1
2016-06-30 13:56:29,921 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Error found before invoking supervisord: Cant locate template for uri 'supervisee.conf'
2016-06-30 13:56:29,921 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 93(HiveTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Unexpected error. Unable to verify database connection.
2016-06-30 13:56:29,927 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'OozieTestDatabaseConnection'
2016-06-30 13:56:29,927 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: -1
2016-06-30 13:56:29,927 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Error found before invoking supervisord: Cant locate template for uri 'supervisee.conf'
2016-06-30 13:56:29,927 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 94(OozieTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Unexpected error. Unable to verify database connection.

 

Not sure what I am doing wrong. I saw another question here where they put each db on a different host. Is this really the only solution? 

16 REPLIES 16

Re: Cluster/Database setup: unable to connect to MySQL database

Super Guru

First place I'd recommend checking is the Cloudera Manager log to find out if there are any nice stack traces.

Check the /var/log/cloudera-scm-server/cloudera-scm-server.log

 

-Ben

Re: Cluster/Database setup: unable to connect to MySQL database

Explorer

Added log output to my question. Error is: 

Cant locate template for uri 'supervisee.conf'

Re: Cluster/Database setup: unable to connect to MySQL database

Super Guru

Out of context, it is hard to assess if that is related.  That is a problem reported by the agent,  not Cloudera Manager.

 

I would "tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log" while doing the db tests in the Wizard again.  Watch what happens in the log when you try that.

 

Also, if you post log information, we need some context to go on, so pasting in the surrounding lines would be helpful.

 

-Ben

Re: Cluster/Database setup: unable to connect to MySQL database

Explorer

Sorry. Last few lines of log before running test:

 

2016-06-30 14:23:09,513 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Wizard is active, not reaping scanners or configurators
2016-06-30 14:33:09,524 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands
2016-06-30 14:33:09,525 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2014-07-01T18:33:09.524Z to reap.
2016-06-30 14:33:09,526 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Wizard is active, not reaping scanners or configurators
2016-06-30 14:42:55,936 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files CDH-5.7.1-1.cdh5.7.1.p0.11-trusty.parcel under /opt/cloudera/parcel-repo 2016-06-30 14:43:09,536 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Reaped total of 0 deleted commands 2016-06-30 14:43:09,538 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Found no commands older than 2014-07-01T18:43:09.537Z to reap. 2016-06-30 14:43:09,540 INFO StaleEntityEviction:com.cloudera.server.cmf.StaleEntityEvictionThread: Wizard is active, not reaping scanners or configurators 2016-06-30 14:43:47,731 INFO ScmActive-0:com.cloudera.server.cmf.components.ScmActive: (119 skipped) ScmActive completed successfully.

 

Lines printed while running test:

 

2016-06-30 14:46:26,292 INFO 1766284876@scm-web-1288:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command AmonTestDatabaseConnection BasicCmdArgs{args=[hadoop2.wwbp.org, mysql, hadoop2.wwbp.org, amon, amon, amon_password]}.  Host: DbHost{id=4, hostId=a8907bf3-baac-4e11-a9cd-68c90475ca6e, hostName=hadoop2.wwbp.org}
2016-06-30 14:46:26,296 INFO 1766284876@scm-web-1288:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HiveTestDatabaseConnection BasicCmdArgs{args=[hadoop2.wwbp.org, mysql, hadoop2.wwbp.org, metastore, hive, hive_password]}.  Host: DbHost{id=4, hostId=a8907bf3-baac-4e11-a9cd-68c90475ca6e, hostName=hadoop2.wwbp.org}
2016-06-30 14:46:26,299 INFO 1766284876@scm-web-1288:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command OozieTestDatabaseConnection BasicCmdArgs{args=[hadoop2.wwbp.org, mysql, hadoop2.wwbp.org, oozie, oozie, oozie]}.  Host: DbHost{id=4, hostId=a8907bf3-baac-4e11-a9cd-68c90475ca6e, hostName=hadoop2.wwbp.org}
2016-06-30 14:46:26,345 WARN 623697616@agentServer-250:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=64 name=null host=a8907bf3-baac-4e11-a9cd-68c90475ca6e/hadoop2.wwbp.org
2016-06-30 14:46:26,345 WARN 623697616@agentServer-250:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=65 name=null host=a8907bf3-baac-4e11-a9cd-68c90475ca6e/hadoop2.wwbp.org
2016-06-30 14:46:26,345 WARN 623697616@agentServer-250:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=63 name=null host=a8907bf3-baac-4e11-a9cd-68c90475ca6e/hadoop2.wwbp.org
2016-06-30 14:46:41,386 WARN 623697616@agentServer-250:com.cloudera.server.cmf.AgentProtocolImpl: (11 skipped) Received client config heartbeat for unknown client config. Ignoring. name=hbase-conf path=/etc/hbase/conf.cloudera.hbase host=a8907bf3-baac-4e11-a9cd-68c90475ca6e/hadoop2.wwbp.org
2016-06-30 14:46:41,394 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'AmonTestDatabaseConnection'
2016-06-30 14:46:41,395 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: -1
2016-06-30 14:46:41,395 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Error found before invoking supervisord: Cant locate template for uri 'supervisee.conf'
2016-06-30 14:46:41,395 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 95(AmonTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Unexpected error. Unable to verify database connection.
2016-06-30 14:46:41,401 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'HiveTestDatabaseConnection'
2016-06-30 14:46:41,401 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: -1
2016-06-30 14:46:41,401 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Error found before invoking supervisord: Cant locate template for uri 'supervisee.conf'
2016-06-30 14:46:41,401 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 96(HiveTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Unexpected error. Unable to verify database connection.
2016-06-30 14:46:41,407 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'OozieTestDatabaseConnection'
2016-06-30 14:46:41,407 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: -1
2016-06-30 14:46:41,407 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Error found before invoking supervisord: Cant locate template for uri 'supervisee.conf'
2016-06-30 14:46:41,407 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 97(OozieTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Unexpected error. Unable to verify database connection.

 

It then repeats the first lines after a few minutes. 

Re: Cluster/Database setup: unable to connect to MySQL database

Super Guru

Thanks, that really helps.

What you are seeing is quite a strange issue and it makes me suspect that there is a problem with the agent on one or more hosts.  I would try going to the hosts where you are attempting to install the roles and try running:

 

service cloudera-scm-agent restart

 

Then, check /var/log/cloudera-scm-agent/cloudera-scm-agent.log for any errors regarding the supervisor (maybe grep for it)

 

Then, try the database tests again... if the problem still occurs, let us know.

Re: Cluster/Database setup: unable to connect to MySQL database

Super Guru

You might also check to see that there really is a "supervisee.conf"

 

try:

 

find /usr/ -name supervisee.conf

 

Anything returned?

Re: Cluster/Database setup: unable to connect to MySQL database

Explorer

I grepped through the log file and the only lines that mention "supervisor" is the line mentioning supervisee.conf

 

 

2016-06-30 14:46:41,407 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Error found before invoking supervisord: Cant locate template for uri 'supervisee.conf'

Nothing comes up when I search for supervisee.conf.

 

I'm using the same host all the roles in question. Also, I'm not running cloudera-scm-agent on any hosts. I saw this was an optional step in the installation and that they could be activated through the Cloudera Manager, so I skipped it.  

 

Werid thing is that I had Cloudera Manager v5.6 running last week (following Installation Path A - Automated Installation by Cloudera Manager) with a fully functional cluster. Since I was starting the production server I uninstalled this and started Path B with v5.7. I had a similar error with supervisee.conf when running the Path A -Automated Installation with v5.7 (when trying to start Zookeeper, if I remember correctly). 

 

Re: Cluster/Database setup: unable to connect to MySQL database

Super Guru

that supervisee.conf should be added by the agent config. If you are installing services to be managed by cloudera manager, you need to have an agent on any host that will be running services.

 

I would install the agent package on your host manually, configure the /etc/cloudera-scm-agent/config.ini server_host to point to your CM host (if it is not localhost), and then start with "service cloudera-scm-agent start"

 

Install of the agent is ONLY optional if your are running Cloudera Manager by itself on a host.  If you are installing on a single host, the CM host needs an agent since it is going to run Management Service and CDH roles.

 

-Ben

Re: Cluster/Database setup: unable to connect to MySQL database

Explorer

First, thank you for all of the help! It is very much appreciated.

 

I ran through the optional steps: installed JDK and installed and started 

Install Cloudera Manager Agent Packages. I also ran the 1-click Install package on every host but when I try to install the CDH packages I get some error, usually related to the Spark history-server

 

 

Setting up spark-history-server (1.6.0+cdh5.7.1+193-1.cdh5.7.1.p0.16~trusty-cdh5.7.1) ...
 * Failure to start Spark history-server (spark-history-server). Return value: 1
invoke-rc.d: initscript spark-history-server, action "start" failed.
dpkg: error processing package spark-history-server (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 spark-history-server
E: Sub-process /usr/bin/dpkg returned an error code (1)

Wasn't able to fix this so I tried proceeding with the installation anyway. I ran 

 

 

sudo service cloudera-scm-server start

 

as before and logged into the web installer. I come to the "Specify hosts for your CDH cluster installation" page. Here I can see all hosts in my cluster (2 name nodes, 8 data nodes). For the 8 data nodes, under Currently Managed, it says Yes and I am not able to select them (I was previously able to select everything). The two name nodes are not currently managed and I am able to select them and proceed. I proceed through the installation but the Database Setup page never appears. I can make it as far as Cluster Installation and both name nodes fail with the following errors

 

Primary Name Node

BEGIN sudo apt-get -o Dpkg::Options::=--force-confdef -o Dpkg::Options::=--force-confold -y install oracle-j2sdk1.6 
Reading package lists... 
Building dependency tree... 
Reading state information... 
oracle-j2sdk1.6 is already the newest version. 
The following packages were automatically installed and are no longer required: 
linux-headers-3.19.0-56 linux-headers-3.19.0-56-generic postgresql 
Use 'apt-get autoremove' to remove them. 
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 
1 not fully installed or removed. 
After this operation, 0 B of additional disk space will be used. 
Setting up spark-history-server (1.6.0+cdh5.7.1+193-1.cdh5.7.1.p0.16~trusty-cdh5.7.1) ... 
* Failure to start Spark history-server (spark-history-server). Return value: 1 
invoke-rc.d: initscript spark-history-server, action "start" failed. 
dpkg: error processing package spark-history-server (--configure): 
subprocess installed post-installation script returned error exit status 1 
Errors were encountered while processing: 
spark-history-server 
E: Sub-process /usr/bin/dpkg returned an error code (1) 
END (100) 
remote package oracle-j2sdk1.6 could not be installed, giving up 
waiting for rollback request 

Secondary Name Node:

BEGIN sudo tail -n 50 /var/log/cloudera-scm-agent//cloudera-scm-agent.log | sed 's/^/>>/' 
>>[01/Jul/2016 11:39:35 +0000] 19386 MainThread _cplogging INFO [01/Jul/2016:11:39:35] ENGINE No thread running for None. 
>>[01/Jul/2016 11:39:35 +0000] 19386 MainThread _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus STOPPED 
>>[01/Jul/2016 11:39:35 +0000] 19386 MainThread _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus EXITING 
>>[01/Jul/2016 11:39:35 +0000] 19386 MainThread _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus EXITED 
>>[01/Jul/2016 11:39:35 +0000] 19386 MainThread agent INFO Cleaning up daemon 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent INFO Stopping agent... 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent INFO No extant cgroups; unmounting any cgroup roots 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent ERROR Shutdown callback failed. 
>>Traceback (most recent call last): 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 2764, in stop 
>> f() 
>> File "/usr/lib/python2.7/asyncore.py", line 409, in close 
>> self.socket.close() 
>> File "/usr/lib/python2.7/asyncore.py", line 636, in close 
>> os.close(self.fd) 
>>OSError: [Errno 9] Bad file descriptor 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent INFO No processes are being managed; Supervisor will shutdown. 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent INFO Shutting down supervisord, pid 19423 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent ERROR Failed to kill process with pid 19423 
>>Traceback (most recent call last): 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 2819, in kill_process 
>> os.kill(pid, signal.SIGTERM) 
>>OSError: [Errno 3] No such process 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent ERROR Shutdown callback failed. 
>>Traceback (most recent call last): 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 2764, in stop 
>> f() 
>> File "/usr/lib/python2.7/asyncore.py", line 409, in close 
>> self.socket.close() 
>> File "/usr/lib/python2.7/asyncore.py", line 636, in close 
>> os.close(self.fd) 
>>OSError: [Errno 9] Bad file descriptor 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus STOPPING 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('hadoop.wwbp.org', 9000)) already shut down 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE No thread running for None. 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus STOPPED 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus STOPPING 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('hadoop.wwbp.org', 9000)) already shut down 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE No thread running for None. 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus STOPPED 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus EXITING 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus EXITED 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent ERROR Shutdown callback failed. 
>>Traceback (most recent call last): 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 2764, in stop 
>> f() 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/pyinotify-0.9.3-py2.7.egg/pyinotify.py", line 1424, in stop 
>> self._pollobj.unregister(self._fd) 
>>KeyError: 14 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent INFO Clea>>[01/Jul/2016 11:39:35 +0000] 19386 MainThread _cplogging INFO [01/Jul/2016:11:39:35] ENGINE No thread running for None. 
>>[01/Jul/2016 11:39:35 +0000] 19386 MainThread _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus STOPPED 
>>[01/Jul/2016 11:39:35 +0000] 19386 MainThread _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus EXITING 
>>[01/Jul/2016 11:39:35 +0000] 19386 MainThread _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus EXITED 
>>[01/Jul/2016 11:39:35 +0000] 19386 MainThread agent INFO Cleaning up daemon 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent INFO Stopping agent... 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent INFO No extant cgroups; unmounting any cgroup roots 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent ERROR Shutdown callback failed. 
>>Traceback (most recent call last): 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 2764, in stop 
>> f() 
>> File "/usr/lib/python2.7/asyncore.py", line 409, in close 
>> self.socket.close() 
>> File "/usr/lib/python2.7/asyncore.py", line 636, in close 
>> os.close(self.fd) 
>>OSError: [Errno 9] Bad file descriptor 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent INFO No processes are being managed; Supervisor will shutdown. 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent INFO Shutting down supervisord, pid 19423 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent ERROR Failed to kill process with pid 19423 
>>Traceback (most recent call last): 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 2819, in kill_process 
>> os.kill(pid, signal.SIGTERM) 
>>OSError: [Errno 3] No such process 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent ERROR Shutdown callback failed. 
>>Traceback (most recent call last): 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 2764, in stop 
>> f() 
>> File "/usr/lib/python2.7/asyncore.py", line 409, in close 
>> self.socket.close() 
>> File "/usr/lib/python2.7/asyncore.py", line 636, in close 
>> os.close(self.fd) 
>>OSError: [Errno 9] Bad file descriptor 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus STOPPING 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('hadoop.wwbp.org', 9000)) already shut down 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE No thread running for None. 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus STOPPED 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus STOPPING 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('hadoop.wwbp.org', 9000)) already shut down 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE No thread running for None. 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus STOPPED 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus EXITING 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 _cplogging INFO [01/Jul/2016:11:39:35] ENGINE Bus EXITED 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent ERROR Shutdown callback failed. 
>>Traceback (most recent call last): 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.7.1-py2.7.egg/cmf/agent.py", line 2764, in stop 
>> f() 
>> File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/pyinotify-0.9.3-py2.7.egg/pyinotify.py", line 1424, in stop 
>> self._pollobj.unregister(self._fd) 
>>KeyError: 14 
>>[01/Jul/2016 11:39:35 +0000] 19386 Dummy-14 agent INFO Cleaning up daemon 
ning up daemon 
END (0) 
end of agent logs. 
scm agent restarted