Our Community is getting an upgrade! To get everything ready for the relaunch, we’ll be placing the site in read-only mode starting September 21st.
We really appreciate your understanding while we get things set up behind the scenes. Catch up on all the exciting details about the move here.
Need help or have questions? Drop us a line at [email protected]
Created 04-19-2016 12:08 PM
Hi,everyone.
I have a problem While summarising the cluster in Ambari server , the Ambari metrics server is constantly throwing this warning and aborting in the middle.
Below is the log generated by the service while aborting.
stderr:
Command aborted. Aborted by user
stdout:
2016-04-19 12:28:16,359 - Group['hadoop'] {}
2016-04-19 12:28:16,360 - Group['users'] {}
2016-04-19 12:28:16,361 - Group['knox'] {}
2016-04-19 12:28:16,361 - Group['spark'] {}
2016-04-19 12:28:16,361 - User['oozie'] {'gid': 'hadoop', 'groups': ['users']}
2016-04-19 12:28:16,362 - User['hive'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,363 - User['ambari-qa'] {'gid': 'hadoop', 'groups': ['users']}
2016-04-19 12:28:16,363 - User['flume'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,364 - User['hdfs'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,365 - User['knox'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,366 - User['storm'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,367 - User['spark'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,367 - User['mapred'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,368 - User['hbase'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,369 - User['tez'] {'gid': 'hadoop', 'groups': ['users']}
2016-04-19 12:28:16,369 - User['zookeeper'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,370 - User['kafka'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,371 - User['falcon'] {'gid': 'hadoop', 'groups': ['users']}
2016-04-19 12:28:16,372 - User['sqoop'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,372 - User['yarn'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,373 - User['hcat'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,374 - User['ams'] {'gid': 'hadoop', 'groups': ['hadoop']}
2016-04-19 12:28:16,375 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-04-19 12:28:16,376 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] {'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
2016-04-19 12:28:16,384 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa'] due to not_if
2016-04-19 12:28:16,384 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'recursive': True, 'mode': 0775, 'cd_access': 'a'}
2016-04-19 12:28:16,391 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2016-04-19 12:28:16,393 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] {'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
2016-04-19 12:28:16,401 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due to not_if
2016-04-19 12:28:16,402 - Group['hdfs'] {'ignore_failures': False}
2016-04-19 12:28:16,402 - User['hdfs'] {'ignore_failures': False, 'groups': ['hadoop', 'hdfs']}
2016-04-19 12:28:16,404 - Directory['/etc/hadoop'] {'mode': 0755}
2016-04-19 12:28:16,406 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 0777}
2016-04-19 12:28:16,420 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2016-04-19 12:28:16,461 - Directory['/var/log/hadoop'] {'owner': 'root', 'mode': 0775, 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-04-19 12:28:16,464 - Directory['/var/run/hadoop'] {'owner': 'root', 'group': 'root', 'recursive': True, 'cd_access': 'a'}
2016-04-19 12:28:16,465 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'recursive': True, 'cd_access': 'a'}
2016-04-19 12:28:16,475 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
2016-04-19 12:28:16,482 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2016-04-19 12:28:16,664 - Directory['/etc/ams-hbase/conf'] {'owner': 'ams', 'group': 'hadoop', 'recursive': True}
2016-04-19 12:28:16,666 - Directory['/var/lib/ambari-metrics-collector/hbase-tmp'] {'owner': 'ams', 'recursive': True, 'cd_access': 'a'}
2016-04-19 12:28:16,667 - Directory['/var/lib/ambari-metrics-collector/hbase-tmp/local/jars'] {'owner': 'ams', 'cd_access': 'a', 'group': 'hadoop', 'mode': 0775, 'recursive': True}
2016-04-19 12:28:16,668 - XmlConfig['hbase-site.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ams-hbase/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-04-19 12:28:16,682 - Generating config: /etc/ams-hbase/conf/hbase-site.xml
2016-04-19 12:28:16,682 - File['/etc/ams-hbase/conf/hbase-site.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-04-19 12:28:16,711 - XmlConfig['hbase-policy.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ams-hbase/conf', 'configuration_attributes': {}, 'configurations': {'security.admin.protocol.acl': '*', 'security.masterregion.protocol.acl': '*', 'security.client.protocol.acl': '*'}}
2016-04-19 12:28:16,721 - Generating config: /etc/ams-hbase/conf/hbase-policy.xml
2016-04-19 12:28:16,722 - File['/etc/ams-hbase/conf/hbase-policy.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-04-19 12:28:16,735 - File['/etc/ams-hbase/conf/hbase-env.sh'] {'content': InlineTemplate(...), 'owner': 'ams'}
2016-04-19 12:28:16,741 - File['/etc/ams-hbase/conf/hadoop-metrics2-hbase.properties'] {'content': Template('hadoop-metrics2-hbase.properties.j2'), 'owner': 'ams', 'group': 'hadoop'}
2016-04-19 12:28:16,742 - TemplateConfig['/etc/ams-hbase/conf/regionservers'] {'owner': 'ams', 'template_tag': None}
2016-04-19 12:28:16,744 - File['/etc/ams-hbase/conf/regionservers'] {'content': Template('regionservers.j2'), 'owner': 'ams', 'group': None, 'mode': None}
2016-04-19 12:28:16,745 - Directory['/var/run/ambari-metrics-collector/'] {'owner': 'ams', 'recursive': True}
2016-04-19 12:28:16,745 - Directory['/var/log/ambari-metrics-collector'] {'owner': 'ams', 'recursive': True}
2016-04-19 12:28:16,745 - Directory['/var/lib/ambari-metrics-collector/hbase'] {'owner': 'ams', 'recursive': True, 'cd_access': 'a'}
2016-04-19 12:28:16,746 - File['/var/run/ambari-metrics-collector//distributed_mode'] {'owner': 'ams', 'action': ['delete']}
2016-04-19 12:28:16,746 - File['/etc/ams-hbase/conf/log4j.properties'] {'content': ..., 'owner': 'ams', 'group': 'hadoop', 'mode': 0644}
2016-04-19 12:28:16,747 - Directory['/etc/ams-hbase/conf'] {'owner': 'ams', 'group': 'hadoop', 'recursive': True}
2016-04-19 12:28:16,747 - Directory['/var/lib/ambari-metrics-collector/hbase-tmp'] {'owner': 'ams', 'recursive': True, 'cd_access': 'a'}
2016-04-19 12:28:16,748 - Directory['/var/lib/ambari-metrics-collector/hbase-tmp/local/jars'] {'owner': 'ams', 'cd_access': 'a', 'group': 'hadoop', 'mode': 0775, 'recursive': True}
2016-04-19 12:28:16,748 - XmlConfig['hbase-site.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ams-hbase/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-04-19 12:28:16,759 - Generating config: /etc/ams-hbase/conf/hbase-site.xml
2016-04-19 12:28:16,759 - File['/etc/ams-hbase/conf/hbase-site.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-04-19 12:28:16,792 - XmlConfig['hbase-policy.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ams-hbase/conf', 'configuration_attributes': {}, 'configurations': {'security.admin.protocol.acl': '*', 'security.masterregion.protocol.acl': '*', 'security.client.protocol.acl': '*'}}
2016-04-19 12:28:16,803 - Generating config: /etc/ams-hbase/conf/hbase-policy.xml
2016-04-19 12:28:16,804 - File['/etc/ams-hbase/conf/hbase-policy.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-04-19 12:28:16,818 - File['/etc/ams-hbase/conf/hbase-env.sh'] {'content': InlineTemplate(...), 'owner': 'ams'}
2016-04-19 12:28:16,822 - File['/etc/ams-hbase/conf/hadoop-metrics2-hbase.properties'] {'content': Template('hadoop-metrics2-hbase.properties.j2'), 'owner': 'ams', 'group': 'hadoop'}
2016-04-19 12:28:16,822 - TemplateConfig['/etc/ams-hbase/conf/regionservers'] {'owner': 'ams', 'template_tag': None}
2016-04-19 12:28:16,824 - File['/etc/ams-hbase/conf/regionservers'] {'content': Template('regionservers.j2'), 'owner': 'ams', 'group': None, 'mode': None}
2016-04-19 12:28:16,825 - Directory['/var/run/ambari-metrics-collector/'] {'owner': 'ams', 'recursive': True}
2016-04-19 12:28:16,825 - Directory['/var/log/ambari-metrics-collector'] {'owner': 'ams', 'recursive': True}
2016-04-19 12:28:16,826 - File['/etc/ams-hbase/conf/log4j.properties'] {'content': ..., 'owner': 'ams', 'group': 'hadoop', 'mode': 0644}
2016-04-19 12:28:16,827 - Directory['/etc/ambari-metrics-collector/conf'] {'owner': 'ams', 'group': 'hadoop', 'recursive': True}
2016-04-19 12:28:16,827 - Directory['/var/lib/ambari-metrics-collector/checkpoint'] {'owner': 'ams', 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-04-19 12:28:16,827 - XmlConfig['ams-site.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ambari-metrics-collector/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-04-19 12:28:16,837 - Generating config: /etc/ambari-metrics-collector/conf/ams-site.xml
2016-04-19 12:28:16,838 - File['/etc/ambari-metrics-collector/conf/ams-site.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-04-19 12:28:16,871 - XmlConfig['hbase-site.xml'] {'owner': 'ams', 'group': 'hadoop', 'conf_dir': '/etc/ambari-metrics-collector/conf', 'configuration_attributes': {}, 'configurations': ...}
2016-04-19 12:28:16,881 - Generating config: /etc/ambari-metrics-collector/conf/hbase-site.xml
2016-04-19 12:28:16,881 - File['/etc/ambari-metrics-collector/conf/hbase-site.xml'] {'owner': 'ams', 'content': InlineTemplate(...), 'group': 'hadoop', 'mode': None, 'encoding': 'UTF-8'}
2016-04-19 12:28:16,910 - File['/etc/ambari-metrics-collector/conf/log4j.properties'] {'content': ..., 'owner': 'ams', 'group': 'hadoop', 'mode': 0644}
2016-04-19 12:28:16,914 - File['/etc/ambari-metrics-collector/conf/ams-env.sh'] {'content': InlineTemplate(...), 'owner': 'ams'}
2016-04-19 12:28:16,915 - Directory['/var/log/ambari-metrics-collector'] {'owner': 'ams', 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-04-19 12:28:16,915 - Directory['/var/run/ambari-metrics-collector'] {'owner': 'ams', 'group': 'hadoop', 'recursive': True, 'cd_access': 'a'}
2016-04-19 12:28:16,916 - File['/usr/lib/ams-hbase/bin/hadoop'] {'owner': 'ams', 'mode': 0755}
2016-04-19 12:28:16,916 - Directory['/etc/security/limits.d'] {'owner': 'root', 'group': 'root', 'recursive': True}
2016-04-19 12:28:16,918 - File['/etc/security/limits.d/ams.conf'] {'content': Template('ams.conf.j2'), 'owner': 'root', 'group': 'root', 'mode': 0644}
2016-04-19 12:28:16,919 - Execute['ambari-sudo.sh rm -rf /var/lib/ambari-metrics-collector/hbase-tmp/*.tmp'] {}
2016-04-19 12:28:16,929 - Execute['/usr/sbin/ambari-metrics-collector --config /etc/ambari-metrics-collector/conf start'] {'user': 'ams'}
Command aborted. Aborted by user2. And Oozie is shutting down on its own. I had to start it for every two minutes every time manually.
Any useful suggestion can be helpful.
Thanks in advance
Created 04-21-2016 06:11 PM
Please take a look at Troubleshooting wiki at:
https://cwiki.apache.org/confluence/display/AMBARI/Troubleshooting+Guide
Ambari Metrics official documentation:
Created 04-20-2016 08:00 AM
Hi @karthik sai,
what do you see in ambari-mertrics-log (Default path /var/log/ambari-metrics-collector/)?
Because it seems that you have stopped from user, there must be another reason.
Created 05-03-2016 06:58 AM
And what can you say about the oozie server? Why is it shutting down on its own?
Created 04-21-2016 06:11 PM
Please take a look at Troubleshooting wiki at:
https://cwiki.apache.org/confluence/display/AMBARI/Troubleshooting+Guide
Ambari Metrics official documentation: