Reply
Explorer
Posts: 12
Registered: ‎04-11-2017

cloudera manager failed to stop all services

Hello,

after I rebooted the computer without shutdown all cloudera service first,

the cloudera management service and all services cannot be stopped or re-started.

the re-start requires to stop first, I tried to stop the service and got the following below.

 

Problem accessing /cmf/process/1282/logs. Reason:

    http://cahive-master01:9000/process/1282-cloudera-mgmt-ALERTPUBLISHER/files/logs/stdout.log
The server declined access to the page or resource.

Problem accessing /cmf/process/1334/logs. Reason:

    http://cahive-master01:9000/process/1334-spark_on_yarn-SPARK_YARN_HISTORY_SERVER/files/logs/stderr.log
The server declined access to the page or resource.

 

I searched the cloudera exising user's solution to this problem is to delete all node and then add back. is there a reason for this problem and a better solution? thanks

Explorer
Posts: 12
Registered: ‎04-11-2017

Re: cloudera manager failed to stop all services

I nocied that directory  in /run/cloudera-scm-agent/process is empty, not sure why no directory is created over there

Champion
Posts: 710
Registered: ‎05-16-2016

Re: cloudera manager failed to stop all services

Could you share the cloudera scm server logs and agent logs full stack trace that has exceptions or error . 

if the directory is empty that means something in cloudera scm agent   we can narrow down it if you provide the above logs. 

Explorer
Posts: 12
Registered: ‎04-11-2017

Re: cloudera manager failed to stop all services

Hi 

thanks for your help. I checked the log file and found the problem is related to openjdk installed by another engineer.

after I removed it, it works fine. however, I have one problem unable to start hive in one machine, not sure what could be the cause? here is the error message in stderr.log.  many thanks

 

18/02/14 14:29:09 ERROR conf.Configuration: error parsing conf core-default.xml
javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
	at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:982)
	at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1032)
	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1433)
	at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:67)
	at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81)
	at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96)
	at org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool.main(MapreduceDependencyClasspathTool.java:70)
Exception in thread "main" java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2659)

 

Posts: 589
Topics: 1
Kudos: 140
Solutions: 74
Registered: ‎04-22-2014

Re: cloudera manager failed to stop all services

@cloudtiger,

 

This error looks like it comes from within the server you are starting.  I would check the log of the role you are starting.  There may be more information or context in that file or the stderr.log.  Please share the entire stderr.log and also any stack traces in the role log (HiveServer2 or Hive Metastore).

 

Thanks!

 

Ben

Explorer
Posts: 12
Registered: ‎04-11-2017

Re: cloudera manager failed to stop all services

ben,

thanks.

the log file is way too big, I just checked again and found the following error:

Can't open /run/cloudera-scm-agent/process/1431-hive-HIVEMETASTORE/supervisor.conf: Permission denied.

 

all files in directory /run/cloudera-scm-agent/process/ 

is owned by hive except this file.

 

      -rw-------  1 root root  3430 Feb 15 14:30 supervisor.conf

 

 

I have two hive servers, the one works has the same permission issue. but it doesn't prevent it from running successfully.

 

Here is the more detailed log file information:

+ exec /opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/lib/hive/bin/hive --config /run/cloudera-scm-agent/process/1431-hive-HIVEMETASTORE --service metastore -p 9083
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
18/02/15 14:30:16 ERROR conf.Configuration: error parsing conf core-default.xml
javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
	at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:982)
	at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1032)
	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1433)
	at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:67)
	at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81)
	at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96)
	at org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool.main(MapreduceDependencyClasspathTool.java:70)
Exception in thread "main" java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2659)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:982)
	at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1032)
	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1433)
	at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:67)
	at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81)
	at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96)
	at org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool.main(MapreduceDependencyClasspathTool.java:70)
Caused by: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
	at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541)
	... 9 more
18/02/15 14:30:17 ERROR conf.Configuration: error parsing conf core-default.xml
javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
	at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
	at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
	at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
	at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:525)
	at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:543)
	at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:437)
	at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2652)
	at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:2611)
	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:74)
	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:58)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6083)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Exception in thread "main" java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2659)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409)
	at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144)
	at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116)
	at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:525)
	at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:543)
	at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:437)
	at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2652)
	at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:2611)
	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:74)
	at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:58)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6083)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized.
	at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541)
	... 18 more
+ date

 it looks that the following log files do not have any informationl

/var/log/hive/hadoop-cmf-hive-HIVESERVER2-cahive-master01.log.out and 

/var/log/hive/hadoop-cmf-hive-HIVESERVER2-cahive-master01.log.out

 

also, I tried to start the server from command line and it dosn't work.

 

$ sudo service hive-server2 start

Redirecting to /bin/systemctl start hive-server2.servic

Failed to start hive-server2.service: Unit not found.

 

 

thanks

 

Announcements