Created 02-13-2018 05:33 PM
Hello,
after I rebooted the computer without shutdown all cloudera service first,
the cloudera management service and all services cannot be stopped or re-started.
the re-start requires to stop first, I tried to stop the service and got the following below.
Problem accessing /cmf/process/1282/logs. Reason:
http://cahive-master01:9000/process/1282-cloudera-mgmt-ALERTPUBLISHER/files/logs/stdout.log The server declined access to the page or resource.
Problem accessing /cmf/process/1334/logs. Reason:
http://cahive-master01:9000/process/1334-spark_on_yarn-SPARK_YARN_HISTORY_SERVER/files/logs/stderr.log The server declined access to the page or resource.
I searched the cloudera exising user's solution to this problem is to delete all node and then add back. is there a reason for this problem and a better solution? thanks
Created 02-13-2018 05:57 PM
I nocied that directory in /run/cloudera-scm-agent/process is empty, not sure why no directory is created over there
Created 02-13-2018 06:36 PM
Could you share the cloudera scm server logs and agent logs full stack trace that has exceptions or error .
if the directory is empty that means something in cloudera scm agent we can narrow down it if you provide the above logs.
Created 02-14-2018 03:21 PM
Hi csguna,
thanks for your help. I checked the log file and found the problem is related to openjdk installed by another engineer.
after I removed it, it works fine. however, I have one problem unable to start hive in one machine, not sure what could be the cause? here is the error message in stderr.log. many thanks
18/02/14 14:29:09 ERROR conf.Configuration: error parsing conf core-default.xml javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.get(Configuration.java:982) at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1032) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1433) at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:67) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96) at org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool.main(MapreduceDependencyClasspathTool.java:70) Exception in thread "main" java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2659)
Created 02-15-2018 08:50 AM
This error looks like it comes from within the server you are starting. I would check the log of the role you are starting. There may be more information or context in that file or the stderr.log. Please share the entire stderr.log and also any stack traces in the role log (HiveServer2 or Hive Metastore).
Thanks!
Ben
Created 02-15-2018 12:15 PM
ben,
thanks.
the log file is way too big, I just checked again and found the following error:
Can't open /run/cloudera-scm-agent/process/1431-hive-HIVEMETASTORE/supervisor.conf: Permission denied.
all files in directory /run/cloudera-scm-agent/process/
is owned by hive except this file.
-rw------- 1 root root 3430 Feb 15 14:30 supervisor.conf
I have two hive servers, the one works has the same permission issue. but it doesn't prevent it from running successfully.
Here is the more detailed log file information:
+ exec /opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/lib/hive/bin/hive --config /run/cloudera-scm-agent/process/1431-hive-HIVEMETASTORE --service metastore -p 9083 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 18/02/15 14:30:16 ERROR conf.Configuration: error parsing conf core-default.xml javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.get(Configuration.java:982) at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1032) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1433) at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:67) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96) at org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool.main(MapreduceDependencyClasspathTool.java:70) Exception in thread "main" java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2659) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.get(Configuration.java:982) at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1032) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1433) at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:67) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96) at org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool.main(MapreduceDependencyClasspathTool.java:70) Caused by: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541) ... 9 more 18/02/15 14:30:17 ERROR conf.Configuration: error parsing conf core-default.xml javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116) at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:525) at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:543) at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:437) at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2652) at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:2611) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:74) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:58) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6083) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Exception in thread "main" java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2659) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116) at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:525) at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:543) at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:437) at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2652) at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:2611) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:74) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:58) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6083) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541) ... 18 more + date
it looks that the following log files do not have any informationl
/var/log/hive/hadoop-cmf-hive-HIVESERVER2-cahive-master01.log.out and
/var/log/hive/hadoop-cmf-hive-HIVESERVER2-cahive-master01.log.out
also, I tried to start the server from command line and it dosn't work.
$ sudo service hive-server2 start
Redirecting to /bin/systemctl start hive-server2.servic
Failed to start hive-server2.service: Unit not found.
thanks