ben, thanks. the log file is way too big, I just checked again and found the following error: Can't open /run/cloudera-scm-agent/process/1431-hive-HIVEMETASTORE/supervisor.conf: Permission denied.   all files in directory /run/cloudera-scm-agent/process/  is owned by hive except this file.         -rw-------  1 root root  3430 Feb 15 14:30 supervisor.conf     I have two hive servers, the one works has the same permission issue. but it doesn't prevent it from running successfully.   Here is the more detailed log file information: + exec /opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/lib/hive/bin/hive --config /run/cloudera-scm-agent/process/1431-hive-HIVEMETASTORE --service metastore -p 9083 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/lib/zookeeper/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 18/02/15 14:30:16 ERROR conf.Configuration: error parsing conf core-default.xml javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.get(Configuration.java:982) at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1032) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1433) at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:67) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96) at org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool.main(MapreduceDependencyClasspathTool.java:70) Exception in thread "main" java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2659) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.get(Configuration.java:982) at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1032) at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1433) at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBaseConfiguration.java:67) at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseConfiguration.java:81) at org.apache.hadoop.hbase.HBaseConfiguration.create(HBaseConfiguration.java:96) at org.apache.hadoop.hbase.util.MapreduceDependencyClasspathTool.main(MapreduceDependencyClasspathTool.java:70) Caused by: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541) ... 9 more 18/02/15 14:30:17 ERROR conf.Configuration: error parsing conf core-default.xml javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116) at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:525) at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:543) at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:437) at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2652) at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:2611) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:74) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:58) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6083) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Exception in thread "main" java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2659) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2503) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2409) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1144) at org.apache.hadoop.conf.Configuration.set(Configuration.java:1116) at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:525) at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:543) at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:437) at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2652) at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:2611) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:74) at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:58) at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:6083) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2541) ... 18 more + date  it looks that the following log files do not have any informationl /var/log/hive/hadoop-cmf-hive-HIVESERVER2-cahive-master01.log.out and  /var/log/hive/hadoop-cmf-hive-HIVESERVER2-cahive-master01.log.out   also, I tried to start the server from command line and it dosn't work.   $ sudo service hive-server2 start Redirecting to /bin/systemctl start hive-server2.servic Failed to start hive-server2.service: Unit not found.     thanks   ... View more

Hello,   This python script helps to remove the hosts from the cluster. The following are the steps:   1. stop and decommission all roles in a host 2. remove the roles from a host identify and delete the roles one by one 3. remove host from a cluster 4. remove host from cloudera manager This script removes the hosts from the cloudera managed cluster running in aws. It is intend to scale down the worker node(node manager role) and gateway role from the cluster once the demand is over. You can change the script accordingly based on your environment.   #!/bin/python import httplib2 import os import requests import json import boto3 import time from requests.auth import HTTPBasicAuth os.environ["AWS_ACCESS_KEY_ID"] = "ACCESS_KEY" os.environ["AWS_SECRET_ACCESS_KEY"] = "SECRET_ACCESS_KEY" os.environ["AWS_DEFAULT_REGION"] = "us-east-1" region='us-east-1' metadata = requests.get(url='http://169.254.169.254/latest/meta-data/instance-id') instance_id = metadata.text host = requests.get(url='http://169.254.169.254/latest/meta-data/hostname') host_id = host.text username='admin' password='admin' cluster_name='cluster001' scm_protocol='http' scm_host='host.compute-1.amazonaws.com' scm_port='7180' scm_api='v17'   client = boto3.client('autoscaling') ec2 = boto3.client('autoscaling', region_name=region) response = client.describe_auto_scaling_instances(InstanceIds=[instance_id,]) state = response['AutoScalingInstances'][0]['LifecycleState'] print "vm is in " + state if state == 'Terminating:Wait': print "host decommision started" ##decommission host service_url = scm_protocol + '://' + scm_host + ':' + scm_port + '/api/' + scm_api + '/cm/commands/hostsDecommission' #service_url = scm_protocol + '://' + scm_host + ':' + scm_port + '/api/' + scm_api + '/cm/hostsRecommission' #service_url = scm_protocol + '://' + scm_host + ':' + scm_port + '/api/' + scm_api + '/cm/commands/hostsStartRoles' print service_url headers = {'content-type': 'application/json'} req_body = { "items":[ host_id ]} print req_body req = requests.post(url=service_url, auth=HTTPBasicAuth(username, password), data=json.dumps(req_body), headers=headers) print req.text time.sleep(120) ##delete roles in a host api_url = scm_protocol + '://' + scm_host + ':' + scm_port + '/api/' + scm_api + '/hosts/' + host_id req = requests.get(api_url, auth=HTTPBasicAuth(username, password)) a = json.loads(req.content) for i in a['roleRefs']: scm_uri='/api/' + scm_api + '/clusters/' + cluster_name + '/services/'+i['serviceName']+'/roles/'+i['roleName'] scm_url = scm_protocol + '://' + scm_host + ':' + scm_port + scm_uri print scm_url req = requests.delete(scm_url, auth=HTTPBasicAuth(username, password)) print req.text time.sleep(10) ##remove host from cluster service_url = scm_protocol + '://' + scm_host + ':' + scm_port + '/api/' + scm_api + '/clusters/' + cluster_name + '/hosts/' + host_id print service_url req = requests.delete(service_url, auth=HTTPBasicAuth(username, password)) time.sleep(10) ##remove host from cloudera manager os.system("/etc/init.d/cloudera-scm-agent stop") service_url = scm_protocol + '://' + scm_host + ':' + scm_port + '/api/' + scm_api + '/hosts/' + host_id print service_url req = requests.delete(service_url, auth=HTTPBasicAuth(username, password)) print req.text time.sleep(10) ##refresh cluster configuration service_url = scm_protocol + '://' + scm_host + ':' + scm_port + '/api/' + scm_api + '/clusters/' + 'commands/refresh' print service_url req = requests.post(service_url, auth=HTTPBasicAuth(username, password)) print req.text time.sleep(10) ##deploy client configuration service_url = scm_protocol + '://' + scm_host + ':' + scm_port + '/api/' + scm_api + '/clusters/' + 'commands/deployClientConfig' print service_url req = requests.post(service_url, auth=HTTPBasicAuth(username, password)) print req.text time.sleep(10) Best Regards, Radhakrishnan Rk ... View more

try add allow port on firewalld on host, this solved my prob $ firewall-cmd --zone=public --add-port=9000/tcp (Centos7) or $ service firewalld disable Hope this may help ... View more

The issue was with java heap space on the data nodes. Increasing the java heap space based on the data node block size resolved the issue. ... View more

@hamed2005,   I looked more closely at the exception and now think I understand...   com.mysql.jdbc.ExportControlled.getSSLSocketFactoryDefaultOrConfigured(ExportControlled.java:428)   It seems that the MySQL JDBC driver may require a password.  I was testing in an environment where I wasn't using TLS for my db connection, so it didn't matter.   Not sure how to get around this... I'll let you know if I come up with something   ... View more

I got rid of cloudera-manager.repo file and was able to install it successfully. I agree with you that i got caught in the switch-over. Thank you for your advice. ... View more

Thank you Michael. I appreciate the links. ... View more