@sparkd,   While we can't be sure, it is likely that some permissions were changed on the /tmp directory so that the Service Monitor (that executes the HDFS canary health check) could not access the directory.  Service Monitor utilizes the "hue" user and principal to access other resources so it is reasonable to assume that /tmp in HDFS did not allow the hue user or group to write to /tmp.   Are you having similar trouble?  If so, check your service monitor log file for stack traces and errors related to the hdfs canary. ... View more

Hi Roberto, Thank you for asking! 1) Please kindly provide the name and date/time of creation of the cluster. Similarly, if this fails for termination, please provide the same for the time the termination was attempted. 2) Do you repeatedly see the same cluster creation failure over Azure? 3) For Azure-service related issues with Altus clusters, please gather the following info from the Azure Portal: - Login to https://portal.azure.com - Navigate to Resource Groups - Select the Resource Group of the cluster in question - On the left hand column under Overview, select Activity Log - Filter Event Severity to Error and Critical - Review the Operations Names that have Failed or Critical Statuses, click any related items, select JSON and copy the contents to the filed support case. Or if you prefer, we can coordinate a webex session to gather this information) Regards, Daisuke ... View more

Thank you for uploading the files! Okay, let us investigate to identify what is going on. Daisuke ... View more

I did the changes in the CM but didn't update my files... I had to add: default_tgs_enctypes = rc4-hmac aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc default_tkt_enctypes = rc4-hmac aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc permitted_enctypes = rc4-hmac aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc in every host manually... after that things starting to work ... View more

@bgooley ok. I'll submit a new post and upload my agent log.  ... View more

Hi Dice,   I am using HBase version - 1.2.0-cdh5.8.3 I am getting similar error. Can you please take a look? @dice wrote: Both HBASE-14533 and HBASE-14196 are being included in CDH 5.6. Can you upload 1) the client log, 2) thriftserver log, and 3) the result of 'hbase version' command in your console? Dice. 2017-02-27 12:47:28,123 ERROR [main] org.apache.hadoop.hbase.client.AsyncProcess: Failed to get region location org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Call to XXXXX:60020 failed on local exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to XXXX:60020 is closing. Call id=34, waitTime=4 at org.apache.hadoop.hbase.ipc.AbstractRpcClient.wrapException(AbstractRpcClient.java:289) at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1272) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336) at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:34070) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1589) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1398) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1199) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:395) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:344) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:230) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:146) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.mutate(BufferedMutatorImpl.java:113) at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:138) at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:94) at org.apache.pig.backend.hadoop.hbase.HBaseStorage.putNext(HBaseStorage.java:931) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98) at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:467) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:432) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:412) at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:256) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1776) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: Connection to XXXX:60020 is closing. Call id=34, waitTime=4 at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.cleanupCalls(RpcClientImpl.java:1084) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.close(RpcClientImpl.java:863) at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.run(RpcClientImpl.java:580) ... View more

Do you find something related error in the postgres logs that's located under /var/lib/cloudera-scm-server-db/data/pg_log/? ... View more

Ubuntu has the issue on OpenStack,that is,when we do with ’sudo’ on the Ubuntu instance. it checks resolver thus can’t make resolution or wait the time to reply and at worst it going to fail with ‘timeout’.perhaps when you formatting hdfs,it needs resolver thus the failure happens as well.   ubuntu@sahara-test1:~$ sudo -i sudo: unable to resolve host sahara-test1 ubuntu@sahara-test1:~$ sudo apt update sudo: unable to resolve host sahara-test1   The root cause is the difference between /etc/hostname and /etc/host,that is,hostname is changed by instance name but /etc/hosts won’t be updated by the name.the the difference is happened both Ubuntu and CentOS,but the resolver issue is happened only Ubuntu.CentOS doesn’t get any error for the issue. The solution is we could use updating the /etc/hosts with cloud-init. This cloud-init can set the hostname and the FQDN, as well as updating /etc/hosts on the instance.   #cloud-config hostname: sahara-test1 fqdn: sahara-test1.localdomain manage_etc_hosts: true   for more detail,please check following the link about could-init for hostname. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/4/html/End_User_Guide/user-data.html Certainly Mirantis sets the DNS test through google DNS (www.google.com 8.8.8.8 or 8.8.4.4) so the issue might happen. And the issue should be affected sahara diskimage-create. https://github.com/openstack/sahara-image-elements perhaps the cloud-init should be added to post-install.d on hadoop-cloudera https://github.com/openstack/sahara-image-elements/tree/master/elements/hadoop-cloudera ... View more

You can change the parcel directory from /opt/cloudera/parcels (by default) to given directory per the following guide: Before following the steps, please shutdown the cluster for a safe. http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_parcels.html?scroll=cmug_topic_7_11_5_unique_1__section_srx_xyx_bm_unique_1 Configuring the Host Parcel Directory Otherwise, you need to add more disk to /opt. ... View more

You need to install the following:   sudo yum install sentry-hdfs-plugin   ... View more

Hi Dice,    Have you had the chance to look at those logs?  ... View more

What is the i18n configuration in those targeted hosts? ... View more

It is pointing to /dfs/nn and there is ./current folder in there I think it's corrupted. I replaced with /dfs/snn/current and it's still not working Then I use Cloudera Manager to redeploy the Namenode / Secondary Namenode to some other VMs and it seems to work .. for now. Strange. ... View more

To enable Kerberos in your CM managed CDH cluster, please follow the document below: http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cm_sg_authentication.html As you've already noticed, files under /etc take effect only on the client programs. HTH. -- Sent from my mobile ... View more

In addition to Dice's notes, please also read the design and efficiency overview at http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html#Overview. It will help gain a better understanding of the feature. ... View more