Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Issues with HiveServer2 & Datanodes services in EC2.

Issues with HiveServer2 & Datanodes services in EC2.

Contributor

I am trying to spin up an EC2 instance from a custom AMI.AMI is created from hortonworks virtual box file(vmdk). using a t2.2x large machine

Below error message is there in HiveServer2 ?

resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.5.0.0-1245/hadoop/mapreduce.tar.gz 'http://sandbox.hortonworks.com:50070/webhdfs/v1/hdp/apps/2.5.0.0-1245/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'' returned status_code=403.

After I manually start the data node it is up for some time(30 sec) ,then it goes down. I can see the below message

2017-02-24 17:02:58,376 - 
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
Directory /hadoop/hdfs/data became unmounted from / . Current mount point: /hadoop . Please ensure that mounts are healthy. If the mount change was intentional, you can update the contents of /var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist.
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****

{
  "RemoteException": {
    "exception": "IOException", 
    "javaClassName": "java.io.IOException", 
    "message": "Failed to find datanode, suggest to check cluster health."
  }
}

Let me know if somebody faced this issues and how to resolve. any help is appreciated.
3 REPLIES 3
Highlighted

Re: Issues with HiveServer2 & Datanodes services in EC2.

@Joby Johny

Could you elaborate on how you mounted /data drives? What type of AWS storage?

Highlighted

Re: Issues with HiveServer2 & Datanodes services in EC2.

Contributor

I am using ebs storage.Please find the details from dfs_data_dir_mount.hist file

# Comments begin with a hash (#) symbol # dir,mount_point

/hadoop/hdfs/data,/hadoop

/hadoop,/

Error shown in the ambari data nodes logs are gone after the update. Intially datanode is started and then it goes to stopped status.

Highlighted

Re: Issues with HiveServer2 & Datanodes services in EC2.

Expert Contributor

@Joby Johny

Did you get a chance to review

https://community.hortonworks.com/questions/29538/detected-data-dirs-that-became-unmounted-and-are-n...

It would be a good idea to review the hive and datanode logs as well. Also, please refrain pasting the public IP address of your AWS instance.

Don't have an account?
Coming from Hortonworks? Activate your account here