Issues with HiveServer2 & Datanodes services in EC2.

[#1]

I am trying to spin up an EC2 instance from a custom AMI.AMI is created from hortonworks virtual box file(vmdk). using a t2.2x large machine

Below error message is there in HiveServer2 ?

resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.5.0.0-1245/hadoop/mapreduce.tar.gz 'http://sandbox.hortonworks.com:50070/webhdfs/v1/hdp/apps/2.5.0.0-1245/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'' returned status_code=403.

After I manually start the data node it is up for some time(30 sec) ,then it goes down. I can see the below message

2017-02-24 17:02:58,376 - 
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
Directory /hadoop/hdfs/data became unmounted from / . Current mount point: /hadoop . Please ensure that mounts are healthy. If the mount change was intentional, you can update the contents of /var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist.
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****

{
  "RemoteException": {
    "exception": "IOException", 
    "javaClassName": "java.io.IOException", 
    "message": "Failed to find datanode, suggest to check cluster health."
  }
}

Let me know if somebody faced this issues and how to resolve. any help is appreciated.

[#2]

@Joby Johny

Could you elaborate on how you mounted /data drives? What type of AWS storage?

[#3]

I am using ebs storage.Please find the details from dfs_data_dir_mount.hist file

# Comments begin with a hash (#) symbol # dir,mount_point

/hadoop/hdfs/data,/hadoop

/hadoop,/

Error shown in the ambari data nodes logs are gone after the update. Intially datanode is started and then it goes to stopped status.

[#4]

@Joby Johny

Did you get a chance to review

https://community.hortonworks.com/questions/29538/detected-data-dirs-that-became-unmounted-and-are-n...

It would be a good idea to review the hive and datanode logs as well. Also, please refrain pasting the public IP address of your AWS instance.