Created 02-24-2017 05:13 PM
I am trying to spin up an EC2 instance from a custom AMI.AMI is created from hortonworks virtual box file(vmdk). using a t2.2x large machine
Below error message is there in HiveServer2 ?
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X PUT --data-binary @/usr/hdp/2.5.0.0-1245/hadoop/mapreduce.tar.gz 'http://sandbox.hortonworks.com:50070/webhdfs/v1/hdp/apps/2.5.0.0-1245/mapreduce/mapreduce.tar.gz?op=CREATE&user.name=hdfs&overwrite=True&permission=444'' returned status_code=403.
After I manually start the data node it is up for some time(30 sec) ,then it goes down. I can see the below message
2017-02-24 17:02:58,376 - ***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** ***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** ***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** Directory /hadoop/hdfs/data became unmounted from / . Current mount point: /hadoop . Please ensure that mounts are healthy. If the mount change was intentional, you can update the contents of /var/lib/ambari-agent/data/datanode/dfs_data_dir_mount.hist. ***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** ***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** ***** WARNING ***** WARNING ***** WARNING ***** WARNING ***** WARNING *****
{ "RemoteException": { "exception": "IOException", "javaClassName": "java.io.IOException", "message": "Failed to find datanode, suggest to check cluster health." } } Let me know if somebody faced this issues and how to resolve. any help is appreciated.
Created 02-24-2017 05:50 PM
Could you elaborate on how you mounted /data drives? What type of AWS storage?
Created 02-27-2017 04:33 PM
I am using ebs storage.Please find the details from dfs_data_dir_mount.hist file
# Comments begin with a hash (#) symbol # dir,mount_point
/hadoop/hdfs/data,/hadoop
/hadoop,/
Error shown in the ambari data nodes logs are gone after the update. Intially datanode is started and then it goes to stopped status.
Created 02-24-2017 06:37 PM
Did you get a chance to review
It would be a good idea to review the hive and datanode logs as well. Also, please refrain pasting the public IP address of your AWS instance.