Ambari data node did manage to register manually as it was picked up by Ambari console. However, ambari console's register/confirm failed. Moreover, I can SSH master node (ambari server) and datanode only with ssh .pem username@address. FYI. The id_pub.rsa is the same for both machines. I tried ssh username@address. It failed with Access denied. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'ec2-xx.compute-1.amazonaws.com,xx' (ECDSA) to the list of known hosts. Permission denied (publickey,gssapi-keyex,gssapi-with-mic). ... View more

sorted. managed to ssh with .pem between instances. will post further ... View more

I hope someone can throw some light on this. Both nodes have the same set of files; the .pem key is common for both of them. the only difference is the master node server crash. I created a new instance, used the same .pem key for the master node and created a new volume using the ami. [ec2-user@namenode .ssh]$ sudo ssh -v -i ~/.ssh/UKBigDataKeypair.pem ec2-user@ec2-xx-x-xx-xxx.compute-1.amazonaws.com Warning: Identity file /home/ec2-user/.ssh/UKBigDataKeypair.pem not accessible: No such file or directory. OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 56: Applying options for * debug1: Connecting to ec2-xx-x-xx-xx.compute-1.amazonaws.com [xxx.xx.x.xxx] port 22. debug1: Connection established. debug1: permanently_set_uid: 0/0 debug1: identity file /root/.ssh/id_rsa type -1 debug1: identity file /root/.ssh/id_rsa-cert type -1 debug1: identity file /root/.ssh/id_dsa type -1 debug1: identity file /root/.ssh/id_dsa-cert type -1 debug1: identity file /root/.ssh/id_ecdsa type -1 debug1: identity file /root/.ssh/id_ecdsa-cert type -1 debug1: identity file /root/.ssh/id_ed25519 type -1 debug1: identity file /root/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.6.1 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1 debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16 debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16 debug1: sending SSH2_MSG_KEX_ECDH_INIT debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: xxx debug1: Host 'ec2-xx-x-xxx-xxx.compute-1.amazonaws.com' is known and matches the ECDSA host key. debug1: Found key in /root/.ssh/known_hosts:7 debug1: ssh_ecdsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available debug1: Unspecified GSS failure. Minor code may provide more information debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available debug1: Next authentication method: publickey debug1: Trying private key: /root/.ssh/id_rsa debug1: Trying private key: /root/.ssh/id_dsa debug1: Trying private key: /root/.ssh/id_ecdsa debug1: Trying private key: /root/.ssh/id_ed25519 debug1: No more authentication methods to try. Permission denied (publickey,gssapi-keyex,gssapi-with-mic). I don not know why it is suggesting "no kerberos credentials available" during ssh. I never installed/deployed kerberos. Previous instances have been cleaned out fully. ... View more

I am running into ssh problems. I checked both the nodes, which have authorized_keys, id_rsa, id_rsa.pub in the \home\ec2-user\.ssh\ folder I could ssh with .pem key from the data node to master node, which I could not even achieve it from the master to slave. ssh without keys/pws do not work at all. sudo ssh -i keypair.pem ec2-user@ec2-xx-x-xxx-xxx.compute-1.amazonaws.com Warning: Identity file keypair.pem not accessible: No such file or directory. Permission denied (publickey,gssapi-keyex,gssapi-with-mic). ... View more

@ Neeraj Sabharwal Sorry about bringing multiple issues as come in multiples! I have sorted out the access problem. I will focus on your recommendation and post how I got on. Thanks ... View more

Additional comment - this makes me a little suspicious of the way the instances were presented as ready to install. Mount: wrong fs type, bad option, bad superblock on /dev/xvdf, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so. ... View more

Thanks. I will check this out and post after fixing the ssh access denied (public key) issue of the master node, where ambari server is installed. I attached the volume to a running instance. The attached volume (of the master node) shows a GPT system. I am getting an unknown file system type GPT. Since both instances have same permissions, I want to clear this permission issue on the mounted volume and reattach to the master. 1: I do not what I cannot mount the volume with GPT type. Is there any recommendation please? 2: The linux instances were given in ready to install state. I wonder if GPT file type is not causing issues, considering I already successfully deployed Hadoop. Ambari automatically creates the HDFS format. So, writing data/output/codes in hadoop should not be an issue, should it? Warning: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion. Disk /dev/xvdf: 1099.5 GB, 1099511627776 bytes, 2147483648 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: gpt # Start End Size Type Name 1 2048 4095 1M BIOS boot parti 2 4096 2147483614 1024G Microsoft basic [ec2-user@datanode /]$ sudo mount /dev/xvdf2 /bad -t gpt mount: unknown filesystem type 'gpt' ... View more

Thanks Neeraj Sabharwal. I have a new problem to face 🙂 I can no longer access my master node from Putty, WinSCP or through browser. it throws an error: "Server refused or Key - (publickey,gssapi-keyex,gssapi-with-mic). Nothing has changed. I can use the same key and access the other instance! I need to see if I can attach the volume on the second node and check the issues. I have indeed setup ssh without pw as the earlier install was successful before the stop/start created issues - hence the elastic ip. The questions below are naive. Sorry for asking them! Do I need to enable this pw-less ssh every time I reboot/start the instances, ambari server and hadoop services please? Does it mean, ambari/hadoop services wont start after stop/restart please? How do check ssh localhost in ambari server please? I guess providing the ssh key for an automatic registration takes care of it! ... View more

Could someone please help ? I have not made any progress as I am unable to find the root cause. What I did for manual registration: /etc/sysconfig/network and etc/hosts contain relevant entries on stop/start/reboot Redhat on Ec2 recognises the changed hostnames installed, configured and started ambari-agent - datanode It could connect to the and establish a two-way ssl connection. Server refused registration - cert do not match / unknown host: datanode.teg. registration failed Automatic install clean uninstall of ambari* on datanode ambari server is running registration failed .. error: Permission denied (publickey,gssapi-keyex,gssapi-with-mic). SSH command execution finished host=namenode.teg, exitcode=255 Command end time 2016-03-04 19:51:42 ERROR: Bootstrap of host namenode.teg fails because previous action finished with non-zero exit code (255) ERROR MESSAGE: Permission denied (publickey,gssapi-keyex,gssapi-with-mic). STDOUT: Permission denied (publickey,gssapi-keyex,gssapi-with-mic). Please advise. I am unable to trouble shoot. ... View more

I think they are talking to each other - please see the agent log below: serverlog file: Serverout agent-log - seemed to have established connection...registration failed at datanode --- 2 way ssl auth turned off?? ... View more