Error Installing HDB HAWQ Standby Master

[#1]

I am not able to install the HAWQ Standby master on an AWS cluster running HDP 2.4.2 and Ambari 2.2.2 Here is the error: "This can be run only on master or standby host"

Not sure what that means. It is not being installed on a DN, with PFX installed. It is not bing installed on the Ambari node. I am using the Name Node (since I only have 3 HDP master nodes) to install the HAWQ Standby Master. I attempted to remove the HAWQ Standby (that does not start) from the Name node and placed it on another node just to test. It gives the same error. So right now I am just running without a standby master. See

How do I begin trouble shooting this?

[#2]

Looking at the source code does not help:

https://github.com/apache/incubator-hawq/blob/master/tools/bin/hawq

[#3]

It can be a network configuration issue .Having said that could you please make sure the following

1.Confirm you are able to do ssh between the master and standby master ( Please make sure you can ssh to localhost as well)

2.Master ip addresses are different.

3. Please try to intialize standby master from the command line and share the logs

The hawq init object --standby-host= option allows defining the host for a standby at initialization.

Also if you can upload the logs from hawqAdminLogs it would be useful

[#4]

@Pratheesh Nair

There are no logs on /data/hawq/masteron the HAWQ standby master. In fact nothing is created there unlike on the HAWQ master node.

Yes the ips are different for the masters.

When you run the command from the command line, it immediately returns

This can be run only on master or standby host

Even in verbose mode. Yes I can do passwordless ssh if I provide the -i with pem file (as this is AWS) i.e. ssh -i <.pem> node I cannot do a ssh node directly without passing the -i option.

[#5]

I hope you ran the command from the hawq master node only .Also can you give the output of gp_segement_configuration table .

su - gpadmin

psql template1

select * from gp_segement_configuration

[#6]

@Pratheesh Nair

[#7]

@Pratheesh Nair ok I did not run it from hawq master, because Ambari is trying to run it from the HAWQ Standby node. So I ran it from the HAWQ Master and got the dreaded error:

gpadmin-[ERROR]:-Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

So I am attempting to run as gpadmin

hawq ssh-exkeys -f Hosts

It is asking for password for each hosts I am attempting to set up passwordless ssh for the gpadmin user (since this is AWS, it was setup for the ec2-user) Hopefully that may solve it.

[#8]

Yep .It will resolve the issue .Also please make sure you can do a "ssh localhost" as well after the key exchange

[#9]

@Ancil McBarnett I'm curious as to why you ran into this issue. Ambari should have automatically created gpadmin account and exchanged keys for you.

Unless you manually created gpadmin account, and gave a different password on HAWQ config screen in Ambari during install?

http://hdb.docs.pivotal.io/hdb/install/install-ambari.html

[#10 · Accepted solution]

@Vineet @Pratheesh Nair Ok.. Here is solution

So apparently if you are installing on Amazon AWS EC2 (remember you are only given access to ec2-user, not root), and if you decided to NOT do passwordless ssh with default key named id_rsa for that ec2-user when installing Ambari and its agents, when you try to install Hortonworks HDB (HAWQ) via Ambari, Ambari WILL NOT exchange the keys for the gpadmin user for you. It would create the gpadmin user with the password you give it on the HAWQ config screen during install, but no keys exchanged.

NOTE: for my nodes I had a key for ssh that was NOT default name of id_rsa. I do not know if this is a combination of using a non root user or the fact that ec2-user did not have its own passwordless ssh with default key named id_rsa. In anycase ONLY on the HAWQ Master, for gpadmin would the keys exist. If you tried the following to generate keys on the HAWQ Master you would still get an error, where it would NOT even accept the default gpadmin user password you set, even though it works. That was surprising.

su gpadmin

> source /usr/local/hawq/greenplum_path.sh
> hawq ssh-exkeys -f Hosts

>Enter password for existing user for node <......>
>Enter password for existing user for node <......>
>Enter password for existing user for node <......>

So in essence you have to manually go to each node, and copy authorized_keys file from HAWQ Master to each node (chmod 600), into the /home/gpadmin/.ssh/ so that you can at least password ssh from that HAWQ master node. Then you run the ssh-exkeys manually and it would work.

[#11]

I'll just add two more bits that I have seen some people run into often. Hopefully this is helpful in similar scenarios.

1) /etc/ssh/sshd_config file: PasswordAuthentication should be set to yes

2) In AWS, /etc/cloud/cloud.cfg should have ssh_pwauth set to True (or 1, I think). Otherwise, cloud init keeps disabling PasswordAuthentication options in ssh config.

[#12]

For me this solved the problem. I was initially doing the cloud.cfg change on all the HAWQ hosts but ignoring one of the master nodes as it was not supposed to have HAWQ components. However, I had configured for using the YARN scheduler for which that node was master.

This dependency was creating the above mentioned problem.