cannot start hbase in cluster mode

[#1]

I have three system with hbase installed, one of them a master node, and the rest are slaves. Following are the details:

/etc/hosts file contains: (all files in master node)

127.0.0.1    localhost
127.0.1.1    saichanda-OptiPlex-9020

10.0.3.18       master
10.0.3.68       slave1
10.0.3.70       slave2

hbase-site.xml file contains:

<configuration>
    <property>
        <name>hbase.master</name>
        <value>master:60000</value>
    </property>
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://master:50000/hbase</value>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
    </property>
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>master</value>
    </property>
</configuration>

/usr/local/hbase/conf/regionservers contains

master
slave1
slave2

After starting zookeeper in my system,

when I run in my system(master) hbase, I get the following:

 sudo ./bin/start-hbase.sh starting master, logging to /usr/local/hbase/bin/../logs/hbase-root-master-saichanda-OptiPlex-9020.out
root@master's password: root@slave1's password: root@slave1's password:

I get all the password requests at once, it doesn't allow me to type password...

Another doubt is, it says, root@master, root@slave1, root@slave2, but actually I don't have my system names to be master, slave1, slave2.. rather my system names are saichanda,dmacs,dmacs with IP addresses 10.0.3.18, 10.0.3.68, 10.0.3.70 respectively. Where should I modify my files to run hbase cluster successfully.

[#2]

Looks like you have an issue with the SSH login to the machines. Based on your configuration in the /etc/hosts file, root@slave1 actually is root@10.0.3.68, given your config in /usr/local/hbase/conf/regionservers, this is actually what you should expect. It will try to connect to the named configured in regionservers, try to resolve the name to an IP address using DNS or /etc/hosts in your case. So when you run on a console ssh slave1 it should get you to the login of 10.0.3.68, I think that's alright here.

Your issue is that the logins are happening in parallel, not giving you a real chance to enter the login? My recommendation to solve it is to provide SSH key authentication, so that the password isn't prompted.

How you can do this is described here: https://www.ssh.com/ssh/keygen/

Basically it is about using ssh-keygen and ssh-copy-id (which you can do manually as well). In your case you should create the key pair on your master system (seemingly for the root user), and copy the public keys to the slave machines. If you don't want to be prompted at all you should not enter a passphrase, but you will have to keep the key protected at all times.

If you really want to enter a password, you can change the start script that the second server login is waiting for the first to be done, but this is working for 3 machines, but will take quite a long time if you have setup with many nodes.

[#3]

I haven't got rid of the problem inspite of creating the key and copying it to the slaves.. let me show you what I have done...

a similar thing has been done to copy the (SAME) key to the other system...too.. is that right..

still I get the same prompt.. asking for passwords without gap... described below...irrespective of the location I get the prompt.. this way...

[#4]

You have created the key for root@saichanda-Optiplex-9020, but you are starting start-hbase.sh as saichanda@saichanda-OptiPlex-9020. I got this wrong from your original post, sorry. Now you have enabled root@saichanda-Optiplex-9020 to login with dmacs@10.0.3.68, but you want to enable saichanda to login as root.

You will have to create the key for saichanda@saichanda-OptiPlex-9020 and copy (authorize) it to the root user of your target machines. Basically skip the sudo su - as the first command. First, if not already there, create a pair of keys as saichanda@saichanda-OptiPlex-9020, so that the public key is in /home/saichanda/.ssh/id_rsa.pub. Then still as saichanda, copy the key to the root user of the installation machines.

<code>ssh-copy-id -i /root/.ssh/id_rsa root@slave1<br>ssh-copy-id -i /root/.ssh/id_rsa root@slave2<br>ssh-copy-id -i /root/.ssh/id_rsa root@master<br>

In case you don't have the root password for the machines and you can only run sudo, you can copy the file ~/.ssh/id_rsa.pub with any other user to the machines in your home dir (slave1, slave2 and master, which is your current machine i guess), and then copy the key into the root dir with (on each machine):

sudo cat ~/id_rsa.pub >> /root/.ssh/authorized_keys<br>

[#5 · Accepted solution]

@saichand akella

Don't use "sudo" in the start command. Also make sure you are able to do a password less ssh from saichanda user to all other machines(including self "ssh localhost" should not prompt for password).

[#6]

$ ssh localhost
sign_and_send_pubkey: signing failed: agent refused operation
saichanda@localhost's password:

This is what I get when I do ssh localhost.. Event after adding id_rsa to authoriztion_keys

[#7]

When I do jps on master node, I get the following. . .

18193 Jps

Does this mean, As of now, only jps is running?

[#8]

I am able to ssh to all the nodes, including master(localhost), but I get the following error...

saichanda@saichanda-OptiPlex-9020:~$ /usr/local/hbase/bin/start-hbase.sh

starting master, logging to /usr/local/hbase/logs/hbase-saichanda-master-saichanda-OptiPlex-9020.out

log4j:ERROR setFile(null,true) call failed.

java.io.FileNotFoundException: /usr/local/hbase/logs/SecurityAuth.audit (Permission denied)

    at java.io.FileOutputStream.open0(Native Method)

    at java.io.FileOutputStream.open(FileOutputStream.java:270)

    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)

    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)

    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)

    at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)

    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)

    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)


The authenticity of host 'slave2 (10.0.3.68)' can't be established.

ECDSA key fingerprint is SHA256:UBvk5FkG3KfSMREsDIKbob1l782fozXQHT6+3J6F2Y4.

Are you sure you want to continue connecting (yes/no)? slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@<br/>@@@@@@@@@@@@@@@@@@@@@@@@@

slave1: @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @

slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

slave1: The ECDSA host key for slave1 has changed,

slave1: and the key for the corresponding IP address 10.0.3.70

slave1: is unchanged. This could either mean that

slave1: DNS SPOOFING is happening or the IP address for the host

slave1: and its host key have changed at the same time.

slave1: Offending key for IP in /home/saichanda/.ssh/known_hosts:18

slave1:   remove with:

slave1:   ssh-keygen -f "/home/saichanda/.ssh/known_hosts" -R 10.0.3.70

slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

slave1: @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @

slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

slave1: IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

slave1: Someone could be eavesdropping on you right now (man-in-the-middle attack)!

slave1: It is also possible that a host key has just been changed.

slave1: The fingerprint for the ECDSA key sent by the remote host is

slave1: SHA256:D9udCvOaKAFGwNbuIpBa+pQErZSDTjLg4m4LJ+s8DzM.

slave1: Please contact your system administrator.

slave1: Add correct host key in /home/saichanda/.ssh/known_hosts to get rid of this message.

slave1: Offending ECDSA key in /home/saichanda/.ssh/known_hosts:22

slave1:   remove with:

slave1:   ssh-keygen -f "/home/saichanda/.ssh/known_hosts" -R slave1

slave1: ECDSA host key for slave1 has changed and you have requested strict checking.

slave1: Host key verification failed.

master: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-saichanda-regionserv
er-saichanda-OptiPlex-9020.out
master: log4j:ERROR setFile(null,true) call failed.

master: java.io.FileNotFoundException: /usr/local/hbase/bin/../logs/SecurityAuth.audit (Permissio
n denied)
master:     at java.io.FileOutputStream.open0(Native Method)

master:     at java.io.FileOutputStream.open(FileOutputStream.java:270)

master:     at java.io.FileOutputStream.<init>(FileOutputStream.java:213)

master:     at java.io.FileOutputStream.<init>(FileOutputStream.java:133)

master:     at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)

master:     at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)

master:     at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)

master:     at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)

slave2: Host key verification failed.

[#9]

@saichand akella Since you were trying to start with root user the file permissions would be for root.

Clean up this directory /usr/local/hbase/logs and make sure your user has permission to write. Then start the Hbase.

[#10]

@Sandeep Nemuri

As you said, I cleaned up the directory /usr/local/hbase/logs from master node, What I understand by saying write permission for user is able to ssh to localhost without password.

Right now, I am facing the error as follows:

$./bin/start-hbase.sh 
starting master, logging to /usr/local/hbase/logs/hbase-saichanda-master-saichanda-OptiPlex-9020.out
The authenticity of host 'slave2 (10.0.3.68)' can't be established.
ECDSA key fingerprint is SHA256:UBvk5FkG3KfSMREsDIKbob1l782fozXQHT6+3J6F2Y4.
Are you sure you want to continue connecting (yes/no)? slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
slave1: @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
slave1: The ECDSA host key for slave1 has changed,
slave1: and the key for the corresponding IP address 10.0.3.70
slave1: is unchanged. This could either mean that
slave1: DNS SPOOFING is happening or the IP address for the host
slave1: and its host key have changed at the same time.
slave1: Offending key for IP in /home/saichanda/.ssh/known_hosts:18
slave1:   remove with:
slave1:   ssh-keygen -f "/home/saichanda/.ssh/known_hosts" -R 10.0.3.70
slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
slave1: @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
slave1: IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
slave1: Someone could be eavesdropping on you right now (man-in-the-middle attack)!
slave1: It is also possible that a host key has just been changed.
slave1: The fingerprint for the ECDSA key sent by the remote host is
slave1: SHA256:D9udCvOaKAFGwNbuIpBa+pQErZSDTjLg4m4LJ+s8DzM.
slave1: Please contact your system administrator.
slave1: Add correct host key in /home/saichanda/.ssh/known_hosts to get rid of this message.
slave1: Offending ECDSA key in /home/saichanda/.ssh/known_hosts:22
slave1:   remove with:
slave1:   ssh-keygen -f "/home/saichanda/.ssh/known_hosts" -R slave1
slave1: ECDSA host key for slave1 has changed and you have requested strict checking.
slave1: Host key verification failed.
master: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-saichanda-regionserver-saichanda-OptiPlex-9020.out
slave2: Host key verification failed.

Why is it saying Host key verfication failed, when I am able to ssh to 10.0.3.68 and 10.0.3.70 without password.

How to resolve this?

I hope this is the last problem I am facing, after which hopefully I can start hbase