cannot start hbase in cluster mode

[#1]

I have three system with hbase installed, one of them a master node, and the rest are slaves. Following are the details:

/etc/hosts file contains: (all files in master node)

127.0.0.1    localhost
127.0.1.1    saichanda-OptiPlex-9020

10.0.3.18       master
10.0.3.68       slave1
10.0.3.70       slave2

hbase-site.xml file contains:

<configuration>
    <property>
        <name>hbase.master</name>
        <value>master:60000</value>
    </property>
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://master:50000/hbase</value>
    </property>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <name>hbase.zookeeper.property.clientPort</name>
        <value>2181</value>
    </property>
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>master</value>
    </property>
</configuration>

/usr/local/hbase/conf/regionservers contains

master
slave1
slave2

After starting zookeeper in my system,

when I run in my system(master) hbase, I get the following:

 sudo ./bin/start-hbase.sh starting master, logging to /usr/local/hbase/bin/../logs/hbase-root-master-saichanda-OptiPlex-9020.out
root@master's password: root@slave1's password: root@slave1's password:

I get all the password requests at once, it doesn't allow me to type password...

Another doubt is, it says, root@master, root@slave1, root@slave2, but actually I don't have my system names to be master, slave1, slave2.. rather my system names are saichanda,dmacs,dmacs with IP addresses 10.0.3.18, 10.0.3.68, 10.0.3.70 respectively. Where should I modify my files to run hbase cluster successfully.

[#2]

Looks like you have an issue with the SSH login to the machines. Based on your configuration in the /etc/hosts file, root@slave1 actually is root@10.0.3.68, given your config in /usr/local/hbase/conf/regionservers, this is actually what you should expect. It will try to connect to the named configured in regionservers, try to resolve the name to an IP address using DNS or /etc/hosts in your case. So when you run on a console ssh slave1 it should get you to the login of 10.0.3.68, I think that's alright here.

Your issue is that the logins are happening in parallel, not giving you a real chance to enter the login? My recommendation to solve it is to provide SSH key authentication, so that the password isn't prompted.

How you can do this is described here: https://www.ssh.com/ssh/keygen/

Basically it is about using ssh-keygen and ssh-copy-id (which you can do manually as well). In your case you should create the key pair on your master system (seemingly for the root user), and copy the public keys to the slave machines. If you don't want to be prompted at all you should not enter a passphrase, but you will have to keep the key protected at all times.

If you really want to enter a password, you can change the start script that the second server login is waiting for the first to be done, but this is working for 3 machines, but will take quite a long time if you have setup with many nodes.

[#3]

I haven't got rid of the problem inspite of creating the key and copying it to the slaves.. let me show you what I have done...

a similar thing has been done to copy the (SAME) key to the other system...too.. is that right..

still I get the same prompt.. asking for passwords without gap... described below...irrespective of the location I get the prompt.. this way...

[#4]

You have created the key for root@saichanda-Optiplex-9020, but you are starting start-hbase.sh as saichanda@saichanda-OptiPlex-9020. I got this wrong from your original post, sorry. Now you have enabled root@saichanda-Optiplex-9020 to login with dmacs@10.0.3.68, but you want to enable saichanda to login as root.

You will have to create the key for saichanda@saichanda-OptiPlex-9020 and copy (authorize) it to the root user of your target machines. Basically skip the sudo su - as the first command. First, if not already there, create a pair of keys as saichanda@saichanda-OptiPlex-9020, so that the public key is in /home/saichanda/.ssh/id_rsa.pub. Then still as saichanda, copy the key to the root user of the installation machines.

<code>ssh-copy-id -i /root/.ssh/id_rsa root@slave1<br>ssh-copy-id -i /root/.ssh/id_rsa root@slave2<br>ssh-copy-id -i /root/.ssh/id_rsa root@master<br>

In case you don't have the root password for the machines and you can only run sudo, you can copy the file ~/.ssh/id_rsa.pub with any other user to the machines in your home dir (slave1, slave2 and master, which is your current machine i guess), and then copy the key into the root dir with (on each machine):

sudo cat ~/id_rsa.pub >> /root/.ssh/authorized_keys<br>

[#5 · Accepted solution]

@saichand akella

Don't use "sudo" in the start command. Also make sure you are able to do a password less ssh from saichanda user to all other machines(including self "ssh localhost" should not prompt for password).

[#6]

$ ssh localhost
sign_and_send_pubkey: signing failed: agent refused operation
saichanda@localhost's password:

This is what I get when I do ssh localhost.. Event after adding id_rsa to authoriztion_keys

[#7]

When I do jps on master node, I get the following. . .

18193 Jps

Does this mean, As of now, only jps is running?

[#8]

I am able to ssh to all the nodes, including master(localhost), but I get the following error...

saichanda@saichanda-OptiPlex-9020:~$ /usr/local/hbase/bin/start-hbase.sh

starting master, logging to /usr/local/hbase/logs/hbase-saichanda-master-saichanda-OptiPlex-9020.out

log4j:ERROR setFile(null,true) call failed.

java.io.FileNotFoundException: /usr/local/hbase/logs/SecurityAuth.audit (Permission denied)

    at java.io.FileOutputStream.open0(Native Method)

    at java.io.FileOutputStream.open(FileOutputStream.java:270)

    at java.io.FileOutputStream.<init>(FileOutputStream.java:213)

    at java.io.FileOutputStream.<init>(FileOutputStream.java:133)

    at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)

    at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)

    at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)

    at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)


The authenticity of host 'slave2 (10.0.3.68)' can't be established.

ECDSA key fingerprint is SHA256:UBvk5FkG3KfSMREsDIKbob1l782fozXQHT6+3J6F2Y4.

Are you sure you want to continue connecting (yes/no)? slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@<br/>@@@@@@@@@@@@@@@@@@@@@@@@@

slave1: @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @

slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

slave1: The ECDSA host key for slave1 has changed,

slave1: and the key for the corresponding IP address 10.0.3.70

slave1: is unchanged. This could either mean that

slave1: DNS SPOOFING is happening or the IP address for the host

slave1: and its host key have changed at the same time.

slave1: Offending key for IP in /home/saichanda/.ssh/known_hosts:18

slave1:   remove with:

slave1:   ssh-keygen -f "/home/saichanda/.ssh/known_hosts" -R 10.0.3.70

slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

slave1: @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @

slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

slave1: IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

slave1: Someone could be eavesdropping on you right now (man-in-the-middle attack)!

slave1: It is also possible that a host key has just been changed.

slave1: The fingerprint for the ECDSA key sent by the remote host is

slave1: SHA256:D9udCvOaKAFGwNbuIpBa+pQErZSDTjLg4m4LJ+s8DzM.

slave1: Please contact your system administrator.

slave1: Add correct host key in /home/saichanda/.ssh/known_hosts to get rid of this message.

slave1: Offending ECDSA key in /home/saichanda/.ssh/known_hosts:22

slave1:   remove with:

slave1:   ssh-keygen -f "/home/saichanda/.ssh/known_hosts" -R slave1

slave1: ECDSA host key for slave1 has changed and you have requested strict checking.

slave1: Host key verification failed.

master: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-saichanda-regionserv
er-saichanda-OptiPlex-9020.out
master: log4j:ERROR setFile(null,true) call failed.

master: java.io.FileNotFoundException: /usr/local/hbase/bin/../logs/SecurityAuth.audit (Permissio
n denied)
master:     at java.io.FileOutputStream.open0(Native Method)

master:     at java.io.FileOutputStream.open(FileOutputStream.java:270)

master:     at java.io.FileOutputStream.<init>(FileOutputStream.java:213)

master:     at java.io.FileOutputStream.<init>(FileOutputStream.java:133)

master:     at org.apache.log4j.FileAppender.setFile(FileAppender.java:294)

master:     at org.apache.log4j.RollingFileAppender.setFile(RollingFileAppender.java:207)

master:     at org.apache.log4j.FileAppender.activateOptions(FileAppender.java:165)

master:     at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)

slave2: Host key verification failed.

[#9]

@saichand akella Since you were trying to start with root user the file permissions would be for root.

Clean up this directory /usr/local/hbase/logs and make sure your user has permission to write. Then start the Hbase.

[#10]

@Sandeep Nemuri

As you said, I cleaned up the directory /usr/local/hbase/logs from master node, What I understand by saying write permission for user is able to ssh to localhost without password.

Right now, I am facing the error as follows:

$./bin/start-hbase.sh 
starting master, logging to /usr/local/hbase/logs/hbase-saichanda-master-saichanda-OptiPlex-9020.out
The authenticity of host 'slave2 (10.0.3.68)' can't be established.
ECDSA key fingerprint is SHA256:UBvk5FkG3KfSMREsDIKbob1l782fozXQHT6+3J6F2Y4.
Are you sure you want to continue connecting (yes/no)? slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
slave1: @       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
slave1: The ECDSA host key for slave1 has changed,
slave1: and the key for the corresponding IP address 10.0.3.70
slave1: is unchanged. This could either mean that
slave1: DNS SPOOFING is happening or the IP address for the host
slave1: and its host key have changed at the same time.
slave1: Offending key for IP in /home/saichanda/.ssh/known_hosts:18
slave1:   remove with:
slave1:   ssh-keygen -f "/home/saichanda/.ssh/known_hosts" -R 10.0.3.70
slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
slave1: @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
slave1: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
slave1: IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
slave1: Someone could be eavesdropping on you right now (man-in-the-middle attack)!
slave1: It is also possible that a host key has just been changed.
slave1: The fingerprint for the ECDSA key sent by the remote host is
slave1: SHA256:D9udCvOaKAFGwNbuIpBa+pQErZSDTjLg4m4LJ+s8DzM.
slave1: Please contact your system administrator.
slave1: Add correct host key in /home/saichanda/.ssh/known_hosts to get rid of this message.
slave1: Offending ECDSA key in /home/saichanda/.ssh/known_hosts:22
slave1:   remove with:
slave1:   ssh-keygen -f "/home/saichanda/.ssh/known_hosts" -R slave1
slave1: ECDSA host key for slave1 has changed and you have requested strict checking.
slave1: Host key verification failed.
master: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-saichanda-regionserver-saichanda-OptiPlex-9020.out
slave2: Host key verification failed.

Why is it saying Host key verfication failed, when I am able to ssh to 10.0.3.68 and 10.0.3.70 without password.

How to resolve this?

I hope this is the last problem I am facing, after which hopefully I can start hbase

[#11]

With SSH each server has an authentication key. When you connect the first time, you are asked if you really trust the server to be the correct one. If you confirm, the key is stored in ~/..ssh/known_hosts. When you connect again, this key is compared to the key the server provides to authenticate himself. If they differ you get the error message you just see. Typically it means that the host has changed, which can be an attack.

I guess you have that issue, as the name/IP association has changed from you OP
OP:

10.0.3.68 slave1
10.0.3.70 slave2

Now as per your error message:10.0.3.70 slave1 ?

Maybe you will have to correct the IP addresses for the names?

If you are sure your server/IP are correct and uncompromised, you can edit the fie /home/saichanda/.ssh/known_hosts with any editor and replace the key, or you simply do as the message proposes:

ssh-keygen -f "/home/saichanda/.ssh/known_hosts" -R slave1
ssh-keygen -f "/home/saichanda/.ssh/known_hosts" -R 10.0.3.70

Then you can again ssh the host slave1 and slave2 and accept the host key,

[#12]

I have done everything possible.. looked into every site possible, Got down my error lines to as small as follows

saichanda@saichanda-OptiPlex-9020:~$ /usr/local/hbase/bin/start-hbase.sh 
starting master, logging to /usr/local/hbase/logs/hbase-saichanda-master-saichanda-OptiPlex-9020.out
The authenticity of host 'slave2 (10.0.3.68)' can't be established.
ECDSA key fingerprint is SHA256:UBvk5FkG3KfSMREsDIKbob1l782fozXQHT6+3J6F2Y4.
Are you sure you want to continue connecting (yes/no)? The authenticity of host 'slave1 (10.0.3.70)' can't be established.
ECDSA key fingerprint is SHA256:D9udCvOaKAFGwNbuIpBa+pQErZSDTjLg4m4LJ+s8DzM.
Are you sure you want to continue connecting (yes/no)? master: starting regionserver, logging to /usr/local/hbase/bin/../logs/hbase-saichanda-regionserver-saichanda-OptiPlex-9020.out
slave2: Host key verification failed.
slave1: Host key verification failed.
saichanda@saichanda-OptiPlex-9020:~$ 

Can you help me conclude this doubt here??plz...

[#13]

I found few interesting facts in the process of learning how to start hbase:

Firstly, Be sure that on all the systems there are users by which you start the hbase, if not, then use `adduser` command in respective systems.

Secondly, Be sure that you (master) have the write permission to create the log file in the /usr/local/../logs directory in slave nodes. Otherwise, it throws permission denied error...

Finally, Be sure that you copied the ssh-copy-id to all the systems for a passwordless access into regionservers.

[#14]

Glad you were able to start hbase.

>>> Firstly, Be sure that on all the systems there are users by which you start the hbase, if not, then use adduser command in respective systems.

This was pre-requisite to have same user in all machines. Then only you can do a passwordless ssh from

user1@source --> user1@destination.

The next two points are covered in either my comment or @Harald Berghoff comments below.

[#15]

@Sandeep NemuriI wish to acknowledge all who have contributed to this post more than I did.. Can you summarize the solutions touching on all the errors that I have committed, may be a brief ANSWER that I can accept as best answer. Truly in a positive sense...

[#16]

@saichand akella That should be ok 🙂 you can accept this answer.