Problems running a write query on an accumulo cluster

[#1]

Dear all,

I am trying to run a basic write query on a multi-node(3 nodes) Accumulo cluster. When I run my code from a node inside the cluster it works fine, but when I use the same code on a machine which is not a node in the cluster the code fails to do anything and exits with this message in the console:

"WARN: Failed to find an available server from a list of servers."

The machine fails to connect to the zookeeper servers.

Has anybody encountered this problem?

Any help would be useful.

Thanks and Regards,
Vaibhav

[#2 · Accepted solution]

As Mike previously mentioned, those configuration files don't exist when the cluster is handled by CM.

It sounds like the underlying problem might be incorrect host name resolution. Accumulo and Hadoop require forward and reverse DNS to be set up correctly. You should not have IP addresses in your configuration files.

If the problem is incorrect host names, you can check a few things

1) What does CM think the name of the hosts are?

If you go to http://cm.example.com:7180/cmf/hardware/hosts (where "cm.example.com" is the name of your CM host), what is listed in the "name" column? It should be all full qualified domain names.

2) What does the host think its name is?

Log into each of the cluster machines and run the "hostname" command. It should return a fuly qualified domain name and this name should match the one found in the "name" column above.

3) What do Accumulo processes think the host names are?

You can see this by looking inside of ZooKeeper. Because ZooKeeper is used to maintain critical information for Accumulo, you should be very careful while dealing with it directly. It's also important to note that this information is deep in the internals of Accumulo; you must not presume it will be the same across versions. Below I'll show an example from a cluster running Accumulo 1.6.0-cdh5.1.0.

Connect to zookeeper and see what shows up for tablet servers in the /accumulo/%UUID%/tservers node

$ zookeeper-client -server zoo1.example.com,zoo2.example.com,zoo3.example.com
Connecting to zoo1.example.com,zoo2.example.com,zoo3.example.com
... SNIP ...
2014-11-27 10:50:11,499 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=zoo1.example.com,zoo2.example.com,zoo3.example.com sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@8d80be3
Welcome to ZooKeeper!
2014-11-27 10:50:11,535 [myid:] - INFO  [main-SendThread(zoo2.example.com:2181):ClientCnxn$SendThread@975] - Opening socket connection to server zoo2.example.com/10.17.72.3:2181. Will not attempt to authenticate using SASL (java.lang.SecurityException: Unable to locate a login configuration)
JLine support is enabled
2014-11-27 10:50:11,546 [myid:] - INFO  [main-SendThread(zoo2.example.com:2181):ClientCnxn$SendThread@852] - Socket connection established to zoo2.example.com/10.17.72.3:2181, initiating session
2014-11-27 10:50:11,560 [myid:] - INFO  [main-SendThread(zoo2.example.com:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server zoo2.example.com/10.17.72.3:2181, sessionid = 0x349ace5c95e63c4, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: zoo1.example.com,zoo2.example.com,zoo3.example.com(CONNECTED) 0] ls /accumulo/e8f3afdf-a59c-4ebd-ae0c-15d47b9dd5e1/

users               problems            monitor             root_tablet         hdfs_reservations   gc
table_locks         namespaces          recovery            fate                tservers            tables
next_file           tracers             config              masters             bulk_failed_copyq   dead
[zk: zoo1.example.com,zoo2.example.com,zoo3.example.com(CONNECTED) 0] ls /accumulo/e8f3afdf-a59c-4ebd-ae0c-15d47b9dd5e1/tservers 
[tserver1.example.com:10011,tserver2.example.com:10011,tserver3.example.com:10011,tserver4.example.com:10011,tserver5.example.com:10011]
[zk: zoo1.example.com,zoo2.example.com,zoo3.example.com(CONNECTED) 1] 

The UUID in the top level /accumulo node is the internal id used to track your Accumulo instance. If there are multiple of these, you can find the one for your current cluster by listing all instance information (presuming you have an Accumulo gateway on the node). This utility is also an Accumulo internal, so neither its name, usage, nor output format can be counted on across versions.

$>  accumulo org.apache.accumulo.server.util.ListInstances
INFO : Using ZooKeepers zoo1.example.com,zoo2.example.com,zoo3.example.com

 Instance Name       | Instance ID                          | Master                        
---------------------+--------------------------------------+-------------------------------
          "accumulo" | e8f3afdf-a59c-4ebd-ae0c-15d47b9dd5e1 |master2.example.com:10010
         "dedicated" | 496b74ab-316c-41bc-badb-4b908039f725 |                              
         "dedicatum" | e49b451b-4607-4a0e-9dda-b49dc938080e |                        

4) Is HDFS confused?

Can you use hdfs commands from inside / outside of the cluster? E.g. can you list the root directory? Can the Accumulo user list their home directory or the /accumulo directory?

[#3 · Accepted solution]

Hi,

Thanks everyone for your valuable inputs.

I finally was able to troubleshoot the problem with my cluster. As it turns out the remote machine needed to add the ip-address of the nodes in the cluster in their /etc/hosts/ file.

After doing that I was able to get the desired results.

Thanks everyone!!!

Vaibhav