Target Replicas is 5 but found 3 live replica(s)

[#1]

Hello,

I have 1 NameNode and 3 DataNodes using the default in >dfs.replication< (3).

"hdfs fsck /" shows this example output:

........ /apps/accumulo/data/tables/!0/table_info/A0000ncg.rf: Under replicated BP-1501447354-10.79.210.78-1461068133478:blk_1073774243_33440. Target Replicas is 5 but found 3 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).

.

.............................................................................................

/user/accumulo/.Trash/Current/apps/accumulo/data/tables/!0/table_info/A0000ncf.rf: Under replicated BP-1501447354-10.79.210.78-1461068133478:blk_1073774242_33439. Target Replicas is 5 but found 3 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s). ..........

After "hdfs dfs -setrep -w 3 /" the messages are gone but after a while shown again.

How and where can I define that accumulo will use 3 replicas or is it a config issue of HDFS?

🙂 Klaus

[#2]

@Klaus Lucas

I see same issue here - https://community.hortonworks.com/questions/6259/target-replicas-is-10-but-found-3-replicas.html

Check if you get any info from here.

[#3]

Hello Sagar!

I reviewed this already. This blog concerns to mapred. As I wrote the setrep command reduce the replicas but not any new incomming ones.

🙂 Klaus

[#4 · Accepted solution]

Accumulo sets the replication for its metadata tables to 5 by default; the hdfs setrep command will change the existing files, but Accumulo will still be using 5 for new files it creates. To change this setting, start the accumulo shell and run the following:

config -t accumulo.root -s table.file.replication=3
config -t accumulo.metadata -s table.file.replication=3

(Note that older versions of Accumulo have just one metadata table named !METADATA instead of the two tables listed above.) You'll have to change these properties AND run the hdfs setrep command so that both new and existing files have their settings changed. You could set the table.file.replication to 3 directly as in the commands above, or you could set it to 0 if you want Accumulo to use the HDFS default replication value. To see what value Accumulo has for the table.file.replication property and verify that you have set it properly, you can run:

config -t accumulo.root -f table.file.replication
config -t accumulo.metadata -f table.file.replication

[#5]

Hello Billie

sounds good but Ambari has installed Accumulo and has not set any environment variables needed by the Accumolo shell script (/usr/hdp/2.4.0.0-169/accumulo/bin/accumulo). Could I set the parameters with Ambari e.g. in "custom accumulo-site"?

[#6]

An Accumulo client will be set up on nodes you selected during the "Assign Slaves and Clients" step of the Ambari Install Wizard. You should be able to find out where clients are installed by going to the Ambari Dashboard, clicking on Accumulo in the service list on the left, then clicking on Accumulo Client.

[#7]

Which clients are affected is not the case. As you described this was documented in Ambari very well. The problem was starting the Accumulo shell in /usr/hdp/2.4.0.0-169/accumulo/bin on the namenode (hosting Master, Monitor, Trace and CG):

root@NameNode:/usr/hdp/2.4.0.0-169/accumulo/bin# accumulo JAVA_HOME is not set or is not a directory. Please make sure it's set globally or in conf/accumulo-env.sh

Starting it on the datanode hosting the TServer, I was able to start the shell (both nodes have no JAVA_HOME entry in the environment) and to set the values to 0 meaning use the replica value from hdfs.

root@hdp-accumulo-instance> config -t accumulo.metadata -f table.file.replication SCOPE | NAME | VALUE default | table.file.replication .. | 0 root@hdp-accumulo-instance> config -t accumulo.metadata -f table.file.replication SCOPE | NAME | VALUE default | table.file.replication .. | 0

Finishing with: hdfs dfs -setrep -w 3 / and the problem was solved.

Many thanks for you Support.

🙂 Klaus

[#8]

Client installations are not the same as server installations. The Accumulo components (Master, TServer etc.) do not need a client installation of Accumulo, so clients are not installed except on the nodes you request in the wizard. The server components use a separate ACCUMULO_CONF_DIR with different permissions, since the server configuration files contain sensitive information. To run the Accumulo shell from a server installation, you would run ACCUMULO_CONF_DIR=/etc/accumulo/conf/server accumulo shell. But for normal client operations you should not do this; you should run Accumulo shell from a client installation.

In any case, I'm glad you were able to fix the problem!

[#9]

Can this be set in "Accumulo Service Advanced Configuration Snippet (Safety Valve) for accumulo-site.xml"