Archives of Support Questions (Read Only)

This is an archived board for historical reference. Information and links may no longer be available or relevant
Announcements
This board is archived and read-only for historical reference. To ask a new question, please post a new topic on the appropriate active board.

Accumulo under-replication issue

avatar
Super Collaborator

HDP 2.4 installed using Ambari 2.2.2.0.

This existing thread mentions clearly the under-replication issue that exists with Accumulo.

Accordingly, I executed several times the below command and also restarted the Accumulo service, the under-replicated blocks dropped to 0 momentarily, then again started to increase :

[hdfs@l4327pp root]$ hdfs dfs -setrep -w 3 /
[hdfs@l4327pp root]$ hdfs fsck /
Connecting to namenode via http://l4327pp.sss.com:50070/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /138.106.33.139 for path / at Wed Jun 22 11:56:56 CEST 2016
.......
/apps/accumulo/data/tables/!0/table_info/A000025h.rf:  Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745134_4310. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
.
/apps/accumulo/data/tables/!0/table_info/A000025i.rf:  Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745135_4311. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
.
/apps/accumulo/data/tables/!0/table_info/F000025g.rf:  Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745133_4309. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
.
/apps/accumulo/data/tables/+r/root_tablet/A000025k.rf:  Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745137_4313. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
..........................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
...............................................................
/user/accumulo/.Trash/Current/apps/accumulo/data/tables/+r/root_tablet/delete+A000025k.rf+F000025j.rf:  Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745136_4312. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
.....................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
........................................................................Status: HEALTHY
 Total size:    44223528893 B (Total open files size: 426 B)
 Total dirs:    139
 Total files:   1172
 Total symlinks:                0 (Files currently being written: 2)
 Total blocks (validated):      1490 (avg. block size 29680220 B) (Total open file blocks (not validated): 2)
 Minimally replicated blocks:   1490 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       5 (0.33557048 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0033557
 Corrupt blocks:                0
 Missing replicas:              5 (0.11160714 %)
 Number of data-nodes:          4
 Number of racks:               1
FSCK ended at Wed Jun 22 11:56:56 CEST 2016 in 31 milliseconds
The filesystem under path '/' is HEALTHY

I even tried to add an entry to the custom accumulo-site.xml and restarted the service but the issue persists :

5147-accumulo-custom-site-entry.png

1 ACCEPTED SOLUTION

avatar
Expert Contributor

As described in the answer to this question: https://community.hortonworks.com/questions/35138/target-replicas-is-5-but-found-3-live-replicas.htm...

Start the accumulo shell on a host where the Accumulo client is installed and run the following:

config -t accumulo.root -s table.file.replication=3
config -t accumulo.metadata -s table.file.replication=3

You'll have to change these properties AND run the hdfs setrep command so that both new and existing files have their settings changed.

View solution in original post

2 REPLIES 2

avatar
Expert Contributor

As described in the answer to this question: https://community.hortonworks.com/questions/35138/target-replicas-is-5-but-found-3-live-replicas.htm...

Start the accumulo shell on a host where the Accumulo client is installed and run the following:

config -t accumulo.root -s table.file.replication=3
config -t accumulo.metadata -s table.file.replication=3

You'll have to change these properties AND run the hdfs setrep command so that both new and existing files have their settings changed.

avatar
Super Collaborator

Accumulo(as always)gave a nasty surprise, was unable to log-in the shell using the credentials I provided during the installation, the tracer failed to start. As per this thread, I kept a simple text password and then executed the commands you provided