Accumulo under-replication issue

kaliyugantagoni — Sun, 18 Aug 2019 12:57:36 GMT

HDP 2.4 installed using Ambari 2.2.2.0.

This existing thread mentions clearly the under-replication issue that exists with Accumulo.

Accordingly, I executed several times the below command and also restarted the Accumulo service, the under-replicated blocks dropped to 0 momentarily, then again started to increase :

[hdfs@l4327pp root]$ hdfs dfs -setrep -w 3 /

[hdfs@l4327pp root]$ hdfs fsck /
Connecting to namenode via http://l4327pp.sss.com:50070/fsck?ugi=hdfs&path=%2F
FSCK started by hdfs (auth:SIMPLE) from /138.106.33.139 for path / at Wed Jun 22 11:56:56 CEST 2016
.......
/apps/accumulo/data/tables/!0/table_info/A000025h.rf:  Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745134_4310. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
.
/apps/accumulo/data/tables/!0/table_info/A000025i.rf:  Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745135_4311. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
.
/apps/accumulo/data/tables/!0/table_info/F000025g.rf:  Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745133_4309. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
.
/apps/accumulo/data/tables/+r/root_tablet/A000025k.rf:  Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745137_4313. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
..........................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
...............................................................
/user/accumulo/.Trash/Current/apps/accumulo/data/tables/+r/root_tablet/delete+A000025k.rf+F000025j.rf:  Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745136_4312. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s).
.....................................
....................................................................................................
....................................................................................................
....................................................................................................
....................................................................................................
........................................................................Status: HEALTHY
 Total size:    44223528893 B (Total open files size: 426 B)
 Total dirs:    139
 Total files:   1172
 Total symlinks:                0 (Files currently being written: 2)
 Total blocks (validated):      1490 (avg. block size 29680220 B) (Total open file blocks (not validated): 2)
 Minimally replicated blocks:   1490 (100.0 %)
 Over-replicated blocks:        0 (0.0 %)
 Under-replicated blocks:       5 (0.33557048 %)
 Mis-replicated blocks:         0 (0.0 %)
 Default replication factor:    3
 Average block replication:     3.0033557
 Corrupt blocks:                0
 Missing replicas:              5 (0.11160714 %)
 Number of data-nodes:          4
 Number of racks:               1
FSCK ended at Wed Jun 22 11:56:56 CEST 2016 in 31 milliseconds
The filesystem under path '/' is HEALTHY

I even tried to add an entry to the custom accumulo-site.xml and restarted the service but the issue persists :

Re: Accumulo under-replication issue

billie — Thu, 23 Jun 2016 01:41:53 GMT

As described in the answer to this question: https://community.hortonworks.com/questions/35138/target-replicas-is-5-but-found-3-live-replicas.html

Start the accumulo shell on a host where the Accumulo client is installed and run the following:

config -t accumulo.root -s table.file.replication=3
config -t accumulo.metadata -s table.file.replication=3

You'll have to change these properties AND run the hdfs setrep command so that both new and existing files have their settings changed.

Re: Accumulo under-replication issue

kaliyugantagoni — Thu, 23 Jun 2016 16:25:44 GMT

Accumulo(as always)gave a nasty surprise, was unable to log-in the shell using the credentials I provided during the installation, the tracer failed to start. As per this thread, I kept a simple text password and then executed the commands you provided

question Accumulo under-replication issue in Archives of Support Questions (Read Only)

Accumulo under-replication issue

Re: Accumulo under-replication issue

Re: Accumulo under-replication issue