Created on 06-22-2016 10:06 AM - edited 08-18-2019 05:57 AM
HDP 2.4 installed using Ambari 2.2.2.0.
This existing thread mentions clearly the under-replication issue that exists with Accumulo.
Accordingly, I executed several times the below command and also restarted the Accumulo service, the under-replicated blocks dropped to 0 momentarily, then again started to increase :
[hdfs@l4327pp root]$ hdfs dfs -setrep -w 3 /
[hdfs@l4327pp root]$ hdfs fsck / Connecting to namenode via http://l4327pp.sss.com:50070/fsck?ugi=hdfs&path=%2F FSCK started by hdfs (auth:SIMPLE) from /138.106.33.139 for path / at Wed Jun 22 11:56:56 CEST 2016 ....... /apps/accumulo/data/tables/!0/table_info/A000025h.rf: Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745134_4310. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s). . /apps/accumulo/data/tables/!0/table_info/A000025i.rf: Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745135_4311. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s). . /apps/accumulo/data/tables/!0/table_info/F000025g.rf: Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745133_4309. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s). . /apps/accumulo/data/tables/+r/root_tablet/A000025k.rf: Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745137_4313. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s). .......................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ............................................................... /user/accumulo/.Trash/Current/apps/accumulo/data/tables/+r/root_tablet/delete+A000025k.rf+F000025j.rf: Under replicated BP-736310146-138.106.33.139-1466422465726:blk_1073745136_4312. Target Replicas is 5 but found 4 live replica(s), 0 decommissioned replica(s) and 0 decommissioning replica(s). ..................................... .................................................................................................... .................................................................................................... .................................................................................................... .................................................................................................... ........................................................................Status: HEALTHY Total size: 44223528893 B (Total open files size: 426 B) Total dirs: 139 Total files: 1172 Total symlinks: 0 (Files currently being written: 2) Total blocks (validated): 1490 (avg. block size 29680220 B) (Total open file blocks (not validated): 2) Minimally replicated blocks: 1490 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 5 (0.33557048 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 3.0033557 Corrupt blocks: 0 Missing replicas: 5 (0.11160714 %) Number of data-nodes: 4 Number of racks: 1 FSCK ended at Wed Jun 22 11:56:56 CEST 2016 in 31 milliseconds The filesystem under path '/' is HEALTHY
I even tried to add an entry to the custom accumulo-site.xml and restarted the service but the issue persists :
Created 06-22-2016 06:41 PM
As described in the answer to this question: https://community.hortonworks.com/questions/35138/target-replicas-is-5-but-found-3-live-replicas.htm...
Start the accumulo shell on a host where the Accumulo client is installed and run the following:
config -t accumulo.root -s table.file.replication=3 config -t accumulo.metadata -s table.file.replication=3
You'll have to change these properties AND run the hdfs setrep command so that both new and existing files have their settings changed.
Created 06-22-2016 06:41 PM
As described in the answer to this question: https://community.hortonworks.com/questions/35138/target-replicas-is-5-but-found-3-live-replicas.htm...
Start the accumulo shell on a host where the Accumulo client is installed and run the following:
config -t accumulo.root -s table.file.replication=3 config -t accumulo.metadata -s table.file.replication=3
You'll have to change these properties AND run the hdfs setrep command so that both new and existing files have their settings changed.
Created 06-23-2016 09:25 AM
Accumulo(as always)gave a nasty surprise, was unable to log-in the shell using the credentials I provided during the installation, the tracer failed to start. As per this thread, I kept a simple text password and then executed the commands you provided