Very often we run into permissions issue in our dev cluster "Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hbase, access=WRITE, inode="/apps/hbase/data/archive/data/default/TABLE_NAME/98e81b5c08b4f4fcedf61cd934cc7e1a":hdfs:hdfs:drwxr-xr-x"
this blocks any operations on hbase.
Note that the permission of the file in HDFS is hdfs:hdfs:drwxr-xr-x
Data that HBase is accessing should be owned by "hbase" not "hdfs".
Try checking each RegionServer to make sure:
1. If you don't have Kerberos authentication enabled, the RegionServer is running as the hbase user
2. If you have Kerberos authentication enabled, the RS is using the HBase service keytab
3. If you have Kerberos authentication enabled, core-site.xml has the correct auth_to_local configuration mapping the Kerberos principal to "hbase" (and not "hdfs").
It is possible that you have at least one RegionServer which is running as the hdfs user and creating files as hdfs instead of hbase.