Thank you for adding the version and source detail. Could you please share the full log snippet? The block report size is just a hint to a past problem that used large IPC sizes - on 5.7.x you should be seeing it capped to 1 million max blocks per IPC which wouldn't come close to this limit, so your issue could very well be different and over some other IPC instead. The full error would usually tell you what the call was or who the sender was. ... View more

Could you please add some context here? - What CDH version are you facing this on? - Which service or client role log do you see this message in, and do you have the full actual log to share? In very old CDH5 HDFS releases prior to certain optimisations of large messages (such as block reports) this was a problem you could hit as a function of growing # of blocks in the DNs, but unless we know your version and exact context/component of the error its too vague to help you out. ... View more

Could you please add some context here? - What CDH version are you facing this on? - Which service or client role log do you see this message in, and do you have the full actual log to share? In very old CDH5 HDFS releases prior to certain optimisations of large messages (such as block reports) this was a problem you could hit as a function of growing # of blocks in the DNs, but unless we know your version and exact context/component of the error its too vague to help you out. ... View more

Two DataNodes cannot carry 3 replicas as each replica must reside on a unique DataNode host. Either add a third DataNode, or change the replication factor to 2 to match your DataNode count. ... View more

Your cluster is running a custom authorization plugin inside the NameNode, which is likely controlling this directory specifically. You'll need to contact the authors of the "com.company.department.sf.hdfs.authz.provider.SfAuthzProvider" module to gain more information on why this is done and how to change the permissions. Sentry HDFS ACLs work in similar fashion (a Sentry HDFS Authz plugin is inserted via the same config you noticed above) and begins to ignore permissions being applied on the controlled paths like I'd described before, but in your case it seems like something locally engineered and configured. I'd recommend contacting the developers of your plugin for more information, instead of removing it from your HDFS Configuration safety valves (which would resolve the issue, but its probably there for a reason). ... View more

While this would work, note that its recommended not to rely on "hadoop-core" or "hadoop-common", but use the meta/blanket-wrapper "hadoop-client" dependency instead. This dependency will cover everything typically required by hadoop-side dependency and would cause lesser missing library surprises in future. See also https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_hadoop_api_dependencies.html ... View more

Could you please run and pass the output of the following commands, all run from the same shell session? hadoop fs -ls -d / hadoop fs -ls -d /app hadoop fs -ls -d /app/drop hadoop fs -getfacl /app/drop Additionally, on the NameNode host, could you post the output by running the below command as-is? grep -F authorization.provider -A1 $(ls -rtd /var/run/cloudera-scm-agent/process/*-NAMENODE | tail -1)/hdfs-site.xml ... View more

Does your program open a lot of files for write, in parallel, on HDFS? The DN's "available space" is a more of a guarantee number, i.e. it discounts entire block sizes for any open for write block that's undergoing writes presently. This would reflect in your charts for DN's available space as well, you should see it grow and fall in relation with # of files being opened in parallel. On the policy class note, the AvailableSpaceVolumeChoosingPolicy uses RoundRobinVolumeChoosingPolicy internally once it has decided on a subset of disks to choose from (against its configured thresholds). The policy doesn't appear to be the cause here, however. What version of CDH5 is this cluster running? ... View more