Hello community!    I recently added 4 more DNs to my Hadoop cluster, now there are 46 DNs up and running. I'm running the balanncer since 5 days and today a message appears in the top of the NameNode web console (my_name_node_url:50070) about "There are 1169 missing blocks. The following files may be corrupted:" and a list of some of this corrupted blocks. After I saw that message I decided to run the command "hdfs dfsadmin -report" and the result was:   Configured Capacity: 1706034579673088 (1.52 PB) Present Capacity: 1683943231506526 (1.50 PB) DFS Remaining: 559797934331658 (509.13 TB) DFS Used: 1124145297174868 (1022.40 TB) DFS Used%: 66.76% Under replicated blocks: 1169 Blocks with corrupt replicas: 0 Missing blocks: 1169 Missing blocks (with replication factor 1): 21062   For storage capacity reasons a group of developers decided to avoid my advice and set the replication factor in 1 for some files. What does "Missing blocks: 1169" means? Is "Missing blocks  (with replication factor 1)" message telling that those 21062 blocks from files with replication factor 1 cannot be recover?   I'll be very grateful if anyone can clarify this concept.   Thanks!   Guido. ... View more

Hello!   I'm trying to add a new DataNode in a running cluster. The cluster is in HA for HDFS (NNs) and for Yarn (RMs) and is secured by kerberos integration. When I performed the necessary steps to add a new DN and started the hadoop-hdfs-datanode service the new node didn't shows up in the list of DNs (I perfomed a refresh on the NNs). In /var/log/hadoop-hdfs/hadoop-hdfs-datanode-mynode.log there is nothing logged. The output for command "hdfs datanode" is:   2017-06-26 15:42:58,544 INFO security.UserGroupInformation: Login successful for user hdfs/my.datanode.fqdn@MY.REALM.FQDN using keytab file /path/to/hdfs.keytab 2017-06-26 15:42:59,292 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 2017-06-26 15:42:59,333 INFO impl.MetricsSinkAdapter: Sink collectd started 2017-06-26 15:42:59,385 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 2017-06-26 15:42:59,385 INFO impl.MetricsSystemImpl: DataNode metrics system started 2017-06-26 15:42:59,404 INFO datanode.DataNode: File descriptor passing is enabled. 2017-06-26 15:42:59,407 INFO datanode.DataNode: Configured hostname is my.datanode.fqdn 2017-06-26 15:42:59,415 FATAL datanode.DataNode: Exception in secureMain java.lang.RuntimeException: Cannot start secure DataNode without configuring either privileged resources or SASL RPC data transfer protection and SSL for HTTP. Using privileged resources in combination with SASL RPC data transfer protection is not supported. at org.apache.hadoop.hdfs.server.datanode.DataNode.checkSecureConfig(DataNode.java:1205) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:1106) at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:451) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2406) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2293) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2340) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2517) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2541) 2017-06-26 15:42:59,436 INFO util.ExitUtil: Exiting with status 1 2017-06-26 15:42:59,475 INFO datanode.DataNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down DataNode at my.datanode.fqdn/my-datanode-ip-address ************************************************************/   Thanks for your help!   Guido. ... View more