Thanks. I successfully rescued the unrecognized blocks. Addressing the underlying issue of missing but finalized blocks will take time. Hopefully upgrading to a later CDH will work. ... View more

The histogram of {count,disk}  was preliminary. Here is a final tally:   count, drive 144 00 6 01 290 02 155 03 154 04 134 05 167 06 172 08 2 09 144 10 2 11 143 12 7 13 130 15 151 16 280 17 7 19 5 20 2 21 172 22 139 23 4 24 160 25 171 26 162 27 10 28 184 29 171 30 3 31 5 32 7 33 4 34 248 35   ... View more

> Do you get a full strack trace in the Namenode log at the time of the error in the datanode?   No. I showed the entire log msg. that was visible. I checked NN stdout, and did not see a stacktrace.    > Have all these files with missing blocks got replication factor of 1 or have they a replication factor 3?   repl factor is 3, but the problem files did not reach that level due to competing acivity. The DN in question had to be bounced due to a failed disk (it has 36 8TB disks), apparently at an inopportune moment.    I think the excessive RBW files (up to 1 year old) are not the cause since I moved most of them away, restarted DN, and there was no change.    Current problem summary:  finalized blocks/replicas in a DN are reported as missing, and "RemoteException in offerService" WARN appears in DN log.    More info:    I found that of the 4400 block/replicas missing cluster-wide, there are 3500 spread across 36 disks on one DN under /finalized/, unevenly:   count, disk 2 09 2 11 2 21 3 31 4 24 4 34 5 20 5 32 6 01 7 13 7 19 7 33 10 28 15 05 139 23 154 04 159 27 160 25 171 26 280 17   hence, I am unable to place blame on one disk. This DN has 14M blk files, so only a tiny percentage are affected.    It sure seems like an uncaught exception caused the block report from this DN to be incomplete.    ... View more

Another symptom: the DataNode /blockScannerReport on the problem DN always returns this:      Periodic block scanner is not yet initialized. Please check back again after some time.     ... View more

That helped. It works now. We had no reason to expect that the instance and instances clauses were necessary for a CM that already exists and was generated by Cloudera Director. ... View more

We tried it with and without " Deployment" and " Environment" appended, but alway get the same error. We have successfully added more than one cluster to the particular CM.    The tail end of the application.log is below, if that helps.    Alternatively, how can we specify "Cloudera Manager URL"?   [2016-09-15 02:11:34] INFO [main] - c.c.l.m.PrivateKeySshCredentialsValidator: Validating SSH credentials for ec2-user [2016-09-15 02:11:34] ERROR [main] - c.c.l.commands.ValidateCommand: Failed to parse deployment configuration java.lang.IllegalArgumentException: You have to either specify a virtual instance configuration or a Cloudera Manager URL for a new deployment at com.google.common.base.Preconditions.checkArgument(Preconditions.java:93) ~[guava-15.0.jar!/:na] at com.cloudera.launchpad.model.deployment.DeploymentTemplate.<init>(DeploymentTemplate.java:154) ~[launchpad-model-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.model.deployment.DeploymentTemplateBuilder.build(DeploymentTemplateBuilder.java:474) ~[launchpad-model-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.templates.ConfigToDeploymentTemplate.apply(ConfigToDeploymentTemplate.java:183) ~[launchpad-templates-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.commands.ValidateCommand.run(ValidateCommand.java:136) ~[launchpad-cli-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.commands.BootstrapRemoteCommand.run(BootstrapRemoteCommand.java:74) [launchpad-cli-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.commands.RemoteCommand.run(RemoteCommand.java:198) [launchpad-cli-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.Application.run(Application.java:140) [launchpad-cli-2.1.0.jar!/:2.1.0] at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:806) [spring-boot-1.3.2.RELEASE.jar!/:1.3.2.RELEASE] at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:790) [spring-boot-1.3.2.RELEASE.jar!/:1.3.2.RELEASE] at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:777) [spring-boot-1.3.2.RELEASE.jar!/:1.3.2.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) [spring-boot-1.3.2.RELEASE.jar!/:1.3.2.RELEASE] at com.cloudera.launchpad.Application.start(Application.java:97) [launchpad-cli-2.1.0.jar!/:2.1.0] at com.cloudera.launchpad.Application.main(Application.java:47) [launchpad-cli-2.1.0.jar!/:2.1.0] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_72] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_72] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_72] at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_72] at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:54) [launchpad-cli-2.1.0.jar!/:2.1.0] at java.lang.Thread.run(Thread.java:745) [na:1.7.0_72] [2016-09-15 02:11:34] INFO [Thread-9] - o.s.c.a.AnnotationConfionConfigApplicationContext@4f04b01a: startup date [Thu Sep 15 02:11:15 UTC 2016]; root of context hierarchy ... View more