We have a CDH 5.13.0 cluster that had to be moved from Azure VMs to Google Cloud. Image of each VM was captured in Azure, and then VMs recreated in GCP with all nodes now, obviously, getting new names. To resolve nodes name change an entry for each host was added to hosts file (to map from old name to new node names). Cluster seems to be running, HDFS is up, but I'm running into multiple problems like HDFS nodes having high Block Count, Hive queries not working (I don't see any jobs starting when I submit a query, and then I'd get a random "The operation doesn't have handle attached." or a timeout). First quesiton, is this scenario of moving to new hosts, and just remapping old nost names to new via hosts file entries even supported? Second question, how to troubleshoot the cluster now? Rebalancing doesn't seem to resolve the Block Count issue. Cluster is running, but I feel something is broken -- as I can't run any workflows. Where do I start?
... View more
The error quotes a missing function that has been present in Oozie since CDH 5.5.0. It therefore appears that somehow your environment is keeping or passing around an older jar of 'oozie-sharelib-oozie' artifact that is without this added function. If its your sharelib that's carrying a bad file, you can inspect it via: # hadoop fs -ls -R /user/oozie/ | grep sharelib-oozie The above should return only a single jar file size and the version of the filename should match what you are running. If you get 3 or more files in the output, consider redeploying your ShareLib via https://www.cloudera.com/documentation/enterprise/latest/topics/admin_oozie_sharelib.html#concept_i2f_r5t_2r If you just get one version of the jar instead, then perhaps some application jar of your project(s) is assembling a fat jar that includes Oozie Sharelib dependencies in it, albeit from a non-CDH version, or a very old CDH version (< 5.5.0). You can inspect suspect jars by running: # jar tf filename.jar | grep LauncherMain Repack all such Oozie-including jars to exclude Oozie dependencies in them, as the system classpath will already provide the dependencies and of the right version.
... View more