We have a CDH 5.13.0 cluster that had to be moved from Azure VMs to Google Cloud. Image of each VM was captured in Azure, and then VMs recreated in GCP with all nodes now, obviously, getting new names. To resolve nodes name change an entry for each host was added to hosts file (to map from old name to new node names). Cluster seems to be running, HDFS is up, but I'm running into multiple problems like HDFS nodes having high Block Count, Hive queries not working (I don't see any jobs starting when I submit a query, and then I'd get a random "The operation doesn't have handle attached." or a timeout). First quesiton, is this scenario of moving to new hosts, and just remapping old nost names to new via hosts file entries even supported? Second question, how to troubleshoot the cluster now? Rebalancing doesn't seem to resolve the Block Count issue. Cluster is running, but I feel something is broken -- as I can't run any workflows. Where do I start?
... View more
Have a 2 node cluster, things seem to work, HOWEVER, workflows occasionally (like 70% of the time) fail with the following error: Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], main() threw exception, org.apache.oozie.action.hadoop.Hive2Main.setYarnTag(Lorg/apache/hadoop/conf/Configuration;)V
java.lang.NoSuchMethodError: org.apache.oozie.action.hadoop.Hive2Main.setYarnTag(Lorg/apache/hadoop/conf/Configuration;)V !!! I would understand if she was failing consistently, which would mean misconfiguration. But sometimes it passes, and workflows complete successfully. However, most of the time she fails with that same error. Please advise how to troubleshoot or where to look for more hints/logs that would point out the problem! Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.Hive2Main], main() threw exception, org.apache.oozie.action.hadoop.Hive2Main.setYarnTag(Lorg/apache/hadoop/conf/Configuration;)V
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.security.AccessController.doPrivileged(Native Method)
Oozie Launcher failed, finishing Hadoop job gracefully
... View more