Hi, We have a 12 nodes cluster running CDH 5.4.9 under CM 5.4.5. Each node has 12 cpus (24 vcores with HT), 128gb of ram and 6 HDDs. Each of them runs a HDFS datanode daemon, a yarn nodemanager and an impala daemon Cluster is pretty simple : no Hbase, no sentry, no kerberos. Since we are on 5.4.9, we were running Impala's ressource management inside of YARN. Because this setup is not supported in CDH 5.5, we wanted to separate Impala from Yarn and reconfigure the static pools accordingly. These are the steps that were taken on May 17th 1-Shutdown Impala 2-Delete the 2 llama role instances 3-Modify the Impala config to remove Yarn as a ressource manager 4-Start the Static Pool config wizard 5-Set the % as : 5% for HDFS, 10% for Impala and 85% for Yarn. (was 5% HDFS and 95% Yarn before) 6-Restart everything. Now, after the restart, everything looks like it's running fine. Hive and Impala query runs fine and no errors whatsoever. This morning (May 18th), we noticed that most of our ETLs (mainly Hive jobs) did not run. Looking up the "Yarn Applications" page in Cloudera Manager, there were 3 running applications and a huge list of pending ones. The 3 running applications had the EXACT same problem : 1-M/R was started with 3000+mapper to run and 1099 reducers 2-Most of the mappers completed successfully 3-The reducer started the copy phase while the rest of the mappers continued their job 4-Then, at one point, the job hangs because, for some reasons, pending mappers are never started. So we get stuck with a job that has 2-3 pending mappers and 100+ running reducers and it stays like that forever because for some reasons, the pending mappers never get submitted. At first, I noticed that we had a failover of the ressource manager during the night but it was unrelated : re-running the query ends up doing the exact same problem! What I've tried : 1-Disabled CGroups altogether --> all M/R jobs now failing 2-Rollback my config by manually setting back Impala inside of Yarn using Llama, re-enabling CGroups and rolling back the following config : Default Value for : Container Memory Maximum,Container Virtual CPU Cores Maximum,Cgroup CPU Shares,Cgroup I/O Weight Old values for : Default Number of Reduce Tasks per Job, Container Memory #2 did the trick but I'm back to square 1 where I can't upgrade to CDH5.5 or newer. Any clues on where to start the investigation? It's a bit of a pain since this behavior cannot be replicated on our test cluster (not enough data), only in Production... Thanks!
... View more
