@Saumil Mayani NM manager stores apps/containers info inside /var/log/hadoop-yarn/nodemanager/recovery-state/yarn-nm-state for recovery but I'm not aware of any tool which can read these files. Either you can parse RM and NM logs to find the rough idea of containers count. Also I would recommend you to increase the NM heapsize from 1G to 3G and restart the NM service. ... View more

@Saumil Mayani I don't know if we have any calculation to find out the exact heapsize for NM but typically NM doesn't require huge memory, I have seen customers using b/w 2-4GB heapsize is more than sufficient for node having 256g+ physical memory. What was the heapsize when you saw "GC Overhead limit exceeded" and how many jobs were running to get recovered? ... View more

@Farrukh Mahmood This came up as an issue within HDP version, as a workaround please put extra "--" with a space as prefix along with "--log-dir" or "--max-errors". Example: sqoop import --direct --connect jdbc:netezza://nzserverip:5480/db1 --username user1 --password pass1 --table tab1 --target-dir /test/tab1 --delete-target-dir --m 1 -- --log-dir /logdir/ I will ask Doc team to update the HDP netezza document. ... View more

@Neeraj Sabharwal possibly a yarn client version difference issue. ... View more

@nejm hadjmbarek Either your cluster doesn't have resources to run the AM container of that oozie job or nodemager is not available or you might have some resource queue settings which restricting the resources. ... View more

Yeah..so /usr/hdp/current/hive-server2/bin/hiveserver2 was the culprit :). Do you know how that line got removed? ideally we don't change these files manually. ... View more

@Andrey Nikitin Well it is showing debug logs on my cluster, so something might be messed up while installating HS2. HIVE_CONF_DIR=/usr/hdp/current/hive-server2/conf/conf.server /usr/hdp/current/hive-server2/bin/hiveserver2 -hiveconf hive.metastore.uris=" " -hiveconf hive.root.logger=DEBUG,console WARNING: Use "yarn jar" to launch YARN applications. 16/05/24 04:39:12 [main]: DEBUG common.LogUtils: Using hive-site.xml found on CLASSPATH at /etc/hive/hdp/conf.server/hive-site.xml 16/05/24 04:39:12 [main]: DEBUG server.HiveServer2: Logging initialized using configuration in file:/etc/hive/hdp/conf.server/hive-log4j.properties 16/05/24 04:39:12 [main]: INFO server.HiveServer2: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting HiveServer2 STARTUP_MSG: host = ey.example.com/240.0.0.11 STARTUP_MSG: args = [--hiveconf, hive.aux.jars.path=file:///usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar, -hiveconf, hive.metastore.uris= , -hiveconf, hive.root.logger=DEBUG,console] STARTUP_MSG: version = 1.2.1000.2.4.2.0-258 ...... .... 6/05/24 04:39:12 [main]: INFO server.HiveServer2: Starting HiveServer2 16/05/24 04:39:12 [main]: DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[Rate of successful kerberos logins and latency (milliseconds)], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops) 16/05/24 04:39:12 [main]: DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[Rate of failed kerberos logins and latency (milliseconds)], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops) 16/05/24 04:39:12 [main]: DEBUG lib.MutableMetricsFactory: field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(value=[GetGroups], about=, valueName=Time, type=DEFAULT, always=false, sampleName=Ops) 16/05/24 04:39:12 [main]: DEBUG impl.MetricsSystemImpl: UgiMetrics, User and group related metrics 16/05/24 04:39:12 [main]: DEBUG security.Groups: Creating new Groups object 16/05/24 04:39:12 [main]: DEBUG util.NativeCodeLoader: Trying to load the custom-built native-hadoop library... 16/05/24 04:39:12 [main]: DEBUG util.NativeCodeLoader: Loaded the native-hadoop library 16/05/24 04:39:12 [main]: DEBUG security.JniBasedUnixGroupsMapping: Using JniBasedUnixGroupsMapping for Group resolution 16/05/24 04:39:12 [main]: DEBUG security.JniBasedUnixGroupsMappingWithFallback: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMapping 16/05/24 04:39:12 [main]: DEBUG security.Groups: Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000 16/05/24 04:39:12 [main]: DEBUG session.SessionState: SessionState user: null 16/05/24 04:39:12 [main]: DEBUG security.UserGroupInformation: hadoop login 16/05/24 04:39:12 [main]: DEBUG security.UserGroupInformation: hadoop login commit 16/05/24 04:39:12 [main]: DEBUG security.UserGroupInformation: using local user:UnixPrincipal: root 16/05/24 04:39:12 [main]: DEBUG security.UserGroupInformation: Using user: "UnixPrincipal: root" with name root 16/05/24 04:39:12 [main]: DEBUG security.UserGroupInformation: User entry: "root" 16/05/24 04:39:12 [main]: DEBUG security.UserGroupInformation: UGI loginUser:root (auth:SIMPLE) ... View more

@Andrey Nikitin Can you run below command and share the console output? HIVE_CONF_DIR=/usr/hdp/current/hive-server2/conf/conf.server /usr/hdp/current/hive-server2/bin/hiveserver2 -hiveconf hive.metastore.uris=" " -hiveconf hive.root.logger=DEBUG,console ... View more