Seeing the following error on HDP 2.3.0: 2015-10-19 07:33:03,353 ERROR mapred.ShuffleHandler (ShuffleHandler.java:exceptionCaught(1053)) - Shuffle error: java.lang.OutOfMemoryError: GC overhead limit exceeded at java.util.Arrays.copyOf(Arrays.java:2219) at java.util.ArrayList.grow(ArrayList.java:242) at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:216) at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:208) at java.util.ArrayList.add(ArrayList.java:440) -- at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536) at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) 2015-10-21 07:05:13,532 FATAL yarn.YarnUncaughtExceptionHandler (YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread Thread[Container Monitor,5,main] threw an Error. Shutting down now... java.lang.OutOfMemoryError: GC overhead limit exceeded at java.io.BufferedReader.<init>(BufferedReader.java:98) at java.io.BufferedReader.<init>(BufferedReader.java:109) at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:545) at org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.updateProcessTree(ProcfsBasedProcessTree.java:225) at org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:439) Not only try increasing "mapreduce.reduce.memory.mb" but also can I add "-XX:-UseGCOverheadLimit" in "mapreduce.admin.reduce.child.java.opts"? Also would it be a good idea to reduce "mapreduce.reduce.shuffle.input.buffer.percent" ? ... View more

HDP: 2.3.2 and Ambari 2.1.2 Have been trying to follow http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_hdfs_admin_tools/content/configuring_archival_storage.html and https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html but having a few issues. 1) Ambari doesn't allow me to type "[SSD]file:///...." 2) By using "[DISK]/hadoop/hdfs/data,[SSD]/hadoop/hdfs/ssd", I can read the data but can't write 3) Command examples in both URL do not work (I'm guessing it shouldn't be "dfsadmin") I'm getting tired of following docs.., so I was wondering if someone could share the note to set up Archival Storage. ... View more

Basically followed the instruction in http://hortonworks.com/blog/enabling-kerberos-hdp-active-directory-integration/ From ambari, everything looks OK but beeline command fails with "GSS initiate failed (state=08S01,code=0)" Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ... 28 more Checked time is synchronized and resolving IP/hostname is correct. Built another HDP 2.3.2 and Ambari 2.1.2 without AD this time (simple MIT KDC) And Ambari looks OK but, again, beeline fails with same error. How I'm starting beeline is like below: su - hive beeline -u "jdbc:hive2://hiveserver2_fqdn:10000/default;principal=hive/hiveserver2_fqdn@MY_REALM" I think i'm forgetting some setting... I appreciate any advice from you. Thank you ... View more

I was always wondering what value I should use for "jobtTacker" in my job.properites for Resource Manager HA. Now a customer asked same question, so I thought this might be a good opportunity to find out. Does anyone know which string we should use to utilize YARN Resource Manager HA? According to Google, Cloudera uses "logicaljt" but I don't see this string in HDP code so far. ... View more