sumit.nigam
Contributor
My Accepted Solutions
Show All

Re: How to set yarn.nodemanager.pmem-check-enabled...

by Contributor in Support Questions
‎04-06-2018 01:05 AM
‎04-06-2018 01:05 AM
Hi,   This does not seem to have worked with a latter version of CDH (5.13.1). There we had to set this through -  YARN Client Advanced Configuration Snippet (Safety Valve) for yarn-site.xml   So, what is the correct way to set this? Is this really changed with newer releases?   Thanks, Sumit ... View more

Re: Fetch JAVA_HOME set on cluster using Cloudera ...

by Contributor in Support Questions
‎04-25-2017 06:43 AM
‎04-25-2017 06:43 AM
One way I could find so far where I can find ENV variables is say by going to some instance specific page (such as say, NodeManager):   http://inldmcdh1.example.com:7180/cmf/services/7/instances/71/processes   The page shows: Environment Variables: INFA_TRUSTSTORE=/opt/manojkeyfiles JAVA_HOME=/opt/java/jdk1.8.0_45 YARN_LOG_DIR=/data/hadooplogs/var/log/hadoop-yarn YARN_ROOT_LOGGER=INFO,RFA IS_KERBERIZED=false YARN_LOGFILE=hadoop-cmf-yarn-NODEMANAGER-inldmcdh2.example.com.log.out NM_LOCAL_DIRS=/data/yarn/nm /dfs/1/yarn/nm /dfs/2/yarn/nm /dfs/3/yarn/nm /dfs/4/yarn/nm /dfs/5/yarn/nm CDH_VERSION=5 KRB5_CONFIG=/opt/keytab/krb5.conf YARN_NODEMANAGER_OPTS=-Xms1073741824 -Xmx1073741824 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled -Dhadoop.event.appender=,EventCatcher -XX:OnOutOfMemoryError={{AGENT_COMMON_DIR}}/killparent.sh HADOOP_CLIENT_CONF_DIR=/etc/hadoop/conf.cloudera.yarn   Is there a direct API to get the ENV vars from here?   Thanks, Sumit   ... View more

Fetch JAVA_HOME set on cluster using Cloudera Mana...

by Contributor in Support Questions
‎04-25-2017 03:02 AM
‎04-25-2017 03:02 AM
Hi,   Is there a way to programmatically fetch value of JAVA_HOME env variable set on cluster using Cloudera Manager APIs?    Thanks Sumit ... View more
Labels:

Re: Zookeeper slow fsync followed by CancelledKeyE...

by Contributor in Support Questions
‎02-06-2017 07:33 PM
‎02-06-2017 07:33 PM
@samurai - Yes, there were 2 main issues. One, was that these were VMs and another was that zookeeper was collocated with another service which shared the same disk. ... View more

Re: How to set yarn.nodemanager.pmem-check-enabled...

by Contributor in Support Questions
‎04-07-2016 03:05 AM
‎04-07-2016 03:05 AM
No, those are 2 different properties. One for vmem and the other one for pmem. ... View more

Re: Zookeeper kerberos issue or quorum issue?

by Contributor in Support Questions
‎12-02-2015 07:45 PM
‎12-02-2015 07:45 PM
Thanks Harsh,   So, to generalize, the mechanism level subcodes can always be taken as some failure in communicating with KDC, right?   I also see that despite this error, ZK does continue to function ... so is this error to be really treated seriously?   Thanks again. ... View more

Zookeeper kerberos issue or quorum issue?

by Contributor in Support Questions
‎11-23-2015 01:01 AM
‎11-23-2015 01:01 AM
I use a kerberized cluster and once in a while I notice following error in my zookeeper client logs:   15/11/15 15:46:53 ERROR client.ZooKeeperSaslClient: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Connection reset)] ) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state. 15/11/15 15:46:53 ERROR zookeeper.ClientCnxn: SASL authentication with Zookeeper Quorum member failed: javax.security.sasl.SaslException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Connection reset)] ) occurred when evaluating Zookeeper Quorum Member's received SASL token. Zookeeper Client will go to AUTH_FAILED state.   So, I had following doubt with this:   It is showing actual error to be connection reset. I am not sure connection RST to what? Is it to Kerberos KDC? But the log further seems to indicate that connection issue happened when connecting to ZK quorum member. So, in that case the RST flag is recd from ZK quorum member?   Thanks, Sumit ... View more
Labels:

Zookeeper slow fsync followed by CancelledKeyExcep...

by Contributor in Support Questions
‎10-12-2015 09:27 PM
‎10-12-2015 09:27 PM
I am trying to enable HA for Resource Mgr as well NameNode. However, very often the masters failover to standby. There is no issue with HA as such, but every failover ends up exhausting one application attempt. I notice following issues:   A series of slow fsync followed (sometimes only) by CancelledKeyException.     2015-10-12 17:22:41,000 - WARN [SyncThread:3:FileTxnLog@334] - fsync-ing the write ahead log in SyncThread:3 took 6943ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide 2015-10-12 17:22:41,001 - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@494] - Processed session termination for sessionid: 0x1505bcdb3e3054e 2015-10-12 17:22:41,002 - INFO [ProcessThread(sid:3 cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when processing sessionid:0x1505bcdb3e30003 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe txntype:unknown reqpath:n/a Error Path:null Error:KeeperErrorCode = Session moved 2015-10-12 17:22:41,004 - INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1007] - Closed socket connection for client /10.65.144.35:36030 which had sessionid 0x1505bcdb3e30003 2015-10-12 17:22:41,006 - ERROR [CommitProcessor:3:NIOServerCnxn@178] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404) at org.apache.zookeeper.server.quorum.Leader$ToBeAppliedRequestProcessor.processRequest(Leader.java:644) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) The time taken is some time as high as 10sec. This could surely timeout the clients, I suppose leading to deletion of ephemeral nodes that masters created.   Around this time, the masters switch over. I have seen that disk space is not a concern. However, at times the await time to ZK dataDir drive does show a surge. I also confirmed that GC pauses are minimal.   Any pointers would be really appreciated.  ... View more
Labels:

Re: Max apps per user.

by Contributor in Support Questions
‎08-25-2015 08:08 AM
‎08-25-2015 08:08 AM
I notice that memory on another CDH host is overcommitted.   Of the total 62GB phy memory, the memory allotted to YARN containers is 40GB and that consumes the major chunk. I notice that my apps remain in ACCEPTED state despite requesting for only 1GB for each container.    1. Is this 40GB reserved by YARN at startup? 2. Plus, I assume that does not contribute to apps remaining in ACCEPTED state because it is from this pool of 40GB that my apps should get memory allocated, right?   Thanks, Sumit ... View more

Re: Max apps per user.

by Contributor in Support Questions
‎08-20-2015 08:22 PM
‎08-20-2015 08:22 PM
So, the only thing which is not clear to me now is why was I allowed only 2 max apps per user when it was not set anywhere explicitly? Is it possible to infer that from data being given above? Or what other data point(s) influenced limiting it to 2 apps per user?   Thanks, Sumit ... View more

Re: Max apps per user.

by Contributor in Support Questions
‎08-20-2015 07:27 AM
‎08-20-2015 07:27 AM
Thank you Wilfred.   Where do I check " Max Application Master Share"?   Plus, does Fair Scheduler take independent disks into account when calculating number of containers possible?  ... View more

Re: Max apps per user.

by Contributor in Support Questions
‎08-13-2015 12:10 AM
‎08-13-2015 12:10 AM
Ok, figured one thing that Num Containers implies total running containers.   I have 2 independent physical disks on both nodes. ... View more

Max apps per user.

by Contributor in Support Questions
‎08-12-2015 10:40 PM
‎08-12-2015 10:40 PM
I have a 2 node cluster with each node having 8GB RAM and 4 cores.    On both node, there are apps running having consumed 2 cores each. This leaves me with 2 cores (x 2) on both nodes. Memory used is 4GB of total 16GB available to YARN containers.   Some important properties: yarn.nodemanager.resource.memory-mb = 20GB   (overcomitted, as I see) yarn.scheduler.minimum-allocation-mb = 1GB yarn.scheduler.maximum-allocation-mb = 5.47GB   yarn.nodemanager.resource.cpu-vcores = 12 yarn.scheduler.minimum-allocation-vcores = 1 yarn.scheduler.maximum-allocation-vcores = 12   Using Fair scheduler.   With above setting. when I spark-submit the app remains in ACCEPTED state. Here is what I am requesting through this:   spark.driver.memory=2G  spark.master=yarn-client spark.executor.memory=1G num-executors = 2 executor-memory = 1G executor-cores = 1   As I see, I am requesting a total of 3 cores (1 for driver, by default and 1x2 for executors). A single node does not have 3 cores but has 2 cores. So, ideally I should see them distributed across 2 nodes. Not sure why the spark job should remain in ACCEPTED state.  My default queue shows only 25% usage.   I notice the following settings too for my default root.default queue: Used Capacity:                                          25.0% Used Resources:                                        <memory:4096, vCores:4> Num Schedulable Applications:                  2 Num Non-Schedulable Applications:          1 Num Containers:                                         4 Max Schedulable Applications:                   2 Max Schedulable Applications Per User:    2   Why do I only get 4 containers in total? Or does it this indicate currently used containers (which in my case is 4)? Plus, why is max schedulable apps only 2? I have not set any user level limits or queue level limits under Dynamic Resource Pool settings.  ... View more
Labels:

Re: Speeding up deployment of app binaries

by Contributor in Support Questions
‎08-10-2015 09:35 AM
‎08-10-2015 09:35 AM
I also think we can probably compress the binaries before being copied to HDFS and have YARN uncompress them somehow? ... View more

Speeding up deployment of app binaries

by Contributor in Support Questions
‎08-07-2015 08:16 PM
‎08-07-2015 08:16 PM
Are there any recommendations to speed up deployment of app binaries to YARN?   I've been using RM REST APIs to submit apps to it with binaries located on HDFS. This tends to take a lot of time when the size of binaries to be deployed as YARN app are big in size (say, >500MB or more), and also when number of containers that I need are high.   I could probably speed this up by :   1. Turning off default 3 copies needed on HDFS 2. Using HDFS cluster-wide cache which can help avoid block reads 3. Using YARN resource localization   Do you have any recommendations which are definitely known to speed this up?    Thanks, Sumit ... View more
Labels:

Re: JOB Stuck in Accepted State

by Contributor in Support Questions
‎07-29-2015 09:41 PM
‎07-29-2015 09:41 PM
Does FairScheduler take only memory into consideration when making a decision or does it also use vcores? If it can depend upon multiple reasons, then again this may be another CR wherein user can get to know the exact reason (possibly through an API call) as to why an app is in ACCPETED state (such as memory, cores, disk space, queue limits, etc.) ... View more

Re: How to set yarn.nodemanager.pmem-check-enabled...

by Contributor in Support Questions
‎07-29-2015 08:48 PM
‎07-29-2015 08:48 PM
You beat me to the answer 🙂   Yes, I figured this has to be set in  NodeManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml.   Thanks! ... View more

Re: How to set yarn.nodemanager.pmem-check-enabled...

by Contributor in Support Questions
‎07-29-2015 08:43 PM
‎07-29-2015 08:43 PM
Thanks Wilfred.   I'd agree about not setting it to false. That's my idea too. The main reason to use that setting is to be able to do some functional testing without getting into tuning as yet.   So, is there a way I can set this property through UI?   ... View more

Re: What does state transition RUNNING --> ACCEPTE...

by Contributor in Support Questions
‎07-28-2015 10:32 PM
‎07-28-2015 10:32 PM
On point 1, I think I am getting hit by     https://issues.apache.org/jira/browse/YARN-3103 ... View more

Re: What does state transition RUNNING --> ACCEPTE...

by Contributor in Support Questions
‎07-28-2015 06:08 AM
‎07-28-2015 06:08 AM
Hbase on Yarn.   On a side note: 1. Is there a reason for security token to just fail like that after 15 mins of trying? Or I have some setup problem? That seems to be the first reason for attempt one to be killed. 2.  The last line about null container - I see it often. Is that a bug? And can that be ignored?   Thanks, Sumit ... View more

Re: What does state transition RUNNING --> ACCEPTE...

by Contributor in Support Questions
‎07-28-2015 03:55 AM
1 Kudo
‎07-28-2015 03:55 AM
1 Kudo
Hi Harsh - You are right, there is a prior attempt which got killed. Here are some log snippets as you asked:   Attempt 1 - app becomes RUNNING 2015-07-24 14:20:40,980 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1437726395811_0010_000001 State change from LAUNCHED to RUNNING 2015-07-24 14:20:40,981 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1437726395811_0010 State change from ACCEPTED to RUNNING   Some hrs later the tokens are renewed (900000ms) 2015-07-25 13:56:35,841 INFO org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager: Rolling master-key for container-tokens 2015-07-25 13:56:35,841 INFO org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager: Going to activate master-key with key-id 1834122077 in 900000ms 2015-07-25 13:56:35,841 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Rolling master-key for nm-tokens 2015-07-25 13:56:35,842 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Going to activate master-key with key-id 516071750 in 900000ms   The following 2 log lines keep repeating for the next 900000ms filling up logs: 2015-07-25 13:56:35,920 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Create AMRMToken for ApplicationAttempt: appattempt_1437726395811_0010_000001 2015-07-25 13:56:35,920 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Creating password for appattempt_1437726395811_0010_000001 ... ... 2015-07-25 14:11:35,772 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Create AMRMToken for ApplicationAttempt: appattempt_1437726395811_0010_00 0001 2015-07-25 14:11:35,772 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Creating password for appattempt_1437726395811_0010_000001   This fails (not sure why) and leads to app termination 2015-07-25 14:11:35,877 INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8030: readAndProcess from client 10.65.144.85 threw exception [org.apache.hadoop.security.token.SecretManager$InvalidToken: Invalid AMRMToken from appattempt_1437726395811_0010_000001] 2015-07-25 14:11:36,888 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1437726395811_0010_01_000001 Container Transitioned from RUNNING to COMPLETED   1st attempt done (RUNNING --> ACCEPTED) 2015-07-25 14:11:36,888 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1437726395811_0010_000001 State change from RUNNING to FINAL_SAVING 2015-07-25 14:11:36,889 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1437726395811_0010_000001 State change from FINAL_SAVING to FAILED 2015-07-25 14:11:36,890 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1437726395811_0010 State change from RUNNING to ACCEPTED   2nd attempt starts  2015-07-25 14:11:36,890 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Added Application Attempt appattempt_1437726395811_0010_000002 to scheduler from user: root 2015-07-25 14:11:36,891 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1437726395811_0010_000002 State change from SUBMITTED to SCHEDULED   Not sure what this Null container indicates: 2015-07-25 14:11:36,910 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Null container completed...   ... View more

What does state transition RUNNING --> ACCEPTED me...

by Contributor in Support Questions
‎07-28-2015 12:52 AM
1 Kudo
‎07-28-2015 12:52 AM
1 Kudo
I notice in RM logs that sometime my application transitions back from RUNNING to ACCEPTED state. Under what conditions would this happen? I thought this usually happens whenever RM or AM dies and recovers the applications. Such apps would transition from RUNNING --> ACCEPTED. Is that correct?   However, in my case both RM and NM recovery is disabled:   yarn.resourcemanager.recovery.enabled = false yarn.nodemanager.recovery.enabled = false   Thanks, Sumit ... View more
Labels:

How to set yarn.nodemanager.pmem-check-enabled?

by Contributor in Support Questions
‎07-27-2015 09:55 PM
‎07-27-2015 09:55 PM
I am using CDH 5.3.2. I am unable to set value for yarn.nodemanager.pmem-check-enabled through UI. I could add following to  ResourceManager Advanced Configuration Snippet (Safety Valve) for yarn-site.xml and restarted it   <property> <name>yarn.nodemanager.pmem-check-enabled</name> <value>false</value> </property>   However, I still see my apps getting killed due to physical limits being breached:   2015-07-27 18:53:46,528 [AMRM Callback Handler Thread] INFO HoyaAppMaster.yarn (HoyaAppMaster.java:onContainersCompleted(847)) - Container Completion for containerID=container_1437726395811_0116_01_000002, state=COMPLETE, exitStatus=-104, diagnostics=Container [pid=36891,containerID=container_1437726395811_0116_01_000002] is running beyond physical memory limits. Current usage: 1.0 GB of 1 GB physical memory used; 2.8 GB of 2.1 GB virtual memory used. Killing container. ... View more
Contact Me
Online
Offline
Last Visited
‎04-06-2018 02:29 AM
Public Statistics
Date Registered ‎07-27-2015 09:45 PM
Last Visited ‎04-06-2018 02:29 AM
Total Messages Posted 35
Total Kudos Received 2