The Resource manager starts and shuts down in couple of mins. It shows unpexected exits and there are no error in logs for resource manager.   Is RHEL 7.5 m5.xlarge (4CPU, 16GB) machine capable of running cloudera with spark2,oozie,yarn,hue,hive and also cloudera manager?   Am i missing something here? Please help. ... View more

I have not run any job yet, this error comes even before running a job. When it tries to invoke the internal dr.who job, it goes down.   I have done the memory setting as per your suggestion earlier.   Are there any other paramters i should change. Following is my current settings:   yarn.app.mapreduce.am.resource.mb - 2 gb mapreduce.map.memory.mb - 1 gb mapreduce.reduce.memory.mb -   1 gb mapreduce.job.heap.memory-mb.ratio - 0.8 Client Java Heap Size in Bytes - 787 mb Java Heap Size of JobHistory Server in Bytes - 431 mb memory.soft_limit_in_bytes - -1 mb memory.hard_limit_in_bytes - -1 mb Java Heap Size of NodeManager in Bytes - 1 gb Container Memory - yarn.nodemanager.resource.memory-mb - 2gb Java Heap Size of ResourceManager in Bytes - 1 gb yarn.scheduler.minimum-allocation-mb - 1 gb yarn.scheduler.increment-allocation-mb - 512 mb yarn.scheduler.maximum-allocation-mb - 2 gb ... View more

New Logs after changing the memory:   9:45:27.031 AM INFO RMAppImpl Application application_1532511906743_0001 failed 2 times due to AM Container for appattempt_1532511906743_0001_000002 exited with exitCode: 143 For more detailed output, check application tracking page:http://ip-172-31-25-185.ap-south-1.compute.internal:8088/proxy/application_1532511906743_0001/Then, click on links to logs of each attempt. Diagnostics: Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 Killed by external signal Failing this attempt. Failing the application. 9:45:27.032 AM INFO RMAppImpl application_1532511906743_0001 State change from FINAL_SAVING to FAILED on event = APP_UPDATE_SAVED 9:45:27.033 AM WARN RMAuditLogger USER=dr.who OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1532511906743_0001 failed 2 times due to AM Container for appattempt_1532511906743_0001_000002 exited with exitCode: 143 For more detailed output, check application tracking page:http://ip-172-31-25-185.ap-south-1.compute.internal:8088/proxy/application_1532511906743_0001/Then, click on links to logs of each attempt. Diagnostics: Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 Killed by external signal Failing this attempt. Failing the application. APPID=application_1532511906743_0001 9:45:27.035 AM INFO RMAppManager$ApplicationSummary appId=application_1532511906743_0001,name=hadoop,user=dr.who,queue=root.users.dr_dot_who,state=FAILED,trackingUrl=http://ip-172-31-25-185.ap-south-1.compute.internal:8088/cluster/app/application_1532511906743_0001,appMasterHost=N/A,startTime=1532511923908,finishTime=1532511926525,finalStatus=FAILED,memorySeconds=820,vcoreSeconds=0,preemptedAMContainers=0,preemptedNonAMContainers=0,preemptedResources=<memory:0\, vCores:0> ... View more

It is releated to memeory, logs shows that, i am not sure what values to set for the same. ... View more

Wed Jul 25 06:42:31 UTC 2018 JAVA_HOME=/usr/java/jdk1.8.0_144 using /usr/java/jdk1.8.0_144 as JAVA_HOME using 5 as CDH_VERSION using /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop-yarn as CDH_YARN_HOME using /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop-mapreduce as CDH_MR2_HOME using /run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER as CONF_DIR CONF_DIR=/run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER CMF_CONF_DIR=/etc/cloudera-scm-agent Wed Jul 25 08:29:22 UTC 2018 JAVA_HOME=/usr/java/jdk1.8.0_144 using /usr/java/jdk1.8.0_144 as JAVA_HOME using 5 as CDH_VERSION using /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop-yarn as CDH_YARN_HOME using /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop-mapreduce as CDH_MR2_HOME using /run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER as CONF_DIR CONF_DIR=/run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER CMF_CONF_DIR=/etc/cloudera-scm-agent # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 724828160 bytes for committing reserved memory. # An error report file with more information is saved as: # /run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER/hs_err_pid16884.log Wed Jul 25 08:29:24 UTC 2018 JAVA_HOME=/usr/java/jdk1.8.0_144 using /usr/java/jdk1.8.0_144 as JAVA_HOME using 5 as CDH_VERSION using /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop-yarn as CDH_YARN_HOME using /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop-mapreduce as CDH_MR2_HOME using /run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER as CONF_DIR CONF_DIR=/run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER CMF_CONF_DIR=/etc/cloudera-scm-agent # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 724828160 bytes for committing reserved memory. # An error report file with more information is saved as: # /run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER/hs_err_pid16973.log Wed Jul 25 08:29:26 UTC 2018 JAVA_HOME=/usr/java/jdk1.8.0_144 using /usr/java/jdk1.8.0_144 as JAVA_HOME using 5 as CDH_VERSION using /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop-yarn as CDH_YARN_HOME using /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop-mapreduce as CDH_MR2_HOME using /run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER as CONF_DIR CONF_DIR=/run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER CMF_CONF_DIR=/etc/cloudera-scm-agent # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 724828160 bytes for committing reserved memory. # An error report file with more information is saved as: # /run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER/hs_err_pid17063.log Wed Jul 25 08:29:29 UTC 2018 JAVA_HOME=/usr/java/jdk1.8.0_144 using /usr/java/jdk1.8.0_144 as JAVA_HOME using 5 as CDH_VERSION using /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop-yarn as CDH_YARN_HOME using /opt/cloudera/parcels/CDH-5.15.0-1.cdh5.15.0.p0.21/lib/hadoop-mapreduce as CDH_MR2_HOME using /run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER as CONF_DIR CONF_DIR=/run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER CMF_CONF_DIR=/etc/cloudera-scm-agent # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (mmap) failed to map 724828160 bytes for committing reserved memory. # An error report file with more information is saved as: # /run/cloudera-scm-agent/process/278-yarn-RESOURCEMANAGER/hs_err_pid17153.log ... View more

Above is the resource manager log. Should i provide any more logs? ... View more

/************************************************************ STARTUP_MSG: Starting ResourceManager STARTUP_MSG: user = yarn STARTUP_MSG: host = ip-172-31-25-185.ap-south-1.compute.internal/172.31.25.185 STARTUP_MSG: args = [] STARTUP_MSG: version = 2.6.0-cdh5.15.0 STARTUP_MSG: classpath = /run/cloudera-scm-agent/process/276-yarn-RESOURCEMANAGER:/run/cloudera-scm-agent/process/276-yarn-RESOURCEMANAGER:/run/cloudera-scm-agent/process/276-yarn- STARTUP_MSG: build = http://github.com/cloudera/hadoop -r e3cb23a1cb2b89d074171b44e71f207c3d6ffa50; compiled by 'jenkins' on 2018-05-24T11:19Z STARTUP_MSG: java = 1.8.0_144 ************************************************************/ org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532505341498_0002_000001 State change from ALLOCATED_SAVING to ALLOCATED on event = ATTEMPT_NEW_SAVED 2018-07-25 07:56:02,246 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching masterappattempt_1532505341498_0002_000001 2018-07-25 07:56:02,249 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up container Container: [ContainerId: container_1532505341498_0002_01_000001, NodeId: ip-172-31-25-185.ap-south-1.compute.internal:8041, NodeHttpAddress: ip-172-31-25-185.ap-south-1.compute.internal:8042, Resource: <memory:2048, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 172.31.25.185:8041 }, ] for AM appattempt_1532505341498_0002_000001 2018-07-25 07:56:02,249 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Create AMRMToken for ApplicationAttempt: appattempt_1532505341498_0002_000001 2018-07-25 07:56:02,249 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Creating password for appattempt_1532505341498_0002_000001 ... View more