Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

service cloudera-scm-server 'active (exited)' with Java 1.8.0_181

service cloudera-scm-server 'active (exited)' with Java 1.8.0_181

New Contributor

Hi All,
please help to find out why after changing Java from 1.7 to 1.8.0_181 we can't deploy the cluster and get an error during deploy client configuration. ( There was an error when communicating with the server)

Status of service 'cloudera-scm-server' changes from 'active (running)' to 'active (exited)' during the first Running and Deploy client configuration.

Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-server; bad; vendor preset: disabled)
   Active: active (running) since Tue 2019-08-20 09:30:55 UTC; 16min ago
     Docs: man:systemd-sysv-generator(8)
   CGroup: /system.slice/cloudera-scm-server.service
           ├─14471 su cloudera-scm -s /bin/bash -c nohup /usr/sbin/cmf-server
           └─14473 /usr/java/jdk1.8.0_181-cloudera/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server -Dlog4j.co..
● cloudera-scm-server.service - LSB: Cloudera SCM Server
   Loaded: loaded (/etc/rc.d/init.d/cloudera-scm-server; bad; vendor preset: disabled)
   Active: active (exited) since Tue 2019-08-20 09:30:55 UTC; 18min ago

All configuration the same, only Java version was changed. 

We use CDH 5.15.2, Altus Director 2.7, and prebaked image with Java 1.8.0_181 on Centos 7.

 

5 REPLIES 5

Re: service cloudera-scm-server 'active (exited)' with Java 1.8.0_181

Expert Contributor

Hi YurriL,

 

can you please provide

  1. the latest output from /var/log/cloudera-scm-server/cloudera-scm-server.log
  2. any errors or warnings from "journalctl -xe"

Re: service cloudera-scm-server 'active (exited)' with Java 1.8.0_181

New Contributor
2019-08-20 12:37:26,745 INFO 179986820@scm-web-19:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command First Run BasicCmdArgs{args=[CD-AWS_S3-Pznldbee, CD-ZOOKEEPER-OaAVutjk, CD-HDFS-CqmTPAuV, CD-KAFKA-ngsMquLo, CD-SENTRY-wFibPeTJ, CD-YARN-mrECVkcx, CD-HIVE-NAMKHwSU, CD-IMPALA-xDNLSCjr, CD-OOZIE-GplqKSro, CD-SQOOP-vUHtkubv, CD-HUE-VMbnNPaO]}.
2019-08-20 12:37:26,812 INFO 179986820@scm-web-19:com.cloudera.cmf.service.FirstRunCommand: Disabling Faster First Run due to presence of HDFS HA.
2019-08-20 12:37:26,812 INFO 179986820@scm-web-19:com.cloudera.cmf.service.FirstRunCommand: Faster First Run enabled: false. Start wait seconds: 20
2019-08-20 12:37:27,044 INFO 179986820@scm-web-19:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 8 steps in sequence
2019-08-20 12:37:27,044 INFO 179986820@scm-web-19:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in parallel
2019-08-20 12:37:27,044 INFO 179986820@scm-web-19:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Waiting for Cloudera Manager Agents to detect release: CDH 5.15.2.
2019-08-20 12:37:27,124 INFO 179986820@scm-web-19:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 10.0.1.180
2019-08-20 12:37:27,138 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Deploy Client Configuration on cluster user-cluster
2019-08-20 12:37:27,669 INFO CommandPusher:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command DeployClusterClientConfig BasicCmdArgs{args=[]}.
2019-08-20 12:37:27,669 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence
2019-08-20 12:37:27,669 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 5 steps in parallel
2019-08-20 12:37:27,669 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Deploy Client Configuration on service SENTRY-1
2019-08-20 12:37:27,727 INFO CommandPusher:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command deployClientConfig SvcCmdArgs{targetRoles=[DbRole{id=53, name=CD-S0265f753-SENTRY_SERVER-58cc2c2ba2290cbea8100cc7d79df953, hostName=ip-10-0-1-108.ec2.internal}], args=[]}. Service: DbService{id=12, name=CD-SENTRY-wFibPeTJ}
2019-08-20 12:37:27,728 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence
2019-08-20 12:37:27,728 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Generate and deploy client configuration.
2019-08-20 12:37:28,077 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Deploy Client Configuration on service HIVE-1
2019-08-20 12:37:28,536 INFO CommandPusher:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command deployClientConfig SvcCmdArgs{targetRoles=DbRole{id=9, name=CD-HIVE-NAMKHwSU-GATEWAY-58cc2c2ba2290cbea8100cc7d79df953, hostName=ip-10-0-1-108.ec2.internal},DbRole{id=10, name=CD-HIVE-NAMKHwSU-HIVEMETASTORE-58cc2c2ba2290cbea8100cc7d79df953, hostName=ip-10-0-1-108.ec2.internal},DbRole{id=18, name=CD-HIVE-NAMKHwSU-GATEWAY-323fb05f1245fb24dcd2a1d5c3e789e7, hostName=ip-10-0-1-50.ec2.internal},DbRole{id=21, name=CD-HIVE-NAMKHwSU-GATEWAY-85cddaaff129ad48e098c8bce79ab651, hostName=ip-10-0-3-22.ec2.internal},DbRole{id=17, name=CD-HIVE-NAMKHwSU-HIVESERVER2-481feee19833ca1500ab7b88936c8b73, hostName=ip-10-0-1-241.ec2.internal},DbRole{id=16, name=CD-HIVE-NAMKHwSU-HIVEMETASTORE-481feee19833ca1500ab7b88936c8b73, hostName=ip-10-0-1-241.ec2.internal},DbRole{id=11, name=CD-HIVE-NAMKHwSU-HIVESERVER2-58cc2c2ba2290cbea8100cc7d79df953, hostName=ip-10-0-1-108.ec2.internal},DbRole{id=13, name=CD-HIVE-NAMKHwSU-GATEWAY-94bb76bb3845d885b0b4b7aef47125e5, hostName=ip-10-0-1-140.ec2.internal},DbRole{id=20, name=CD-HIVE-NAMKHwSU-GATEWAY-737f61d86c70b41923ac33cfede3f583, hostName=ip-10-0-2-210.ec2.internal},DbRole{id=12, name=CD-HIVE-NAMKHwSU-GATEWAY-adf311de339f2c4fe6c7cc6b8d398b30, hostName=ip-10-0-1-113.ec2.internal}...(3 more roles), args=[]}. Service: DbService{id=3, name=CD-HIVE-NAMKHwSU}
2019-08-20 12:37:28,536 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence
2019-08-20 12:37:28,536 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Generate and deploy client configuration.
2019-08-20 12:37:29,212 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Deploy Client Configuration on service YARN-1
2019-08-20 12:37:29,402 INFO CommandPusher:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command deployClientConfig SvcCmdArgs{targetRoles=[DbRole{id=37, name=CD-YARN-mrECVkcx-NODEMANAGER-af56a93fe2cd7bb00ab8f433fbe46739, hostName=ip-10-0-2-161.ec2.internal}, DbRole{id=33, name=CD-Y70cff05d-RESOURCEMANAGER-58cc2c2ba2290cbea8100cc7d79df953, hostName=ip-10-0-1-108.ec2.internal}, DbRole{id=34, name=CD-YARN-mrECVkcx-JOBHISTORY-58cc2c2ba2290cbea8100cc7d79df953, hostName=ip-10-0-1-108.ec2.internal}, DbRole{id=38, name=CD-YARN-mrECVkcx-NODEMANAGER-85cddaaff129ad48e098c8bce79ab651, hostName=ip-10-0-3-22.ec2.internal}, DbRole{id=35, name=CD-YARN-mrECVkcx-NODEMANAGER-b31bec2d07093c32bee7c2f274aed6ba, hostName=ip-10-0-1-235.ec2.internal}, DbRole{id=36, name=CD-Y70cff05d-RESOURCEMANAGER-481feee19833ca1500ab7b88936c8b73, hostName=ip-10-0-1-241.ec2.internal}], args=[]}. Service: DbService{id=8, name=CD-YARN-mrECVkcx}
2019-08-20 12:37:29,402 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence
2019-08-20 12:37:29,402 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Generate and deploy client configuration.
2019-08-20 12:37:30,019 INFO metric-schema-updater:com.cloudera.cmon.components.MetricSchemaManager: Metric schema updated in PT8.326S.
2019-08-20 12:37:30,164 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Deploy Client Configuration on service HDFS-1
2019-08-20 12:37:30,303 INFO CommandPusher:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command deployClientConfig SvcCmdArgs{targetRoles=DbRole{id=40, name=CD-HDFS-CqmTPAuV-JOURNALNODE-adf311de339f2c4fe6c7cc6b8d398b30, hostName=ip-10-0-1-113.ec2.internal},DbRole{id=50, name=CD-HDFS-CqmTPAuV-DATANODE-85cddaaff129ad48e098c8bce79ab651, hostName=ip-10-0-3-22.ec2.internal},DbRole{id=41, name=CD-HDFS-CqmTPAuV-NAMENODE-adf311de339f2c4fe6c7cc6b8d398b30, hostName=ip-10-0-1-113.ec2.internal},DbRole{id=47, name=CD-H9881db27-FAILOVERCONTROLLER-737f61d86c70b41923ac33cfede3f583, hostName=ip-10-0-2-210.ec2.internal},DbRole{id=48, name=CD-HDFS-CqmTPAuV-JOURNALNODE-737f61d86c70b41923ac33cfede3f583, hostName=ip-10-0-2-210.ec2.internal},DbRole{id=44, name=CD-HDFS-CqmTPAuV-BALANCER-323fb05f1245fb24dcd2a1d5c3e789e7, hostName=ip-10-0-1-50.ec2.internal},DbRole{id=49, name=CD-HDFS-CqmTPAuV-NAMENODE-737f61d86c70b41923ac33cfede3f583, hostName=ip-10-0-2-210.ec2.internal},DbRole{id=42, name=CD-HDFS-CqmTPAuV-DATANODE-b31bec2d07093c32bee7c2f274aed6ba, hostName=ip-10-0-1-235.ec2.internal},DbRole{id=45, name=CD-HDFS-CqmTPAuV-HTTPFS-323fb05f1245fb24dcd2a1d5c3e789e7, hostName=ip-10-0-1-50.ec2.internal},DbRole{id=46, name=CD-HDFS-CqmTPAuV-DATANODE-af56a93fe2cd7bb00ab8f433fbe46739, hostName=ip-10-0-2-161.ec2.internal}...(2 more roles), args=[]}. Service: DbService{id=9, name=CD-HDFS-CqmTPAuV}
2019-08-20 12:37:30,309 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence
2019-08-20 12:37:30,309 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Generate and deploy client configuration.
2019-08-20 12:37:30,549 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute command Deploy Client Configuration on service KAFKA-1
2019-08-20 12:37:30,566 INFO CommandPusher:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing service command deployClientConfig SvcCmdArgs{targetRoles=[DbRole{id=51, name=CD-KAFKA-ngsMquLo-KAFKA_BROKER-94bb76bb3845d885b0b4b7aef47125e5, hostName=ip-10-0-1-140.ec2.internal}], args=[]}. Service: DbService{id=10, name=CD-KAFKA-ngsMquLo}
2019-08-20 12:37:30,566 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Execute 1 steps in sequence
2019-08-20 12:37:30,566 INFO CommandPusher:com.cloudera.cmf.command.flow.CmdStep: Executing command work: Generate and deploy client configuration.
2019-08-20 12:37:30,583 INFO CommandPusher:com.cloudera.cmf.service.AbstractRoleHandler: Client configs of DbService{id=4, name=CD-ZOOKEEPER-OaAVutjk} wanted but not available.
2019-08-20 12:37:30,720 INFO avro-servlet-hb-processor-0:com.cloudera.server.common.AgentAvroServlet: (45 skipped) AgentAvroServlet: heartbeat processing stats: average=22ms, min=3ms, max=244ms.
2019-08-20 12:37:31,280 INFO avro-servlet-hb-processor-0:com.cloudera.cmf.command.components.StalenessChecker: No staleness check scheduled, scheduling one in 30 seconds
2019-08-20 12:37:32,516 INFO metric-schema-updater:com.cloudera.cmon.MetricSchema: Switching metrics schema from 3yUo1DPTXB4GQELmbFik8-Hfrds to JD_R1wFZTCnbR7qAoUTAM6uoW9k
2019-08-20 12:37:38,048 INFO 257345501@scm-web-24:com.cloudera.server.web.cmf.AuthenticationSuccessEventListener: Authentication success for user: 'admin' from 10.0.1.180

2. from "journalctl -xe"

Out of memory: Kill process 14476 (java) score 299 or sacrifice child
Killed process 14476 (java) total-vm:5003380kB, anon-rss:2305344kB, file-rss:0kB, shmem-rss:0kB

 

Re: service cloudera-scm-server 'active (exited)' with Java 1.8.0_181

Expert Contributor

While the output of the log does not provide any insights (are these the logs before the server crashed?) the journalctl output could hint a OOM of the Cloudera Manager Server.

 

You can try to change the heap to 4GB and test if the behaviour persists. To do so, alter /etc/default/cloudera-scm-server and change -Xmx2G to -Xmx4G in this line:

export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"

 

Re: service cloudera-scm-server 'active (exited)' with Java 1.8.0_181

New Contributor
benhadoop,
thank for your replays,
 
This logs during the crashed (last logs).
Changed to -Xmx4G didn't help.
Highlighted

Re: service cloudera-scm-server 'active (exited)' with Java 1.8.0_181

Super Guru

@YuriiL,

 

Looking at the information you shared, the key part is:

 

Out of memory: Kill process 14476 (java) score 299 or sacrifice child
Killed process 14476 (java) total-vm:5003380kB, anon-rss:2305344kB, file-rss:0kB, shmem-rss:0kB

 This is not Java returning an OutOfMemoryError exception; rather, this is the Operating System killing the process (which happens to be Java) due to what the OS perceives as a low memory event.  This is for the entire host.  Depending on what your OS OOM killer settings are, it will determine if a low memory situation warrants killing off a process to sacrifice it for the good of the rest of the system.

 

I recommend taking a closer look at how much memory is free on this host and try stopping any unnecessary processes and checking if one process may be taking up more memory than expected.

 

NOTE, I'd back out the -Xmx4G change you made in /etc/default/cloudera-scm-server since it will actually allow CM to use more memory which is more likely to cause the OOM killer to need to do its thing.

I wouldn't blame Java too much here till we find out more about the free memory on this host.