Group Activity
Harness the power of Cloudera Manager Dashboards
... View more
1,902
2
0
Hi! I'm thinking about updating our Nifi and I'd like to ask a few questions. Is Nifi version 2 already recommended for use in production? What about using it in containers? Regards!
... View more
Summary
Cloudera Manager is the best-in-class holistic interface that provides end-to-end system management and key enterprise features to provide granular visibility into and control over every part of an enterprise data hub. From time to time, you’ll need to increase the logging level if you’re troubleshooting issues within Cloudera Manager.
Change your Cloudera Manager Server logging level
The Cloudera Manager Server must be configured from the command line within Cloudera Manager Server itself.
Log in as root to your Cloudera Manager Server:
mbush@mbush-MBP16 CDSW % ssh root@<CM SERVER FQDN>
Check the Cloudera Manager Server log4j file:
[root@<CM SERVER FQDN> ~]# cat /etc/cloudera-scm-server/log4j.properties
# Copyright (c) 2012 Cloudera, Inc. All rights reserved.
#
# !!!!! IMPORTANT !!!!!
# The Cloudera Manager server finds its log file by querying log4j. It
# assumes that the first file appender in this file is the server log.
# See LogUtil.getServerLogFile() for more details.
#
# Define some default values that can be overridden by system properties
cmf.root.logger=INFO,CONSOLE
cmf.log.dir=.
cmf.log.file=cmf-server.log
cmf.perf.log.file=cmf-server-perf.log
cmf.jetty.log.file=cmf-server-nio.log
..
..
..
The key line to control the logging level of the Cloudera Manager Server is highlighted in red.
This can be done by amending the cmf.root.logger parameter and restarting the CM Server:
cmf.root.logger=DEBUG,CONSOLE
#RESTART THE CLOUDERA MANAGER SERVICE TO COMMIT THE AMENDMENT
[root@<CM SERVER FQDN> ~]# systemctl restart cloudera-scm-server
If you would like to know more about controlling further elements, refer to another very helpful Cloudera Community page - How to enable debug logging for Cloudera Manager server. This blog also goes into more detail about how you can amend the logging level at the CM Server binary level (if you are simply unable to get the CM Server to start at all).
... View more
Summary
Cloudera Manager is the best-in-class holistic interface that provides end-to-end cluster management and key enterprise features to provide granular visibility into and control over every part of an open data lakehouse. The optimization steps below complement Cloudera’s Optimize the Cloudera Manager Server page.
Investigation & Resolution
Monitor your Cloudera Manager Server Heap
We have provided a useful suite of Cloudera Manager dashboards in the blog Deploy your Cloudera Manager Dashboards. The dashboard called “MB - MGMT Cluster - JVM GC Sizing” includes a set of charts that focus on the Cloudera Manager Server heap which will enable you to easily visualize abnormal characteristics of the heap. This portion shows a 6hr time series set:
Tune your Cloudera Manager Server Heap
The Cloudera Manager Server must be configured from the command line within Cloudera Manager Server itself.
Log in as root to your Cloudera Manager Server:
mbush@mbush-MBP16 CDSW % ssh root@<CM SERVER FQDN>
Check the Cloudera Manager Server config file:
[root@<CM SERVER FQDN> ~]# cat /etc/default/cloudera-scm-server
#
# Specify any command line arguments for the Cloudera SCM Server here.
#
CMF_SERVER_ARGS=""
#
# Locate the JDBC driver jar file.
#
# The default value is the default system mysql driver on RHEL/CentOS/Ubuntu
# and the standard, documented location for where to put the oracle jar in CM
# deployments.
#
export CMF_JDBC_DRIVER_JAR="/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar"
#
# You can override JAVA_HOME here if your java is not on the normal search path
# export JAVA_HOME=/usr/java/default
#
# Java Options.
#
# Default value sets Java maximum heap size to 2GB, and Java maximum permanent
# generation size to 256MB.
#
export CMF_JAVA_OPTS="-Xmx4G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"
The key line to control the tuning of the Cloudera Manager Server heap is highlighted in red.
Above, you can see the default settings that we get with any vanilla deployment of the Cloudera Manager service. These parameters will need to be tuned as your Cloudera CDP cluster becomes larger and busier, and that tuning is particularly important when any serious use of the Cloudera Manager API is introduced.
We recommend that you suitably raise the overall heap size based on your cluster size and CM API needs, and disable adaptive heap sizing & control the JVM heap ratio.
This can be done by amending the CMF_JAVA_OPTS and restarting the CM Server:
export CMF_JAVA_OPTS="-Xms16G -Xmx16G -XX:NewRatio=2 -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"
#RESTART THE CLOUDERA MANAGER SERVICE TO COMMIT THE AMENDMENT
[root@<CM SERVER FQDN> ~]# systemctl restart cloudera-scm-server
... View more
If you are responsible for managing the CDSW service, some error codes may be a bit of a mystery. Hope this table helps!
Error Code
Explanation
1
Failed with exit code 1
2
Misuse of shell builtins
126
Command invoked cannot execute or exit status unknown
127
Command not found
128
Invalid argument to exit
129
Timeout
130
SIGINT: Script terminated by Control-C or pushing the 'interrupt' button
131
SIGQUIT: Terminal quit signal
132
SIGILL: Illegal instructions
133
SIGTRAP: Trace/Breakpoint Trap
134
SIGABRT: Process abort signal
135
SIGEMT: Emulation trap
136
SIGFPE: Arithmetic exception
137
SIGKILL: Process killed. This sometimes indicates that the console ran out of memory. You may need to use a larger console
138
SIGBUS: Access to an undefined portion of a memory object
139
SIGSEGV: Invalid memory reference
140
SIGSYS: Bad system call
141
SIGPIPE: Broken pipe
142
SIGALRM: Alarm clock
143
SIGTERM: Process terminated normally
144
Worker console started after its master console stopped
146
SIGCHLD: A child process was stopped or terminated
148
SIGTSTP: Terminal stop signal
151
SIGSTOP: Stop the process
152
SIGTSTP: Terminal stopped
153
SIGCONT: Continue executing, if stopped
154
SIGTTIN: Background process attempting read
155
SIGTTOU: Background process attempting write
156
SIGVTALRM: Virtual timer expired
157
SIGPROF: Profiling timer expired
158
SIGXCPU: CPU time limit exceeded
159
SIGXFSZ: File size limit exceeded
... View more