Cloudera Data Analytics (CDA)

Hi! I'm thinking about updating our Nifi and I'd like to ask a few questions. Is Nifi version 2 already recommended for use in production? What about using it in containers? Regards! ... View more

Summary Cloudera Manager is the best-in-class holistic interface that provides end-to-end system management and key enterprise features to provide granular visibility into and control over every part of an enterprise data hub. From time to time, you’ll need to increase the logging level if you’re troubleshooting issues within Cloudera Manager. Change your Cloudera Manager Server logging level The Cloudera Manager Server must be configured from the command line within Cloudera Manager Server itself. Log in as root to your Cloudera Manager Server: mbush@mbush-MBP16 CDSW % ssh root@<CM SERVER FQDN>   Check the Cloudera Manager Server log4j file: [root@<CM SERVER FQDN> ~]# cat /etc/cloudera-scm-server/log4j.properties # Copyright (c) 2012 Cloudera, Inc. All rights reserved. # # !!!!! IMPORTANT !!!!! # The Cloudera Manager server finds its log file by querying log4j. It # assumes that the first file appender in this file is the server log. # See LogUtil.getServerLogFile() for more details. # # Define some default values that can be overridden by system properties cmf.root.logger=INFO,CONSOLE cmf.log.dir=. cmf.log.file=cmf-server.log cmf.perf.log.file=cmf-server-perf.log cmf.jetty.log.file=cmf-server-nio.log .. .. ..   The key line to control the logging level of the Cloudera Manager Server is highlighted in red.   This can be done by amending the cmf.root.logger parameter and restarting the CM Server: cmf.root.logger=DEBUG,CONSOLE #RESTART THE CLOUDERA MANAGER SERVICE TO COMMIT THE AMENDMENT [root@<CM SERVER FQDN> ~]# systemctl restart cloudera-scm-server   If you would like to know more about controlling further elements, refer to another very helpful Cloudera Community page -  How to enable debug logging for Cloudera Manager server. This blog also goes into more detail about how you can amend the logging level at the CM Server binary level (if you are simply unable to get the CM Server to start at all). ... View more

Summary Cloudera Manager is the best-in-class holistic interface that provides end-to-end cluster management and key enterprise features to provide granular visibility into and control over every part of an open data lakehouse. The optimization steps below complement Cloudera’s Optimize the Cloudera Manager Server page. Investigation & Resolution Monitor your Cloudera Manager Server Heap We have provided a useful suite of Cloudera Manager dashboards in the blog Deploy your Cloudera Manager Dashboards. The dashboard called “MB - MGMT Cluster - JVM GC Sizing” includes a set of charts that focus on the Cloudera Manager Server heap which will enable you to easily visualize abnormal characteristics of the heap. This portion shows a 6hr time series set:   Tune your Cloudera Manager Server Heap The Cloudera Manager Server must be configured from the command line within Cloudera Manager Server itself. Log in as root to your Cloudera Manager Server: mbush@mbush-MBP16 CDSW % ssh root@<CM SERVER FQDN> Check the Cloudera Manager Server config file: [root@<CM SERVER FQDN> ~]# cat /etc/default/cloudera-scm-server # # Specify any command line arguments for the Cloudera SCM Server here. # CMF_SERVER_ARGS="" # # Locate the JDBC driver jar file. # # The default value is the default system mysql driver on RHEL/CentOS/Ubuntu # and the standard, documented location for where to put the oracle jar in CM # deployments. # export CMF_JDBC_DRIVER_JAR="/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar" # # You can override JAVA_HOME here if your java is not on the normal search path # export JAVA_HOME=/usr/java/default # # Java Options. # # Default value sets Java maximum heap size to 2GB, and Java maximum permanent # generation size to 256MB. # export CMF_JAVA_OPTS="-Xmx4G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp" The key line to control the tuning of the Cloudera Manager Server heap is highlighted in red.   Above, you can see the default settings that we get with any vanilla deployment of the Cloudera Manager service.  These parameters will need to be tuned as your Cloudera CDP cluster becomes larger and busier, and that tuning is particularly important when any serious use of the Cloudera Manager API is introduced.   We recommend that you suitably raise the overall heap size based on your cluster size and CM API needs, and disable adaptive heap sizing & control the JVM heap ratio. This can be done by amending the CMF_JAVA_OPTS and restarting the CM Server: export CMF_JAVA_OPTS="-Xms16G -Xmx16G -XX:NewRatio=2 -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp" #RESTART THE CLOUDERA MANAGER SERVICE TO COMMIT THE AMENDMENT [root@<CM SERVER FQDN> ~]# systemctl restart cloudera-scm-server   ... View more

If you are responsible for managing the CDSW service, some error codes may be a bit of a mystery. Hope this table helps! Error Code Explanation 1 Failed with exit code 1 2 Misuse of shell builtins 126 Command invoked cannot execute or exit status unknown 127 Command not found 128 Invalid argument to exit 129 Timeout 130 SIGINT: Script terminated by Control-C or pushing the 'interrupt' button 131 SIGQUIT: Terminal quit signal 132 SIGILL: Illegal instructions 133 SIGTRAP: Trace/Breakpoint Trap 134 SIGABRT: Process abort signal 135 SIGEMT: Emulation trap 136 SIGFPE: Arithmetic exception 137 SIGKILL: Process killed. This sometimes indicates that the console ran out of memory. You may need to use a larger console 138 SIGBUS: Access to an undefined portion of a memory object 139 SIGSEGV: Invalid memory reference 140 SIGSYS: Bad system call 141 SIGPIPE: Broken pipe 142 SIGALRM: Alarm clock 143 SIGTERM: Process terminated normally 144 Worker console started after its master console stopped 146 SIGCHLD: A child process was stopped or terminated 148 SIGTSTP: Terminal stop signal 151 SIGSTOP: Stop the process 152 SIGTSTP: Terminal stopped 153 SIGCONT: Continue executing, if stopped 154 SIGTTIN: Background process attempting read 155 SIGTTOU: Background process attempting write 156 SIGVTALRM: Virtual timer expired 157 SIGPROF: Profiling timer expired 158 SIGXCPU: CPU time limit exceeded 159 SIGXFSZ: File size limit exceeded ... View more