About TCloud

TCloud · ‎08-14-2019

Currently stuck on this error: Run a set of services for the first time Completed only 8/9 steps. First failure: Failed to execute command Install Oozie ShareLib on service Oozie Aug 14, 11:32:18 PM 4.9m Execute 8 steps in sequence Completed only 8/9 steps. First failure: Failed to execute command Install Oozie ShareLib on service Oozie Aug 14, 11:32:23 PM 4.8m Execute 9 steps in parallel Completed only 8/9 steps. First failure: Failed to execute command Install Oozie ShareLib on service Oozie Aug 14, 11:32:23 PM 4.8m Installing the Oozie ShareLib in HDFS Failed to execute command Install Oozie ShareLib on service Oozie Oozie Aug 14, 11:32:23 PM 4.8m Install Oozie ShareLib Failed to install Oozie ShareLib. Oozie Aug 14, 11:32:23 PM 4.8m Upload Oozie ShareLib Command aborted because of exception: Command timed-out after 270 seconds Oozie Server (cm-r01en01) Aug 14, 11:32:37 PM 4.6m Upload Oozie ShareLib Command aborted because of exception: Command timed-out after 270 seconds Oozie Server (cm-r01en01) Aug 14, 11:32:37 PM 4.6m $> oozie/oozie.sh ["install-sharelib","oozie-sharelib-yarn","hdfs://cm-r01nn02.mws.mds.xyz:8020","775","/user/oozie","8"] stdout stderr Role Log Wed Aug 14 23:32:58 EDT 2019 JAVA_HOME=/usr/java/latest using 6 as CDH_VERSION CONF_DIR=/var/run/cloudera-scm-agent/process/166-oozie-OOZIE-SERVER-upload-sharelib CMF_CONF_DIR= Found Hadoop that supports Erasure Coding. Trying to disable Erasure Coding for path: /user/oozie/share/lib Done the destination path for sharelib is: /user/oozie/share/lib/lib_20190814233305 Running 1738 copy tasks on 8 threads Full log file $> oozie/oozie.sh ["install-sharelib","oozie-sharelib-yarn","hdfs://cm-r01nn02.mws.mds.xyz:8020","775","/user/oozie","8"] stdout stderr Role Log SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging. Full log file Same effect as the earlier error: nothing else installs after. Not sure if it's significant however originaly had RPM's installed not parcels. Reverted to parcels since the installer failed even detecting a PostgreSQL DB Cluster when RPM's were used (Had a separate post on that here). Cheers, TK

TCloud · ‎08-14-2019

This affects the Oozie setup and configuration. The other services are not installed since this error breaks the installation chain. This is the list of the steps and where it get's stuck on: Completed 1 of 1 step(s). Show All Steps Show Only Failed Steps Show Only Running Steps Run a set of services for the first time Completed only 8/9 steps. First failure: Failed to execute command Install Oozie ShareLib on service Oozie Aug 14, 12:30:25 AM 5m Execute 8 steps in sequence Completed only 8/9 steps. First failure: Failed to execute command Install Oozie ShareLib on service Oozie Aug 14, 12:30:30 AM 4.9m Execute 9 steps in parallel Completed only 8/9 steps. First failure: Failed to execute command Install Oozie ShareLib on service Oozie Aug 14, 12:30:30 AM 4.9m Installing the Oozie ShareLib in HDFS Failed to execute command Install Oozie ShareLib on service Oozie Oozie Aug 14, 12:30:30 AM 4.9m Install Oozie ShareLib Failed to install Oozie ShareLib. Oozie Aug 14, 12:30:31 AM 4.9m Upload Oozie ShareLib Command aborted because of exception: Command timed-out after 270 seconds Oozie Server (cm-r01en01) Aug 14, 12:30:50 AM 4.6m Upload Oozie ShareLib Command aborted because of exception: Command timed-out after 270 seconds Oozie Server (cm-r01en01) Aug 14, 12:30:50 AM 4.6m $> oozie/oozie.sh ["install-sharelib","oozie-sharelib-yarn","hdfs://cm-r01nn02.mws.mds.xyz:8020","775","/user/oozie","8"] stdout stderr Role Log log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395) at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:955) at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89) at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58) at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:947) at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:473) at org.apache.oozie.service.XLogService.init(XLogService.java:178) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactor1".

TCloud · ‎08-14-2019

Hey All, Getting the following on a brand new 6.2 installation and wondering what is the 'proper' fix for this. I can start symlinking to get things working but perhaps I haven't deployed a package or taken another step I should have taken earlier? + /usr/java/latest/bin/java -Xms52428800 -Xmx52428800 -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/oozie_oozie-OOZIE_SERVER-901d5713a53510380392378fa81b483d_pid18817.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh -Doozie.home.dir=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie -Doozie.config.dir=/run/cloudera-scm-agent/process/152-oozie-OOZIE-SERVER-upload-sharelib -Doozie.log.dir=/var/log/oozie -Doozie.log.file=oozie-cmf-oozie-OOZIE_SERVER-cm-r01en01.mws.mds.xyz.log.out -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=cm-r01en01.mws.mds.xyz -Doozie.http.port=11000 -Djava.net.preferIPv4Stack=true -Doozie.admin.port= -Dderby.stream.error.file=/var/log/oozie/derby.log -Doozie.instance.id=cm-r01en01.mws.mds.xyz -Djava.library.path=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/native -cp ':/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/accessors-smart-1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/activation-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/aggdesigner-algorithm-6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/ant-1.6.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/antlr-2.7.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/antlr-runtime-3.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/aopalliance-1.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/aopalliance-repackaged-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/apache-jsp-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/apache-jstl-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/apache-log4j-extras-1.2.17.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/asm-5.0.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/asm-commons-6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/asm-tree-6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/audience-annotations-0.5.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/automaton-1.11-8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/avro.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-anim-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-awt-util-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-bridge-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-codec-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-constants-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-css-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-dom-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-ext-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-gvt-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-i18n-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-parser-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-rasterizer-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-script-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-svg-dom-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-svggen-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-svgrasterizer-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-transcoder-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-util-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-xml-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/bcpkix-jdk15on-1.60.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/bcprov-jdk15on-1.60.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/bonecp-0.8.0.RELEASE.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/c3p0-0.9.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-beanutils-1.9.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-cli-1.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-codec-1.9.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-collections-3.2.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-compress-1.4.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-configuration2-2.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-crypto-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-daemon-1.0.13.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-dbcp-1.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-el-1.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-exec-1.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-httpclient-3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-io-2.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-lang-2.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-lang3-3.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-logging-1.1.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-math3-3.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-net-3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-pool-1.5.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/curator-client-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/curator-framework-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/curator-recipes-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/curator-x-discovery-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/datanucleus-api-jdo-4.2.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/datanucleus-core-4.1.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/datanucleus-rdbms-4.1.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/disruptor-3.3.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/dozer-5.5.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/ecj-4.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/ehcache-3.3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/ehcache-core-2.6.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/fastutil-6.5.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/findbugs-annotations-1.3.9-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/fst-2.50.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/geronimo-jms_1.1_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/geronimo-jpa_2.0_spec-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/geronimo-jta_1.1_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/gmetric4j-1.0.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/graphviz-java-0.7.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/gson-2.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/guava-11.0.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/guice-3.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/guice-assistedinject-3.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/guice-servlet-4.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-auth.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-client-3.0.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-hdfs-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-hdfs.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-mapreduce-client-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-mapreduce-client-core.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-mapreduce-client-jobclient.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-api.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-registry.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-server-applicationhistoryservice.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-server-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-server-resourcemanager.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-server-web-proxy.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-client-2.1.0-cdh6.2.0-tests.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-hadoop2-compat.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-hadoop-compat.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-http.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-mapreduce.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-metrics-api.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-metrics.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-procedure.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-protocol.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-protocol-shaded.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-replication.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-server.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-shaded-miscellaneous.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-shaded-netty.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-shaded-protobuf.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-zookeeper.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/HikariCP-2.6.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/HikariCP-java7-2.4.12.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-classification.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-hcatalog-pig-adapter.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-jdbc.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-llap-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-llap-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-llap-server.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-llap-tez.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-metastore.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-serde.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-service.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-service-rpc.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-shims-0.23.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-shims-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-shims.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-shims-scheduler.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hk2-api-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hk2-locator-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hk2-utils-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hsqldb-1.8.0.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/htrace-core4-4.1.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/httpclient-4.5.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/httpcore-4.4.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/j2v8_linux_x86_64-4.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/j2v8_macosx_x86_64-4.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/j2v8_win32_x86-4.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/j2v8_win32_x86_64-4.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-annotations-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-core-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-core-asl-1.9.13.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-databind-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-jaxrs-1.9.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-jaxrs-base-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-jaxrs-json-provider-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-mapper-asl-1.9.13.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-module-jaxb-annotations-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-xc-1.9.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jamon-runtime-2.3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jansi-1.9.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jasper-compiler-5.5.23.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jasper-runtime-5.5.23.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/JavaEWAH-1.1.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javaparser-1.0.11.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javassist-3.20.0-GA.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/java-util-1.9.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.annotation-api-1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.el-3.0.1-b11.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.inject-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.inject-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.jdo-3.2.0-m3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.servlet-api-3.1.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.servlet.jsp-2.3.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.servlet.jsp-api-2.3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.ws.rs-api-2.0.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javolution-5.5.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb2-basics-1.11.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb2-basics-runtime-1.11.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb2-basics-tools-1.11.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb-api-2.2.11.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb-impl-2.2.3-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jcip-annotations-1.0-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jcl-over-slf4j-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jcodings-1.0.18.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jcommander-1.30.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jdo-api-3.0.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jdom-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-client-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-client-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-common-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-container-servlet-core-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-core-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-guava-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-guice-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-json-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-media-jaxb-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-server-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-server-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-servlet-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jettison-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-annotations-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-http-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-io-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-jaas-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-jndi-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-plus-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-runner-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-schemas-3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-security-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-server-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-servlet-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-util-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-util-ajax-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-webapp-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-xml-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jline-2.12.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jms-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/joda-time-2.9.9.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/joni-2.1.11.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jpam-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jsch-0.1.54.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/json-io-2.5.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/json-simple-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/json-smart-2.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jsp-api-2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jsr305-3.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jsr311-api-1.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jta-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jul-to-slf4j-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jython-standalone-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jzlib-1.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-admin-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-client-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-common-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-core-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-crypto-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-identity-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-server-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-simplekdc-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-util-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-asn1-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-config-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-pkix-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-util-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-xdr-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kryo-2.22.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/leveldbjni-all-1.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/libfb303-0.9.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/libthrift-0.9.3-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/log4j-1.2.17.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/log4j-api-2.8.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/log4j-slf4j-impl-2.8.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/mail-1.4.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-core-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-ganglia-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-graphite-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-json-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-jvm-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/mssql-jdbc-6.2.1.jre7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/nashorn-promise-0.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/netty-3.10.6.Final.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/netty-all-4.1.17.Final.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/nimbus-jose-jwt-4.41.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/objenesis-1.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/okhttp-2.7.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/okio-1.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-client-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-core-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-core.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-fluent-job-api-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-fluent-job-api.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-sharelib-hcatalog-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-sharelib-hcatalog.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-sharelib-oozie-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-sharelib-oozie.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-tools-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-tools.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/opencsv-2.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-jdbc-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-kernel-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-lib-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-persistence-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-persistence-jdbc-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/org.eclipse.jgit-5.0.1.201806211838-r.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/osgi-resource-locator-1.0.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/paranamer-2.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/pig.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/postgresql-9.0-801.jdbc4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/protobuf-java-2.5.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/quartz-2.1.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/re2j-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/serializer-2.7.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/serp-1.15.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/servlet-api-2.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/slf4j-api-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/slf4j-log4j12-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/slf4j-simple-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/slider-core-0.90.2-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/snappy-java-1.1.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/stax2-api-3.1.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/stringtemplate-3.2.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/taglibs-standard-impl-1.2.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/taglibs-standard-spec-1.2.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/tephra-api-0.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/tephra-core-0.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/tephra-hbase-compat-1.0-0.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/transaction-api-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-api-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-common-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-core-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-discovery-api-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-discovery-core-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-zookeeper-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/validation-api-1.1.0.Final.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-api-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-client-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-common-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-server-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-servlet-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/woodstox-core-5.0.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xalan-2.7.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xbean-asm5-shaded-3.17.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xercesImpl-2.11.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xml-apis-1.4.01.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xml-apis-ext-1.3.04.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xmlenc-0.52.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xmlgraphics-commons-2.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xz-1.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/zookeeper.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libext/*.jar' org.apache.oozie.tools.OozieSharelibCLI create -fs hdfs://cm-r01nn02.mws.mds.xyz:8020 -locallib /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/oozie-sharelib-yarn -concurrency 8 log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.apache.oozie.service.XLogService.init(XLogService.java:149) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactorForRootLogger". log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.apache.oozie.service.XLogService.init(XLogService.java:149) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactor2". log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.apache.oozie.service.XLogService.init(XLogService.java:149) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactor1". SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console... log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395) at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:955) at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89) at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58) at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:947) at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:473) at org.apache.oozie.service.XLogService.init(XLogService.java:178) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactorForRootLogger". log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395) at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:955) at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89) at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58) at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:947) at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:473) at org.apache.oozie.service.XLogService.init(XLogService.java:178) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactor2". log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395) at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:955) at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89) at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58) at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:947) at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:473) at org.apache.oozie.service.XLogService.init(XLogService.java:178) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactor1". [14/Aug/2019 00:35:33 +0000] 18806 MainThread redactor INFO Killing with SIGTERM Cheers, TK

TCloud · ‎07-25-2019

So I've moved the HAproxy and Keepalived to another set of hosts. Same issue with certs however I did try to replace the cm-r01nn01 and cm-r01nn02 (Name Node's) with the SAN SSL Certificate that I've generated. Here is the writeup: ==> /var/log/cloudera-scm-agent/cloudera-scm-agent.log <== self.cfg.max_cert_depth) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__ self.conn.connect() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect sock.connect((self.host, self.port)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect ret = self.connect_ssl() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl return m2.ssl_connect(self.ssl, self._timeout) SSLError: certificate verify failed Replace the certificate cm-r01nn01 and cm-r01nn02 with the SAN SSL Cert one as follows: [root@cm-r01nn01 yum.repos.d]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv" cm-r01wn07.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry, cm-r01wn04.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry, cm-r01wn01.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry, cm-r01wn08.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry, cm-r01nn01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry, cm-r01wn05.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry, cm-r01wn02.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry, cm-r01en01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry, acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry, cm-r01nn02.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry, cm-r01wn06.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry, cm-r01wn03.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry, srv-c01.mws.mds.xyz, Jul 18, 2019, PrivateKeyEntry, cm-r01en02.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry, cm-c01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry, Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". [root@cm-r01nn01 yum.repos.d]# [root@cm-r01nn01 yum.repos.d]# [root@cm-r01nn01 yum.repos.d]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"|grep -Ei r01nn01 Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". cm-r01nn01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry, [root@cm-r01nn01 yum.repos.d]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn01.mws.mds.xyz ^C [root@cm-r01nn01 yum.repos.d]# [root@cm-r01nn01 yum.repos.d]# [root@cm-r01nn01 yum.repos.d]# [root@cm-r01nn01 yum.repos.d]# cd /root/srv-c01/ [root@cm-r01nn01 srv-c01]# ls -altri total 40 201326721 dr-xr-x---. 11 root root 4096 Jul 17 22:04 .. 134330464 drwxr-xr-x. 2 root root 4096 Jul 18 22:39 1 2230259 -rw-r--r--. 1 root root 2422 Jul 18 22:46 srv-c01.mws.mds.xyz.keystore.jks 2230300 -rw-r--r--. 1 root root 2801 Jul 18 22:49 srv-c01.mws.mds.xyz.keystore.p12 2230651 -rw-r--r--. 1 root root 1763 Jul 18 22:49 srv-c01.mws.mds.xyz.cert.pem 2230652 -rw-r--r--. 1 root root 1860 Jul 18 22:49 srv-c01.mws.mds.xyz.key.pem 2230653 -rw-r--r--. 1 root root 1505 Jul 18 22:51 srv-c01.mws.mds.xyz.pem 2230654 -rw-r--r--. 1 root root 1679 Jul 18 23:00 srv-c01.mws.mds.xyz.key.nopass.pem 2230257 drwxr-xr-x. 3 root root 4096 Jul 18 23:00 . 2230655 -rw-r--r--. 1 root root 3442 Jul 18 23:00 srv-c01.mws.mds.xyz-haproxy.pem [root@cm-r01nn01 srv-c01]# history|grep srv-c01.mws.mds.xyz.pem|grep openssl|tail 876 openssl x509 -in ./srv-c01.mws.mds.xyz.pem -noout -text 1029 history|grep srv-c01.mws.mds.xyz.pem|grep openssl|tail [root@cm-r01nn01 srv-c01]# openssl x509 -in ./srv-c01.mws.mds.xyz.pem -noout -text Certificate: Data: Version: 3 (0x2) Serial Number: 1594172762 (0x5f05255a) Signature Algorithm: sha256WithRSAEncryption Issuer: C=US, ST=California, L=Los Angeles, O=MDS, OU=MDS, CN=srv-c01.mws.mds.xyz Validity Not Before: Jul 19 02:46:18 2019 GMT Not After : Jul 16 02:46:18 2029 GMT Subject: C=US, ST=California, L=Los Angeles, O=MDS, OU=MDS, CN=srv-c01.mws.mds.xyz Subject Public Key Info: Public Key Algorithm: rsaEncryption Public-Key: (2048 bit) Modulus: 00:c5:a9:00:83:12:9e:02:86:32:4e:2b:a7:c6:1a: 6b:9d:e3:56:00:53:22:01:d8:db:83:cd:14:79:6a: 85:27:20:f6:5d:86:0e:0b:af:df:46:dd:c3:23:72: f0:bf:38:3e:cd:9f:92:e6:65:81:7b:26:32:50:fc: 81:0e:7b:dd:b4:61:6f:a7:56:ec:c8:fe:89:72:ec: e5:e0:63:61:92:77:0b:36:41:98:93:14:6d:53:a0: 24:fb:fb:77:40:98:5b:2f:d2:3c:65:4f:8b:65:33: e5:db:14:ce:01:d2:4f:9f:e4:c6:c8:35:50:09:a2: f3:48:0a:ac:06:fd:66:42:30:10:a4:e7:fa:a8:2b: 0b:2b:ef:ce:83:82:4e:0d:86:34:ce:0c:8d:0c:a2: f5:88:4d:38:9f:3b:dd:2e:6e:e3:8c:60:69:da:8d: a4:d4:db:d5:cd:26:91:95:ca:a2:47:de:3c:f3:8f: 52:b8:e5:b0:09:26:af:77:fb:a3:5b:40:f6:e8:1b: 66:d7:b7:1b:da:2c:6c:34:99:76:de:c4:9b:80:69: 25:d5:12:2f:cb:9b:c5:d2:7e:15:a7:50:5f:54:5c: 9d:6b:8c:c0:9c:03:3f:96:f3:8a:2c:a6:05:ec:a4: d3:83:84:61:13:da:57:6d:e8:8c:93:d9:40:38:24: 96:c9 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication, Code Signing, E-mail Protection, Time Stamping, OCSP Signing X509v3 Subject Alternative Name: DNS:srv-c01.mws.mds.xyz, DNS:cm-r01nn01.mws.mds.xyz, DNS:cm-r01nn02.mws.mds.xyz X509v3 Subject Key Identifier: F6:EA:97:6F:82:20:84:75:E9:63:71:2F:16:D6:41:8B:64:05:07:0D Signature Algorithm: sha256WithRSAEncryption 4f:35:6d:18:dc:5c:4a:65:db:8c:62:75:0b:f8:da:2b:14:72: 22:f7:3a:ba:15:17:58:41:46:3b:6b:6e:40:db:6b:be:e5:07: 82:d1:37:0a:d6:4e:96:14:f6:87:ca:ff:d3:5f:a9:94:de:81: e7:a1:28:94:0a:19:0b:f4:dc:ed:0a:a5:77:78:20:53:3f:3f: 03:54:67:a0:c4:a1:de:49:7d:e8:fc:2d:76:bd:7b:a5:98:cd: 45:7e:ba:21:79:e2:91:7d:f3:e9:d6:5d:b7:91:34:30:3a:e4: 3a:38:e9:33:9b:26:2e:3e:6c:c9:3d:5d:48:81:cb:35:2f:ff: 7a:ff:22:c2:f8:b5:a2:01:d0:54:7f:f2:08:33:89:78:80:af: 72:2d:d7:df:61:f0:4a:7f:d2:19:0d:c6:0c:51:ee:4e:c1:ed: 8d:8b:4f:82:17:47:6b:03:1a:f2:8b:00:cc:17:8a:75:ca:72: c0:a4:a7:12:87:32:16:89:15:2c:80:d1:07:fd:37:e8:bf:f5: 87:6b:a2:dd:9d:a4:c4:2c:68:f8:d9:15:dd:3c:40:6d:8b:e0: 6d:c4:87:6d:39:a9:6b:91:f6:0a:bc:7c:63:e7:f0:37:cb:7a: 5f:35:6c:5c:f9:bb:cb:58:1a:b9:9c:49:ab:24:ac:2a:c9:2d: 3f:b2:2f:68 [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"|grep -Ei r01nn01 Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". cm-r01nn01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry, [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"|grep -Ei r01nn01 Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". cm-r01nn01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry, [root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn01.mws.mds.xyz Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". [root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn02.mws.mds.xyz Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore srv-c01.mws.mds.xyz.keystore.p12 -deststoretype PKCS12 -destalias cm-r01nn01.mws.mds.xyz -srcalias srv-c01.mws.mds.xyz -deststorepass srv-c01.mws.mds.xyz -destkeypass srv-c01.mws.mds.xyz Importing keystore srv-c01.mws.mds.xyz.keystore.jks to srv-c01.mws.mds.xyz.keystore.p12... Enter source keystore password: [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"|grep -Ei r01nn Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". [root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn01.mws.mds.xyz keytool error: java.lang.Exception: Alias <cm-r01nn01.mws.mds.xyz> does not exist [root@cm-r01nn01 srv-c01]# keytool -list -keystore srv-c01.mws.mds.xyz.keystore.p12 Enter keystore password: Keystore type: PKCS12 Keystore provider: SUN Your keystore contains 2 entries srv-c01.mws.mds.xyz, Jul 18, 2019, PrivateKeyEntry, Certificate fingerprint (SHA1): 01:2D:6C:36:E6:7D:76:2B:45:66:91:7F:E8:B9:C2:61:A4:22:FB:D6 cm-r01nn01.mws.mds.xyz, Jul 26, 2019, PrivateKeyEntry, Certificate fingerprint (SHA1): 01:2D:6C:36:E6:7D:76:2B:45:66:91:7F:E8:B9:C2:61:A4:22:FB:D6 [root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn01.mws.mds.xyz keytool error: java.lang.Exception: Alias <cm-r01nn01.mws.mds.xyz> does not exist [root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -destalias cm-r01nn01.mws.mds.xyz Enter alias name: cm-r01nn01.mws.mds.xyz keytool error: java.lang.Exception: Alias <cm-r01nn01.mws.mds.xyz> does not exist [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass cm-r01nn01.mws.mds.xyz -alias cm-r01nn01.mws.mds.xyz keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect [root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass srv-c01.mws.mds.xyz -alias cm-r01nn01.mws.mds.xyz keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect [root@cm-r01nn01 srv-c01]# keytool -delete -keystore srv-c01.mws.mds.xyz.keystore.p12 -storepass srv-c01.mws.mds.xyz -alias cm-r01nn01.mws.mds.xyz [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# keytool -list -keystore srv-c01.mws.mds.xyz.keystore.p12 Enter keystore password: keytool error: java.io.IOException: keystore password was incorrect [root@cm-r01nn01 srv-c01]# keytool -list -keystore srv-c01.mws.mds.xyz.keystore.p12 Enter keystore password: Keystore type: PKCS12 Keystore provider: SUN Your keystore contains 1 entry srv-c01.mws.mds.xyz, Jul 18, 2019, PrivateKeyEntry, Certificate fingerprint (SHA1): 01:2D:6C:36:E6:7D:76:2B:45:66:91:7F:E8:B9:C2:61:A4:22:FB:D6 [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# history|grep import|grep srv-c01.mws.mds.xyz.keystore.p12 699 keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore srv-c01.mws.mds.xyz.keystore.p12 -deststoretype PKCS12 -srcalias srv-c01.mws.mds.xyz -deststorepass srv-c01.mws.mds.xyz 869 keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore srv-c01.mws.mds.xyz.keystore.p12 -deststoretype PKCS12 -srcalias srv-c01.mws.mds.xyz -deststorepass srv-c01.mws.mds.xyz -destkeypass srv-c01.mws.mds.xyz 1034 keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore srv-c01.mws.mds.xyz.keystore.p12 -deststoretype PKCS12 -destalias cm-r01nn01.mws.mds.xyz -srcalias srv-c01.mws.mds.xyz -deststorepass srv-c01.mws.mds.xyz -destkeypass srv-c01.mws.mds.xyz 1044 history|grep import|grep srv-c01.mws.mds.xyz.keystore.p12 [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -srcalias srv-c01.mws.mds.xyz -deststorepass changeit -srcstorepass srv-c01.mws.mds.xyz -destalias cm-r01nn01.mws.mds.xyz Importing keystore srv-c01.mws.mds.xyz.keystore.jks to /etc/pki/ca-trust/extracted/java/jssecacerts... Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -srcalias srv-c01.mws.mds.xyz -deststorepass changeit -srcstorepass srv-c01.mws.mds.xyz -destalias cm-r01nn02.mws.mds.xyz Importing keystore srv-c01.mws.mds.xyz.keystore.jks to /etc/pki/ca-trust/extracted/java/jssecacerts... Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# [root@cm-r01nn01 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias srv-c01.mws.mds.xyz -alias cm-r01nn01.mws.mds.xyz -v Alias name: cm-r01nn01.mws.mds.xyz Creation date: Jul 26, 2019 Entry type: PrivateKeyEntry Certificate chain length: 1 Certificate[1]: Owner: CN=srv-c01.mws.mds.xyz, OU=MDS, O=MDS, L=Los Angeles, ST=California, C=US Issuer: CN=srv-c01.mws.mds.xyz, OU=MDS, O=MDS, L=Los Angeles, ST=California, C=US Serial number: 5f05255a Valid from: Thu Jul 18 22:46:18 EDT 2019 until: Sun Jul 15 22:46:18 EDT 2029 Certificate fingerprints: MD5: A7:C0:9E:E4:CC:DC:7E:4B:3A:96:CF:11:58:6C:86:D2 SHA1: 01:2D:6C:36:E6:7D:76:2B:45:66:91:7F:E8:B9:C2:61:A4:22:FB:D6 SHA256: 62:11:52:4E:7C:1E:03:11:20:CA:41:7E:5D:22:64:F8:CE:CC:85:C6:07:06:A9:21:FE:25:F3:71:DD:20:00:49 Signature algorithm name: SHA256withRSA Subject Public Key Algorithm: 2048-bit RSA key Version: 3 Extensions: #1: ObjectId: 2.5.29.37 Criticality=false ExtendedKeyUsages [ serverAuth clientAuth codeSigning emailProtection timeStamping OCSPSigning ] #2: ObjectId: 2.5.29.17 Criticality=false SubjectAlternativeName [ DNSName: srv-c01.mws.mds.xyz DNSName: cm-r01nn01.mws.mds.xyz DNSName: cm-r01nn02.mws.mds.xyz ] #3: ObjectId: 2.5.29.14 Criticality=false SubjectKeyIdentifier [ KeyIdentifier [ 0000: F6 EA 97 6F 82 20 84 75 E9 63 71 2F 16 D6 41 8B ...o. .u.cq/..A. 0010: 64 05 07 0D d... ] ] Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". [root@cm-r01nn01 srv-c01]# [root@cm-r01wn08 srv-c01]# [root@cm-r01wn08 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -srcalias srv-c01.mws.mds.xyz -deststorepass changeit -srcstorepass srv-c01.mws.mds.xyz -destalias cm-r01nn01.mws.mds.xyz Importing keystore srv-c01.mws.mds.xyz.keystore.jks to /etc/pki/ca-trust/extracted/java/jssecacerts... Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". [root@cm-r01wn08 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -srcalias srv-c01.mws.mds.xyz -deststorepass changeit -srcstorepass srv-c01.mws.mds.xyz -destalias cm-r01nn02.mws.mds.xyz Importing keystore srv-c01.mws.mds.xyz.keystore.jks to /etc/pki/ca-trust/extracted/java/jssecacerts... Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". [root@cm-r01wn08 srv-c01]# [root@cm-r01wn08 srv-c01]# [root@cm-r01wn08 srv-c01]# [root@cm-r01wn08 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv" srv-c01.mws.mds.xyz, Jul 21, 2019, PrivateKeyEntry, acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry, cm-r01wn08.mws.mds.xyz, Jul 10, 2019, trustedCertEntry, cm-r01nn02.mws.mds.xyz, Jul 26, 2019, PrivateKeyEntry, cm-r01nn01.mws.mds.xyz, Jul 26, 2019, PrivateKeyEntry, Warning: The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12". [root@cm-r01wn08 srv-c01]# [root@cm-r01wn08 srv-c01]# But no luck. Again, is there any way to increase the debug level on the cloudera-scm-agent so it will print further information detailing what private and public key files it's opening up? Thx, TK

TCloud · ‎07-21-2019

Received the subject error when I replaced the certs with the SAN one that contained 3 hosts: keytool -genkeypair -alias cm-c01.mws.mds.xyz -keyalg RSA -keysize 2048 -dname "cn=cm-c01.mws.mds.xyz,OU=MDS,O=MDS,L=Los Angeles,ST=California,C=US" -keypass cm-c01.mws.mds.xyz -keystore cm-c01.mws.mds.xyz.keystore.jks -storepass cm-c01.mws.mds.xyz -validity 3650 -ext EKU=serverAuth,clientAuth,codeSigning,emailProtection,timeStamping,OCSPSigning -ext san=dns:cm-c01.mws.mds.xyz,dns:cm-r01nn01.mws.mds.xyz,dns:cm-r01nn02.mws.mds.xyz Updated jssecerts (in path above) with the new cert as well. Same issue. I'm running the load balancer and haproxy on the cm-r01nn01/02 . This is a problem since Cloudera opens up ports such as 7180, 7182 and 7183 on all available interfaces. So if I have a VIP running on the same host, Cloudera services try and do bind to it. Can't really have HAproxy running on port 7183 on the LB VIP if Cloudera services are already bound to the same port. Tried the Cloudera Manager Hostname Override but only got these errors: 12:25:32.076 AM WARN BasicScmProxy Exception while getting fetch configDefaults hash: none java.io.IOException: HTTPS hostname wrong: should be <srv-c01.mws.mds.xyz> at sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:649) at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:573) at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334) at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309) at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259) at com.cloudera.cmf.BasicScmProxy.authenticate(BasicScmProxy.java:277) at com.cloudera.cmf.BasicScmProxy.fetch(BasicScmProxy.java:607) at com.cloudera.cmf.BasicScmProxy.getFragmentAndHash(BasicScmProxy.java:696) at com.cloudera.cmf.DescriptorAndFragments.newDescriptorAndFragments(DescriptorAndFragments.java:65) at com.cloudera.cmon.firehose.Main.main(Main.java:396) Need to spend more time reading on the LB's and CM / CDH. Thx, TK

TCloud · ‎07-17-2019

Breakdown of what I'm getting with different configs: 1) frontend srvin log 127.0.0.1 local0 debug bind srv-c01:17182 ssl crt /etc/haproxy/certs/srv-c01.mws.mds.xyz-haproxy.pem no-sslv3 option tcplog default_backend srvback backend srvback log /dev/log local0 debug mode tcp option tcplog balance roundrobin server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7182 check server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7182 check Results in: [17/Jul/2019 21:12:12 +0000] 25588 MainThread agent ERROR Heartbeating to srv-c01.mws.mds.xyz:17182 failed. Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1396, in _send_heartbeat response = self.requestor.request('heartbeat', heartbeat_data) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request return self.issue_request(call_request, message_name, request_datum) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request call_response = self.transceiver.transceive(call_request) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive result = self.read_framed_message() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 489, in read_framed_message framed_message = response_reader.read_framed_message() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 417, in read_framed_message raise ConnectionClosedException("Reader read 0 bytes.") ConnectionClosedException: Reader read 0 bytes. 2) frontend srvin log 127.0.0.1 local0 debug bind srv-c01:17182 option tcplog default_backend srvback backend srvback log /dev/log local0 debug mode tcp option tcplog balance roundrobin server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7182 check server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7182 check Results in: [17/Jul/2019 21:15:23 +0000] 25588 MainThread agent ERROR Heartbeating to srv-c01.mws.mds.xyz:17182 failed. Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat self.cfg.max_cert_depth) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__ self.conn.connect() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect sock.connect((self.host, self.port)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 313, in connect if not check(self.get_peer_cert(), self.addr[0]): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Checker.py", line 125, in __call__ fieldName='subjectAltName') WrongHost: Peer certificate subjectAltName does not match host, expected srv-c01.mws.mds.xyz, got DNS:cm-r01nn01.mws.mds.xyz 3) frontend srvin log 127.0.0.1 local0 debug bind srv-c01:17182 ssl crt /etc/haproxy/certs/srv-c01.mws.mds.xyz-haproxy.pem no-sslv3 default_backend srvback backend srvback log /dev/log local0 debug mode http balance roundrobin cookie srv-c01 insert indirect nocache server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 cookie cm-r01nn01.mws.mds.xyz sni req.hdr(host) server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 cookie cm-r01nn02.mws.mds.xyz sni req.hdr(host) Results in: parsing [/etc/haproxy/haproxy.cfg:69] : 'server cm-r01nn02.mws.mds.xyz' unknown keyword 'sni'. Config uses a pem file with certs from the two nodes + the VIP ( concatenated from srv-c01, cm-r01nn01 and cm-r01nn02 ). The certs appear to work ok. The traffic isn't passing through, however. Config looks like this: [root@cm-r01wn08 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -v "#" | sed -e "/^$/d" [General] server_host=srv-c01.mws.mds.xyz server_port=17182 max_collection_wait_seconds=10.0 metrics_url_timeout_seconds=30.0 task_metrics_timeout_seconds=5.0 monitored_nodev_filesystem_types=nfs,nfs4,tmpfs local_filesystem_whitelist=ext2,ext3,ext4,xfs impala_profile_bundle_max_bytes=1073741824 stacks_log_bundle_max_bytes=1073741824 stacks_log_max_uncompressed_file_size_bytes=5242880 orphan_process_dir_staleness_threshold=5184000 orphan_process_dir_refresh_interval=3600 scm_debug=DEBUG dns_resolution_collection_interval_seconds=60 dns_resolution_collection_timeout_seconds=30 [Security] use_tls=1 max_cert_depth=9 verify_cert_file=/opt/cloudera/security/pki/cluster-vip.pem client_key_file=/opt/cloudera/security/pki/client-key.pem client_keypw_file=/opt/cloudera/security/pki/agent.pw client_cert_file=/opt/cloudera/security/pki/client-cert.pem [Hadoop] [Cloudera] [JDBC] [Cgroup_Paths] [root@cm-r01wn08 pki]# I would like to keep it pointed to the VIP because this guarantees the config will be identical on all hosts and ready for any kind of failover. Thx, TK

TCloud · ‎07-17-2019

Tried. Certificates appeared fine ( were recognized ). The issue appears to be between the Load Balancer VIP and HAproxy right now since I get this: [17/Jul/2019 03:56:22 +0000] 20834 MainThread agent ERROR Heartbeating to srv-c01.mws.mds.xyz:17182 failed. Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1396, in _send_heartbeat response = self.requestor.request('heartbeat', heartbeat_data) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request return self.issue_request(call_request, message_name, request_datum) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request call_response = self.transceiver.transceive(call_request) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive result = self.read_framed_message() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 489, in read_framed_message framed_message = response_reader.read_framed_message() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 417, in read_framed_message raise ConnectionClosedException("Reader read 0 bytes.") ConnectionClosedException: Reader read 0 bytes. This is really telling me, and please keep me honest, that the traffic isn't being passed between the VIP and the backend servers defined in the HAproxy config (see below). Currently trying to solve the above by incorporating what this page mentions, however, I need a newer HAproxy version since mine doesn't support SNI. My latest HAproxy config is as follows to try and solve the above issue by setting up HAproxy in TLS bridging mode: frontend srvin log 127.0.0.1 local0 debug bind srv-c01:17182 ssl crt /etc/haproxy/certs/srv-c01.mws.mds.xyz-haproxy.pem no-sslv3 default_backend srvback backend srvback log /dev/log local0 debug mode http balance roundrobin cookie srv-c01 insert indirect nocache server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 cookie cm-r01nn01.mws.mds.xyz sni req.hdr(host) server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 cookie cm-r01nn02.mws.mds.xyz sni req.hdr(host) Current error is as follows but that's an HAproxy problem now that I'm following up on separately: [ALERT] 197/040530 (7560) : parsing [/etc/haproxy/haproxy.cfg:69] : 'server cm-r01nn02.mws.mds.xyz' unknown keyword 'sni'. Question I had above is, is there any other page other than the following that demonstrates the use of a VIP w/ HAproxy and TLS config when load balancing Cloudera services? Thx, TK

TCloud · ‎07-16-2019

How do I get SSL to work properly with CM and port 7182 through a VIP provided by a Load Balancer? A visualization: https://ibb.co/hY0GsVY

TCloud · ‎07-15-2019

Thanks very much again for taking the time and explain here. I've got this part working as well. In the meantime, I'm looking to enable high availability on the cluster. Have a few questions in this regard. 1) HAproxy is given as an example. I've used Haproxy + Keepalived for the CMS (7183) and a custom DNS entry, cm-c01.mws.mds.xyz to point to the cluster VIP. Everything works, including the certs for the UI. UI correctly displays the SSL certs for cm-c01 rather then the constituent hosts providing the backend. 2) I tried the same process with the Agent Avro port 7182 (If I'm calling it that correctly). I've set up a VIP, configured HAproxy and the proper SSL certs for srv-c01.mws.mds.xyz. This doesn't work. I've imported the right certs into the jssecerts file as well. HAproxy config: frontend cmin bind cm-c01:443 ssl crt /etc/haproxy/certs/cm-c01.mws.mds.xyz-haproxy.pem no-sslv3 default_backend cmback backend cmback mode http balance roundrobin server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7183 ssl check verify none port 7183 inter 12000 rise 3 fall 3 server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7183 ssl check verify none port 7183 inter 12000 rise 3 fall 3 frontend srvin log 127.0.0.1 local0 debug bind srv-c01:17182 ssl crt /etc/haproxy/certs/srv-c01.mws.mds.xyz-haproxy.pem no-sslv3 default_backend srvback backend srvback log /dev/log local0 debug mode http balance roundrobin server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 Each PEM file has a private and public key. How could I get this working with Self-Signed SSL certs? Do I need to revert the HAproxy config to a tcp pass-through? If so, how will I handle the case when it fails over and needs the cert file of the second host? Do I combine the public pem file certs? Thx, TK

TCloud · ‎07-11-2019

The error messages I get from the agent when I attempt (3): ==> /var/log/cloudera-scm-agent/status-stderr.log <== [11/Jul/2019:23:37:40] ENGINE Error in HTTPServer.tick Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cheroot/server.py", line 1339, in start self.tick() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cheroot/server.py", line 1408, in tick s, ssl_env = self.ssl_adapter.wrap(s) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/status_server.py", line 1048, in wrap ssl.accept_ssl() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 258, in accept_ssl return m2.ssl_accept(self.ssl, self._timeout) SSLError: unexpected eof [11/Jul/2019 23:37:43 +0000] 8193 MainThread agent ERROR Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed. Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat self.cfg.max_cert_depth) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__ self.conn.connect() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect sock.connect((self.host, self.port)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect ret = self.connect_ssl() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl return m2.ssl_connect(self.ssl, self._timeout) SSLError: sslv3 alert certificate unknown The pressing question I have above all others is how do I get the agent to print out enough logging to tell me WHICH certificate it attempted to load so I know the context under which the above is thrown? Right now, given the above error, I can't really take action without knowing the exact file the exceptions are referring too, other then deduce based on the change I've done. My setup as it is now: [ cm-r01en01 ] ( Utility Server ) [root@cm-r01en01 pki]# ls -altri total 32 3870062 drwxr-xr-x. 5 root root 37 Apr 1 23:09 .. 69943169 lrwxrwxrwx. 1 root root 62 Apr 1 23:11 server.jks -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.keystore.jks 69943167 -rw-r--r--. 1 cloudera-scm cloudera-scm 3449 Jul 8 03:32 cm-r01en01.mws.mds.xyz.keystore.jks 67257528 -rw-r--r--. 1 cloudera-scm cloudera-scm 2775 Jul 9 23:51 cm-r01en01.mws.mds.xyz.keystore.p12 67586231 -rw-r--r--. 1 cloudera-scm cloudera-scm 1720 Jul 9 23:52 cm-r01en01.mws.mds.xyz.cert.pem 71202518 -rw-r--r--. 1 cloudera-scm cloudera-scm 1863 Jul 9 23:53 cm-r01en01.mws.mds.xyz.key.pem 71305735 -r--r-----. 1 cloudera-scm cloudera-scm 23 Jul 9 23:53 client-agent.pw 67257529 lrwxrwxrwx. 1 root root 30 Jul 9 23:55 client-key.pem -> cm-r01en01.mws.mds.xyz.key.pem 71202519 lrwxrwxrwx. 1 root root 31 Jul 9 23:55 client-cert.pem -> cm-r01en01.mws.mds.xyz.cert.pem 71305736 -rw-r--r--. 1 cloudera-scm cloudera-scm 1432 Jul 10 20:27 cm-r01en01.mws.mds.xyz.pem 71305755 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Jul 10 22:14 cm-r01nn01.mws.mds.xyz.pem 69943176 lrwxrwxrwx. 1 root root 53 Jul 10 22:14 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem 69943151 drwxr-xr-x. 2 root root 4096 Jul 11 23:01 . [root@cm-r01en01 pki]# hostname cm-r01en01.mws.mds.xyz [root@cm-r01en01 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep -Ei cm Enter keystore password: changeit acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry, cm-r01nn02.mws.mds.xyz, Apr 14, 2019, trustedCertEntry, cm-r01en01.mws.mds.xyz, Jul 11, 2019, trustedCertEntry, cm-r01nn01.mws.mds.xyz, Apr 14, 2019, trustedCertEntry, [root@cm-r01en01 pki]# [root@cm-r01en01 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -Eiv "#"|sed -e "/^$/d" [General] server_host=cm-r01nn01.mws.mds.xyz server_port=7182 max_collection_wait_seconds=10.0 metrics_url_timeout_seconds=30.0 task_metrics_timeout_seconds=5.0 monitored_nodev_filesystem_types=nfs,nfs4,tmpfs local_filesystem_whitelist=ext2,ext3,ext4,xfs impala_profile_bundle_max_bytes=1073741824 stacks_log_bundle_max_bytes=1073741824 stacks_log_max_uncompressed_file_size_bytes=5242880 orphan_process_dir_staleness_threshold=5184000 orphan_process_dir_refresh_interval=3600 scm_debug=logging.DEBUG dns_resolution_collection_interval_seconds=60 dns_resolution_collection_timeout_seconds=30 [Security] use_tls=1 max_cert_depth=9 verify_cert_file=/opt/cloudera/security/pki/agent.pem client_key_file=/opt/cloudera/security/pki/client-key.pem client_keypw_file=/opt/cloudera/security/pki/client-agent.pw client_cert_file=/opt/cloudera/security/pki/client-cert.pem [Hadoop] [Cloudera] [JDBC] [Cgroup_Paths] [root@cm-r01en01 pki]# [ cm-r01en02 ] ( Utility Server ) [root@cm-r01en02 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep -Ei cm Enter keystore password: changeit acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry, cm-r01nn02.mws.mds.xyz, Jul 10, 2019, trustedCertEntry, cm-r01en02.mws.mds.xyz, Jul 10, 2019, trustedCertEntry, cm-r01nn01.mws.mds.xyz, Jul 10, 2019, trustedCertEntry, You have new mail in /var/spool/mail/root [root@cm-r01en02 pki]# ls -altri total 28 135616270 drwxr-xr-x. 5 root root 37 Jul 10 21:28 .. 335605256 -rw-r--r--. 1 cloudera-scm cloudera-scm 2386 Jul 10 21:29 cm-r01en02.mws.mds.xyz.keystore.jks 335605249 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Jul 10 21:51 cm-r01en02.mws.mds.xyz.pem 335605265 lrwxrwxrwx. 1 root root 62 Jul 10 21:56 server.jks -> /opt/cloudera/security/pki/cm-r01en02.mws.mds.xyz.keystore.jks 335605275 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Jul 10 22:14 cm-r01nn01.mws.mds.xyz.pem 335605382 lrwxrwxrwx. 1 root root 53 Jul 10 22:14 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem 335605420 -rw-r--r--. 1 cloudera-scm cloudera-scm 2775 Jul 11 23:09 cm-r01en02.mws.mds.xyz.keystore.p12 335605426 -rw-r--r--. 1 cloudera-scm cloudera-scm 1720 Jul 11 23:10 cm-r01en02.mws.mds.xyz.cert.pem 335605425 -rw-r--r--. 1 cloudera-scm cloudera-scm 1863 Jul 11 23:11 cm-r01en02.mws.mds.xyz.key.pem 335605429 lrwxrwxrwx. 1 root root 30 Jul 11 23:12 client-key.pem -> cm-r01en02.mws.mds.xyz.key.pem 335605430 lrwxrwxrwx. 1 root root 31 Jul 11 23:12 client-cert.pem -> cm-r01en02.mws.mds.xyz.cert.pem 335860926 drwxr-xr-x. 2 root root 4096 Jul 11 23:12 . [root@cm-r01en02 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -Eiv "#"|sed -e "/^$/d" [General] server_host=cm-r01nn01.mws.mds.xyz server_port=7182 max_collection_wait_seconds=10.0 metrics_url_timeout_seconds=30.0 task_metrics_timeout_seconds=5.0 monitored_nodev_filesystem_types=nfs,nfs4,tmpfs local_filesystem_whitelist=ext2,ext3,ext4,xfs impala_profile_bundle_max_bytes=1073741824 stacks_log_bundle_max_bytes=1073741824 stacks_log_max_uncompressed_file_size_bytes=5242880 orphan_process_dir_staleness_threshold=5184000 orphan_process_dir_refresh_interval=3600 scm_debug=logging.DEBUG dns_resolution_collection_interval_seconds=60 dns_resolution_collection_timeout_seconds=30 [Security] use_tls=1 max_cert_depth=9 verify_cert_file=/opt/cloudera/security/pki/agent.pem client_key_file=/opt/cloudera/security/pki/client-key.pem client_keypw_file=/opt/cloudera/security/pki/agent.pw client_cert_file=/opt/cloudera/security/pki/client-cert.pem [Hadoop] [Cloudera] [JDBC] [Cgroup_Paths] You have new mail in /var/spool/mail/root [root@cm-r01en02 pki]# [ cm-r01nn01 ] (Name Node) [root@cm-r01nn01 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep -Ei cm Enter keystore password: changeit acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry, cm-r01nn02.mws.mds.xyz, Apr 14, 2019, trustedCertEntry, cm-c01.mws.mds.xyz, Jul 6, 2019, trustedCertEntry, cm-r01nn01.mws.mds.xyz, Mar 31, 2019, trustedCertEntry, [root@cm-r01nn01 pki]# [root@cm-r01nn01 pki]# [root@cm-r01nn01 pki]# cd /opt/cloudera/security/pki [root@cm-r01nn01 pki]# ls -altri total 32 201424378 drwxr-xr-x. 5 root root 37 Mar 31 23:42 .. 135962833 lrwxrwxrwx. 1 root root 62 Mar 31 23:45 server.jks -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.keystore.jks 135962832 -rw-r--r--. 1 cloudera-scm cloudera-scm 4723 Jul 6 00:00 cm-r01nn01.mws.mds.xyz.keystore.jks 135283830 -rw-r--r--. 1 root root 1435 Jul 6 00:10 cm-c01.mws.mds.xyz.pem 135044185 -rw-r--r--. 1 root root 2751 Jul 8 03:49 cm-r01nn01.mws.mds.xyz.keystore.p12 135044187 -rw-r--r--. 1 root root 1691 Jul 8 03:50 cm-r01nn01.mws.mds.xyz.cert.pem 135044188 -rw-r--r--. 1 root root 1859 Jul 8 03:50 cm-r01nn01.mws.mds.xyz.key.pem 135962830 -rw-r--r--. 1 root root 1453 Jul 10 22:14 cm-r01nn01.mws.mds.xyz.pem 135962831 lrwxrwxrwx. 1 root root 53 Jul 10 22:14 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem 135962829 drwxr-xr-x. 2 root root 4096 Jul 10 22:14 . [root@cm-r01nn01 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -Eiv "#"|sed -e "/^$/d" [General] server_host=cm-r01nn01.mws.mds.xyz server_port=7182 max_collection_wait_seconds=10.0 metrics_url_timeout_seconds=30.0 task_metrics_timeout_seconds=5.0 monitored_nodev_filesystem_types=nfs,nfs4,tmpfs local_filesystem_whitelist=ext2,ext3,ext4,xfs impala_profile_bundle_max_bytes=1073741824 stacks_log_bundle_max_bytes=1073741824 stacks_log_max_uncompressed_file_size_bytes=5242880 orphan_process_dir_staleness_threshold=5184000 orphan_process_dir_refresh_interval=3600 scm_debug=logging.DEBUG dns_resolution_collection_interval_seconds=60 dns_resolution_collection_timeout_seconds=30 [Security] use_tls=1 max_cert_depth=9 verify_cert_file=/opt/cloudera/security/pki/agent.pem [Hadoop] [Cloudera] [JDBC] [Cgroup_Paths] [root@cm-r01nn01 pki]# NOTE: I haven't reconfigured the /etc/cloudera-scm-agent/config.ini yet on this (NN) node since the two other nodes (EN) aren't working anyway. Thx, TK

Online	Offline
Last Visited	‎05-01-2021 09:03 PM

Member Since	‎01-27-2019 08:43 PM
Last Visited	‎05-01-2021 09:03 PM
Posts	52

Cloudera Community

Re: Getting java.lang.ClassNotFoundException: org....

Re: Getting java.lang.ClassNotFoundException: org....

Getting java.lang.ClassNotFoundException: org.clou...

Re: SSLError: certificate verify failed

Re: SSLError: certificate verify failed

Re: SSLError: certificate verify failed

Re: SSLError: certificate verify failed

Re: SSLError: certificate verify failed

Re: SSLError: certificate verify failed

Re: SSLError: certificate verify failed