I'm receiving the following message from my CDH 6.2 installation.  I do see some older posts explaining the issue however as they are old posts, I don't believe they are relevent to this version.   Would anyone know the specifics of what this check does or looks for on RHEL 7 hosts that's specific to Cloudera versions?   Mismatched CDH versions: host has NONE but role expects 6     Cheers, TK ... View more

Hey All,   Getting the following on a brand new 6.2 installation and wondering what is the 'proper' fix for this.   I can start symlinking to get things working but perhaps I haven't deployed a package or taken another step I should have taken earlier?       + /usr/java/latest/bin/java -Xms52428800 -Xmx52428800 -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/oozie_oozie-OOZIE_SERVER-901d5713a53510380392378fa81b483d_pid18817.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh -Doozie.home.dir=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie -Doozie.config.dir=/run/cloudera-scm-agent/process/152-oozie-OOZIE-SERVER-upload-sharelib -Doozie.log.dir=/var/log/oozie -Doozie.log.file=oozie-cmf-oozie-OOZIE_SERVER-cm-r01en01.mws.mds.xyz.log.out -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=cm-r01en01.mws.mds.xyz -Doozie.http.port=11000 -Djava.net.preferIPv4Stack=true -Doozie.admin.port= -Dderby.stream.error.file=/var/log/oozie/derby.log -Doozie.instance.id=cm-r01en01.mws.mds.xyz -Djava.library.path=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/native -cp ':/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/accessors-smart-1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/activation-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/aggdesigner-algorithm-6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/ant-1.6.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/antlr-2.7.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/antlr-runtime-3.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/aopalliance-1.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/aopalliance-repackaged-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/apache-jsp-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/apache-jstl-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/apache-log4j-extras-1.2.17.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/asm-5.0.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/asm-commons-6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/asm-tree-6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/audience-annotations-0.5.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/automaton-1.11-8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/avro.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-anim-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-awt-util-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-bridge-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-codec-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-constants-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-css-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-dom-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-ext-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-gvt-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-i18n-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-parser-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-rasterizer-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-script-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-svg-dom-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-svggen-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-svgrasterizer-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-transcoder-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-util-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-xml-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/bcpkix-jdk15on-1.60.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/bcprov-jdk15on-1.60.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/bonecp-0.8.0.RELEASE.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/c3p0-0.9.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-beanutils-1.9.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-cli-1.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-codec-1.9.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-collections-3.2.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-compress-1.4.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-configuration2-2.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-crypto-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-daemon-1.0.13.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-dbcp-1.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-el-1.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-exec-1.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-httpclient-3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-io-2.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-lang-2.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-lang3-3.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-logging-1.1.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-math3-3.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-net-3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-pool-1.5.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/curator-client-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/curator-framework-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/curator-recipes-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/curator-x-discovery-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/datanucleus-api-jdo-4.2.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/datanucleus-core-4.1.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/datanucleus-rdbms-4.1.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/disruptor-3.3.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/dozer-5.5.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/ecj-4.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/ehcache-3.3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/ehcache-core-2.6.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/fastutil-6.5.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/findbugs-annotations-1.3.9-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/fst-2.50.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/geronimo-jms_1.1_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/geronimo-jpa_2.0_spec-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/geronimo-jta_1.1_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/gmetric4j-1.0.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/graphviz-java-0.7.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/gson-2.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/guava-11.0.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/guice-3.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/guice-assistedinject-3.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/guice-servlet-4.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-auth.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-client-3.0.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-hdfs-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-hdfs.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-mapreduce-client-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-mapreduce-client-core.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-mapreduce-client-jobclient.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-api.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-registry.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-server-applicationhistoryservice.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-server-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-server-resourcemanager.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-server-web-proxy.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-client-2.1.0-cdh6.2.0-tests.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-hadoop2-compat.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-hadoop-compat.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-http.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-mapreduce.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-metrics-api.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-metrics.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-procedure.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-protocol.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-protocol-shaded.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-replication.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-server.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-shaded-miscellaneous.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-shaded-netty.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-shaded-protobuf.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-zookeeper.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/HikariCP-2.6.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/HikariCP-java7-2.4.12.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-classification.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-hcatalog-pig-adapter.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-jdbc.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-llap-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-llap-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-llap-server.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-llap-tez.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-metastore.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-serde.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-service.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-service-rpc.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-shims-0.23.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-shims-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-shims.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-shims-scheduler.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hk2-api-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hk2-locator-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hk2-utils-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hsqldb-1.8.0.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/htrace-core4-4.1.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/httpclient-4.5.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/httpcore-4.4.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/j2v8_linux_x86_64-4.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/j2v8_macosx_x86_64-4.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/j2v8_win32_x86-4.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/j2v8_win32_x86_64-4.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-annotations-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-core-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-core-asl-1.9.13.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-databind-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-jaxrs-1.9.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-jaxrs-base-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-jaxrs-json-provider-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-mapper-asl-1.9.13.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-module-jaxb-annotations-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-xc-1.9.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jamon-runtime-2.3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jansi-1.9.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jasper-compiler-5.5.23.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jasper-runtime-5.5.23.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/JavaEWAH-1.1.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javaparser-1.0.11.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javassist-3.20.0-GA.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/java-util-1.9.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.annotation-api-1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.el-3.0.1-b11.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.inject-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.inject-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.jdo-3.2.0-m3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.servlet-api-3.1.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.servlet.jsp-2.3.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.servlet.jsp-api-2.3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.ws.rs-api-2.0.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javolution-5.5.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb2-basics-1.11.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb2-basics-runtime-1.11.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb2-basics-tools-1.11.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb-api-2.2.11.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb-impl-2.2.3-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jcip-annotations-1.0-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jcl-over-slf4j-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jcodings-1.0.18.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jcommander-1.30.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jdo-api-3.0.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jdom-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-client-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-client-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-common-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-container-servlet-core-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-core-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-guava-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-guice-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-json-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-media-jaxb-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-server-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-server-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-servlet-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jettison-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-annotations-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-http-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-io-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-jaas-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-jndi-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-plus-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-runner-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-schemas-3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-security-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-server-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-servlet-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-util-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-util-ajax-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-webapp-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-xml-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jline-2.12.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jms-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/joda-time-2.9.9.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/joni-2.1.11.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jpam-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jsch-0.1.54.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/json-io-2.5.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/json-simple-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/json-smart-2.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jsp-api-2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jsr305-3.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jsr311-api-1.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jta-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jul-to-slf4j-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jython-standalone-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jzlib-1.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-admin-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-client-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-common-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-core-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-crypto-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-identity-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-server-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-simplekdc-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-util-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-asn1-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-config-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-pkix-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-util-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-xdr-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kryo-2.22.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/leveldbjni-all-1.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/libfb303-0.9.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/libthrift-0.9.3-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/log4j-1.2.17.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/log4j-api-2.8.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/log4j-slf4j-impl-2.8.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/mail-1.4.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-core-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-ganglia-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-graphite-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-json-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-jvm-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/mssql-jdbc-6.2.1.jre7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/nashorn-promise-0.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/netty-3.10.6.Final.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/netty-all-4.1.17.Final.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/nimbus-jose-jwt-4.41.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/objenesis-1.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/okhttp-2.7.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/okio-1.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-client-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-core-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-core.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-fluent-job-api-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-fluent-job-api.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-sharelib-hcatalog-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-sharelib-hcatalog.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-sharelib-oozie-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-sharelib-oozie.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-tools-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-tools.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/opencsv-2.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-jdbc-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-kernel-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-lib-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-persistence-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-persistence-jdbc-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/org.eclipse.jgit-5.0.1.201806211838-r.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/osgi-resource-locator-1.0.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/paranamer-2.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/pig.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/postgresql-9.0-801.jdbc4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/protobuf-java-2.5.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/quartz-2.1.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/re2j-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/serializer-2.7.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/serp-1.15.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/servlet-api-2.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/slf4j-api-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/slf4j-log4j12-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/slf4j-simple-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/slider-core-0.90.2-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/snappy-java-1.1.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/stax2-api-3.1.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/stringtemplate-3.2.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/taglibs-standard-impl-1.2.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/taglibs-standard-spec-1.2.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/tephra-api-0.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/tephra-core-0.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/tephra-hbase-compat-1.0-0.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/transaction-api-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-api-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-common-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-core-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-discovery-api-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-discovery-core-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-zookeeper-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/validation-api-1.1.0.Final.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-api-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-client-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-common-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-server-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-servlet-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/woodstox-core-5.0.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xalan-2.7.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xbean-asm5-shaded-3.17.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xercesImpl-2.11.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xml-apis-1.4.01.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xml-apis-ext-1.3.04.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xmlenc-0.52.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xmlgraphics-commons-2.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xz-1.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/zookeeper.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libext/*.jar' org.apache.oozie.tools.OozieSharelibCLI create -fs hdfs://cm-r01nn02.mws.mds.xyz:8020 -locallib /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/oozie-sharelib-yarn -concurrency 8 log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.apache.oozie.service.XLogService.init(XLogService.java:149) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactorForRootLogger". log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.apache.oozie.service.XLogService.init(XLogService.java:149) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactor2". log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.apache.oozie.service.XLogService.init(XLogService.java:149) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactor1". SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console... log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395) at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:955) at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89) at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58) at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:947) at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:473) at org.apache.oozie.service.XLogService.init(XLogService.java:178) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactorForRootLogger". log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395) at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:955) at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89) at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58) at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:947) at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:473) at org.apache.oozie.service.XLogService.init(XLogService.java:178) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactor2". log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender]. java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395) at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:955) at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89) at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58) at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:947) at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:473) at org.apache.oozie.service.XLogService.init(XLogService.java:178) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.<init>(Services.java:111) at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194) at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97) log4j:ERROR Could not instantiate appender named "redactor1". [14/Aug/2019 00:35:33 +0000] 18806 MainThread redactor INFO Killing with SIGTERM       Cheers, TK ... View more

How do I enable further debugging on cloudera-scm-agents?   I'm working on deploying the cluster using self signed certificates but I'm running into the below issue and can't get past it:   [07/Jul/2019 23:35:05 +0000] 23766 MainThread agent ERROR Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed. Traceback (most recent call last): File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat self.cfg.max_cert_depth) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__ self.conn.connect() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect sock.connect((self.host, self.port)) File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect ret = self.connect_ssl() File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl return m2.ssl_connect(self.ssl, self._timeout) SSLError: certificate verify failed What I have in my certificates folder is the following:   [root@cm-r01en01 pki]# pwd /opt/cloudera/security/pki [root@cm-r01en01 pki]# ls -atlri total 16 69943167 -rw-r--r--. 1 root root 2385 Apr 1 23:06 cm-r01en01.mws.mds.xyz.keystore.jks 69943152 -rw-r--r--. 1 root root 1453 Apr 1 23:07 cm-r01en01.mws.mds.xyz.pem 3870062 drwxr-xr-x. 5 root root 37 Apr 1 23:09 .. 69943169 lrwxrwxrwx. 1 root root 62 Apr 1 23:11 server.jks -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.keystore.jks 69943259 -rw-r--r--. 1 root root 1453 Jul 6 20:01 cm-r01nn01.mws.mds.xyz.pem 69943154 lrwxrwxrwx. 1 root root 53 Jul 6 20:02 rootca.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem 67689060 lrwxrwxrwx. 1 root root 53 Jul 6 20:36 agent.pem -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.pem 69943151 drwxr-xr-x. 2 root root 4096 Jul 6 20:36 . [root@cm-r01en01 pki]# I'm not 100% sure if I have everything right though.  My cloudera-scm-agent config for that one host:   [root@cm-r01en01 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -v "#" | sed -e "/^$/d" [General] server_host=cm-r01nn01.mws.mds.xyz server_port=7182 max_collection_wait_seconds=10.0 metrics_url_timeout_seconds=30.0 task_metrics_timeout_seconds=5.0 monitored_nodev_filesystem_types=nfs,nfs4,tmpfs local_filesystem_whitelist=ext2,ext3,ext4,xfs impala_profile_bundle_max_bytes=1073741824 stacks_log_bundle_max_bytes=1073741824 stacks_log_max_uncompressed_file_size_bytes=5242880 orphan_process_dir_staleness_threshold=5184000 orphan_process_dir_refresh_interval=3600 scm_debug=DEBUG dns_resolution_collection_interval_seconds=60 dns_resolution_collection_timeout_seconds=30 [Security] use_tls=1 max_cert_depth=9 verify_cert_file=/opt/cloudera/security/pki/agent.pem verify_cert_dir=/opt/cloudera/security/pki/ [Hadoop] [Cloudera] [JDBC] [Cgroup_Paths] [root@cm-r01en01 pki]# cm-r01nn01 is the Name Node. cm -r01en01 will be the gateway  / entry point to the cluster.  It will also run a few services.     This is CM 6.2 .  I'm looking to go through the certificate process in preparation for a more formal deployment later on w/ official certificates.  Using self signed certs for now for this POC.   In particular, what certificate has it tried to load and is looking for?  How do I enable further debug logs to see all the calls it's making and files it's loading?   Cheers, TK ... View more

Hey All,   Receiving the following when trying to install services on a brand new cluster:   2019-06-24 23:42:18,402 INFO scm-web-118667:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar 2019-06-24 23:42:18,404 INFO scm-web-119745:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/statusContent 2019-06-24 23:42:18,412 INFO scm-web-118667:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar, Status:200 2019-06-24 23:42:18,420 INFO scm-web-119745:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/statusContent, Status:200 2019-06-24 23:42:19,481 INFO scm-web-119111:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/testConnection 2019-06-24 23:42:19,489 INFO scm-web-119111:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HiveTestDatabaseConnection with sensitive arguments. Host: DbHost{id=28, hostId=a787204f-931d-47e0-8aef-534ab6355a7a, hostName=cm-r01en01.mws.mds.xyz} 2019-06-24 23:42:19,500 INFO scm-web-119111:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command OozieTestDatabaseConnection with sensitive arguments. Host: DbHost{id=28, hostId=a787204f-931d-47e0-8aef-534ab6355a7a, hostName=cm-r01en01.mws.mds.xyz} 2019-06-24 23:42:19,506 INFO scm-web-119111:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing host command HueTestDatabaseConnection with sensitive arguments. Host: DbHost{id=28, hostId=a787204f-931d-47e0-8aef-534ab6355a7a, hostName=cm-r01en01.mws.mds.xyz} 2019-06-24 23:42:19,809 INFO scm-web-119111:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/testConnection, Status:200 2019-06-24 23:42:19,826 INFO avro-servlet-hb-processor-1:com.cloudera.server.common.AgentAvroServlet: (45 skipped) AgentAvroServlet: heartbeat processing stats: average=12ms, min=8ms, max=79ms. 2019-06-24 23:42:21,847 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult 2019-06-24 23:42:21,854 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200 2019-06-24 23:42:23,890 INFO scm-web-120477:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult 2019-06-24 23:42:23,896 INFO scm-web-120477:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200 2019-06-24 23:42:24,373 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=242 name=null host=a787204f-931d-47e0-8aef-534ab6355a7a/cm-r01en01.mws.mds.xyz 2019-06-24 23:42:24,374 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=243 name=null host=a787204f-931d-47e0-8aef-534ab6355a7a/cm-r01en01.mws.mds.xyz 2019-06-24 23:42:24,374 WARN avro-servlet-hb-processor-0:com.cloudera.server.cmf.AgentProtocolImpl: Received Process Heartbeat for unknown (or duplicate) process. Ignoring. This is expected to happen once after old process eviction or process deletion (as happens in restarts). id=244 name=null host=a787204f-931d-47e0-8aef-534ab6355a7a/cm-r01en01.mws.mds.xyz 2019-06-24 23:42:25,935 INFO scm-web-119807:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult 2019-06-24 23:42:25,942 INFO scm-web-119807:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200 2019-06-24 23:42:28,049 INFO scm-web-119778:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult 2019-06-24 23:42:28,056 INFO scm-web-119778:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200 2019-06-24 23:42:29,383 INFO scm-web-120247:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar 2019-06-24 23:42:29,394 INFO scm-web-120247:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar, Status:200 2019-06-24 23:42:29,395 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/statusContent 2019-06-24 23:42:29,422 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/statusContent, Status:200 2019-06-24 23:42:30,084 INFO scm-web-118667:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult 2019-06-24 23:42:30,091 INFO scm-web-118667:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200 2019-06-24 23:42:32,124 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult 2019-06-24 23:42:32,131 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200 2019-06-24 23:42:33,205 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'HueTestDatabaseConnection' 2019-06-24 23:42:33,207 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 1 2019-06-24 23:42:33,207 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: [24/Jun/2019 23:42:28 +0000] 30846 MainThread redactor INFO Started launcher: /opt/cloudera/cm-agent/service/hue/hue.sh is_db_alive [24/Jun/2019 23:42:28 +0000] 30846 MainThread redactor ERROR Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/247-HUE-test-db-connection [24/Jun/2019 23:42:28 +0000] 30846 MainThread redactor INFO Re-exec watcher: /opt/cloudera/cm-agent/bin/cm proc_watcher 30867 + date + date Mon Jun 24 23:42:30 EDT 2019 ++ dirname /opt/cloudera/cm-agent/service/hue/hue.sh + cloudera_config=/opt/cloudera/cm-agent/service/hue ++ cd /opt/cloudera/cm-agent/service/hue/../common ++ pwd + cloudera_config=/opt/cloudera/cm-agent/service/common + . /opt/cloudera/cm-agent/service/common/cloudera-config.sh ++ : /opt/cloudera/cm ++ export CLOUDERA_DIR ++ set -x + source_parcel_environment + '[' '!' -z '' ']' + locate_cdh_java_home + '[' -z '' ']' + '[' -z /usr/libexec/bigtop-utils ']' + local BIGTOP_DETECT_JAVAHOME= + for candidate in '"${JSVC_HOME}"' '"${JSVC_HOME}/.."' '"/usr/lib/bigtop-utils"' '"/usr/libexec"' + '[' -e /usr/libexec/bigtop-utils/bigtop-detect-javahome ']' + for candidate in '"${JSVC_HOME}"' '"${JSVC_HOME}/.."' '"/usr/lib/bigtop-utils"' '"/usr/libexec"' + '[' -e /usr/libexec/bigtop-utils/../bigtop-detect-javahome ']' + for candidate in '"${JSVC_HOME}"' '"${JSVC_HOME}/.."' '"/usr/lib/bigtop-utils"' '"/usr/libexec"' + '[' -e /usr/lib/bigtop-utils/bigtop-detect-javahome ']' + for candidate in '"${JSVC_HOME}"' '"${JSVC_HOME}/.."' '"/usr/lib/bigtop-utils"' '"/usr/libexec"' + '[' -e /usr/libexec/bigtop-detect-javahome ']' + '[' -z '' ']' + echo 'Cannot find CDH'\''s bigtop-detect-javahome.' + exit 1 2019-06-24 23:42:33,207 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 742(HueTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Unexpected error. Unable to verify database connection. 2019-06-24 23:42:34,168 INFO scm-web-119807:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult 2019-06-24 23:42:34,175 INFO scm-web-119807:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200 2019-06-24 23:42:36,228 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult 2019-06-24 23:42:36,234 INFO scm-web-120167:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200 2019-06-24 23:42:37,481 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'OozieTestDatabaseConnection' 2019-06-24 23:42:37,483 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 0 2019-06-24 23:42:37,483 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: + local MISCJAVA_HOME_CANDIDATES + case ${BIGTOP_JAVA_MAJOR} in + JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]}) + '[' -z '' ']' + for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}' ++ ls -rvd /usr/java/jdk1.8.0_181-cloudera + for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`' + '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']' + export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera + JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera + break 2 + verify_java_home + '[' -z /usr/java/jdk1.8.0_181-cloudera ']' + echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera + source_parcel_environment + '[' '!' -z '' ']' + JAVA=/usr/java/jdk1.8.0_181-cloudera/bin/java + [[ -z /opt/cloudera/cm ]] + MGMT_CLASSPATH='/opt/cloudera/cm/lib/*' + JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar + MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*' ++ pwd + MGMT_CLASSPATH='/run/cloudera-scm-agent/process/246-OOZIE.OOZIE_SERVER-test-db-connection:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*' + MGMT_JAVA_OPTS='-Djava.net.preferIPv4Stack=true ' + exec /usr/java/jdk1.8.0_181-cloudera/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp '/run/cloudera-scm-agent/process/246-OOZIE.OOZIE_SERVER-test-db-connection:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*' com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties 2019-06-24 23:42:37,483 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 741(OozieTestDatabaseConnection) has completed. finalstate:FINISHED, success:true, msg:Successful 2019-06-24 23:42:37,497 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Successful 'HiveTestDatabaseConnection' 2019-06-24 23:42:37,498 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 0 2019-06-24 23:42:37,498 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: + MISCJAVA_HOME_CANDIDATES=('/Library/Java/Home' '/usr/java/default' '/usr/lib/jvm/default-java' '/usr/lib/jvm/java-openjdk' '/usr/lib/jvm/jre-openjdk') + local MISCJAVA_HOME_CANDIDATES + case ${BIGTOP_JAVA_MAJOR} in + JAVA_HOME_CANDIDATES=(${JAVA8_HOME_CANDIDATES[@]} ${MISCJAVA_HOME_CANDIDATES[@]} ${OPENJAVA8_HOME_CANDIDATES[@]} ${JAVA11_HOME_CANDIDATES[@]} ${OPENJAVA11_HOME_CANDIDATES[@]}) + '[' -z '' ']' + for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}' ++ ls -rvd /usr/java/jdk1.8.0_181-cloudera + for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`' + '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']' + export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera + JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera + break 2 + verify_java_home + '[' -z /usr/java/jdk1.8.0_181-cloudera ']' + echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera + source_parcel_environment + '[' '!' -z '' ']' + JAVA=/usr/java/jdk1.8.0_181-cloudera/bin/java + [[ -z /opt/cloudera/cm ]] + MGMT_CLASSPATH='/opt/cloudera/cm/lib/*' + JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar + MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*' ++ pwd + MGMT_CLASSPATH='/run/cloudera-scm-agent/process/245-HIVE-test-db-connection:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*' + MGMT_JAVA_OPTS='-Djava.net.preferIPv4Stack=true ' + exec /usr/java/jdk1.8.0_181-cloudera/bin/java -Djava.net.preferIPv4Stack=true -Djava.security.egd=file:///dev/urandom -cp '/run/cloudera-scm-agent/process/245-HIVE-test-db-connection:/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*' com.cloudera.enterprise.dbutil.DbCommandExecutor db.properties 2019-06-24 23:42:37,498 INFO CommandPusher:com.cloudera.cmf.model.DbCommand: Command 740(HiveTestDatabaseConnection) has completed. finalstate:FINISHED, success:true, msg:Successful 2019-06-24 23:42:38,336 INFO scm-web-118667:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult 2019-06-24 23:42:38,351 INFO scm-web-118667:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/dbTestConn/checkConnectionResult, Status:200 2019-06-24 23:42:40,393 INFO scm-web-120237:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar 2019-06-24 23:42:40,418 INFO scm-web-119778:com.cloudera.enterprise.JavaMelodyFacade: Entering HTTP Operation: Method:POST, Path:/clusters/2/statusContent 2019-06-24 23:42:40,421 INFO scm-web-120237:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/healthStatusBar, Status:200 2019-06-24 23:42:40,435 INFO scm-web-119778:com.cloudera.enterprise.JavaMelodyFacade: Exiting HTTP Operation: Method:POST, Path:/clusters/2/statusContent, Status:200 ^C [root@cm-r01nn01 ~]# sudo su - cloudera-scm This account is currently not available. [root@cm-r01nn01 ~]# /opt/cloudera/cm-agent/service/hue/hue.sh is_db_alive ^C [root@cm-r01nn01 ~]# ls -altri /opt/cloudera/cm-agent/service/hue/hue.sh is_db_alive \ls: cannot access is_db_alive: No such file or directory 135962786 -rwxr-xr-x. 1 cloudera-scm cloudera-scm 9658 Mar 14 12:58 /opt/cloudera/cm-agent/service/hue/hue.sh [root@cm-r01nn01 ~]# \ > [root@cm-r01nn01 ~]#   This error comes up during the cluster installation. UI shows:   Unexpected error. Unable to verify database connection.   During the:   Add Cluster - Configuration - Setup Databases step.   Has anyone encountered the same issue and could you let me know please how did you solve it, if you did? This occurs only for the HUE database verification.   Thx, TK ... View more

Hey All,   I'm installing a vanilla CM / CDH 6.2.X and running into the issue below from the Navigator Metadata Server:   10:45:54.757 PM INFO JsonModelLoader [main]: Added audit event model for SENTRY 10:45:54.761 PM INFO JsonModelLoader [main]: Added audit event model for SOLR 10:46:10.631 PM WARN SecurityHandler [main]: ServletContext@o.e.j.w.WebAppContext@241e8ea6{/solr,[file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-solr-4.10.3.war-_solr-any-4050545960767447337.dir/webinf/, file:///var/lib/cloudera-scm-navigator/temp/jetty-0.0.0.0-7187-solr-4.10.3.war-_solr-any-4050545960767447337.dir/webapp/],STARTING}{/opt/cloudera/cm/cloudera-navigator-server/wars/solr-4.10.3.war} has uncovered http methods for path: /* 10:46:12.171 PM WARN IndexSchema [coreLoadExecutor-5-thread-1]: Field user is not multivalued and destination for multiple copyFields (2) 10:46:14.883 PM INFO NavServerUtil [main]: Found 4 documents in solr core nav_elements 10:46:14.895 PM INFO NavServerUtil [main]: Found 1 documents in solr core nav_relations 10:46:14.896 PM INFO SolrSchemaUpgrade [main]: Checking if solr schema upgrade is needed. 10:46:14.928 PM INFO SolrSchemaUpgrade [main]: Schema version 2900 already up to date with latest schema 2900. 10:46:14.956 PM INFO SolrSchemaUpgrade [main]: Current database upgrade ordinal is -1, latest upgrade step has ordinal 28 10:46:15.024 PM INFO AbstractSolrUpgrade [main]: Starting upgrade step: com.cloudera.nav.server.upgrade.DeleteEntities : Delete temporary entities. 10:46:15.031 PM INFO NavServerUtil [main]: Found 4 documents in solr core nav_elements 10:46:15.039 PM INFO NavServerUtil [main]: Found 1 documents in solr core nav_relations 10:46:15.042 PM INFO DeleteEntities [main]: Delete temporary relations query: {!join from=identity to=endpoint1Ids fromIndex=nav_elements}fileSystemPath:(*\/_SUCCESS OR *\/.trash OR \/user\/history OR *\/.staging OR \/tmp\/logs) AND sourceType:HDFS 10:46:15.206 PM ERROR SolrCore [qtp1100399456-26]: org.apache.solr.common.SolrException: Cursor functionality requires a sort containing a uniqueKey field tie breaker at org.apache.solr.search.CursorMark.<init>(CursorMark.java:104) at org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:179) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:197) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602) at com.cloudera.nav.auth.SharedSecretRequestFilter.doFilter(SharedSecretRequestFilter.java:55) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:502) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.lang.Thread.run(Thread.java:748) 10:46:15.221 PM ERROR NavServer [main]: Error upgrading solr configuration or data. org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Cursor functionality requires a sort containing a uniqueKey field tie breaker at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:552) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206) at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:91) at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:310) at com.cloudera.nav.utils.solr.SolrResultSetIterator.getNextBatch(SolrResultSetIterator.java:96) at com.cloudera.nav.utils.solr.SolrResultSetIterator.hasNext(SolrResultSetIterator.java:76) at com.cloudera.nav.server.upgrade.DeleteEntities.deleteForSide(DeleteEntities.java:248) at com.cloudera.nav.server.upgrade.DeleteEntities.blindDelete(DeleteEntities.java:223) at com.cloudera.nav.server.upgrade.DeleteEntities.upgrade(DeleteEntities.java:76) at com.cloudera.nav.server.upgrade.AbstractSolrUpgrade.upgrade(AbstractSolrUpgrade.java:156) at com.cloudera.nav.server.SolrSchemaUpgrade.performJavaUpgrade(SolrSchemaUpgrade.java:400) at com.cloudera.nav.server.SolrSchemaUpgrade.upgradeSolr(SolrSchemaUpgrade.java:292) at com.cloudera.nav.server.NavServer.run(NavServer.java:434) at com.cloudera.nav.server.NavServer.main(NavServer.java:123) 10:46:15.329 PM INFO NavigatorInfrastructureEventDispatcher [pool-12-thread-1]: Waiting for Solr to Start. 10:46:15.331 PM INFO NavigatorInfrastructureEventDispatcher [pool-12-thread-1]: Succesfully started Solr. 10:46:15.332 PM INFO NavigatorInfrastructureEventDispatcher [pool-12-thread-1]: Waiting for Solr upgrade to complete. 10:46:15.489 PM ERROR NavServerUtil [pool-12-thread-1]: Error encountered in fetching the udate progress information.Please enable TRACE level logging for SQL exception details. 10:46:15.496 PM INFO NavServerUtil [pool-12-thread-1]: Exception occurred when checking for upgrade status. java.lang.RuntimeException: Error encountered in fetching the udate progress information. at com.cloudera.nav.persist.PersistUtils.handleDataAccessException(PersistUtils.java:88) at com.cloudera.nav.server.NavServerUtil.getUpgradeProgressInfo(NavServerUtil.java:281) at com.cloudera.nav.server.NavServerUtil.isUpgradeDone(NavServerUtil.java:231) at com.cloudera.nav.utils.UpgradeCompletionChecker.isUpgradeDone(UpgradeCompletionChecker.java:13) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.isUpgradeDone(NavigatorInfrastructureEventDispatcher.java:158) at com.cloudera.nav.server.SolrMaintenanceAware.waitForUpgrade(SolrMaintenanceAware.java:23) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.run(NavigatorInfrastructureEventDispatcher.java:153) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.dispatchSolrUpgradedEvent(NavigatorInfrastructureEventDispatcher.java:107) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.access$300(NavigatorInfrastructureEventDispatcher.java:35) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$3.run(NavigatorInfrastructureEventDispatcher.java:93) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 10:46:15.506 PM INFO SolrMaintenanceAware [pool-12-thread-1]: Checking for upgrade done status to be set. 10:46:15.514 PM ERROR NavServerUtil [pool-12-thread-1]: Error encountered in fetching the udate progress information.Please enable TRACE level logging for SQL exception details. 10:46:15.514 PM INFO NavServerUtil [pool-12-thread-1]: Exception occurred when checking for upgrade status. java.lang.RuntimeException: Error encountered in fetching the udate progress information. at com.cloudera.nav.persist.PersistUtils.handleDataAccessException(PersistUtils.java:88) at com.cloudera.nav.server.NavServerUtil.getUpgradeProgressInfo(NavServerUtil.java:281) at com.cloudera.nav.server.NavServerUtil.isUpgradeDone(NavServerUtil.java:231) at com.cloudera.nav.utils.UpgradeCompletionChecker.isUpgradeDone(UpgradeCompletionChecker.java:13) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.isUpgradeDone(NavigatorInfrastructureEventDispatcher.java:158) at com.cloudera.nav.server.SolrMaintenanceAware.pollForUpgradeDone(SolrMaintenanceAware.java:34) at com.cloudera.nav.server.SolrMaintenanceAware.waitForUpgrade(SolrMaintenanceAware.java:24) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$SolrUpgradePoller.run(NavigatorInfrastructureEventDispatcher.java:153) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.dispatchSolrUpgradedEvent(NavigatorInfrastructureEventDispatcher.java:107) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher.access$300(NavigatorInfrastructureEventDispatcher.java:35) at com.cloudera.nav.utils.NavigatorInfrastructureEventDispatcher$3.run(NavigatorInfrastructureEventDispatcher.java:93) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 10:46:15.516 PM INFO SolrMaintenanceAware [pool-12-thread-1]: Upgrade status not set or available, continue checking for upgrade status to be set. My backend is a PostgreSQL database.  Navigator Metastore Server keeps exiting and restarting as a result:   Unexpected Exits                                     Suppress... This role encountered 4 unexpected exit(s) in the previous 5 minute(s). Critical threshold: any.   How could I solve this?  The other posts and solutions I could find were specific to MySQL only.     Also only the servers are reporting and the Cloudera Management Service is installed.  Nothing else is installed on this Cluster yet.     Thx, TK ... View more

Hey All,   Running into:   Can't open /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission denied.  on a vanilla 6.2 installation.   namei output (from a non-namenode server):   [root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# namei -l /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/supervisor_status f: /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/supervisor_status dr-xr-xr-x root root / drwxr-xr-x root root var lrwxrwxrwx root root run -> ../run dr-xr-xr-x root root .. drwxr-xr-x root root run drwxr-x--x root root cloudera-scm-agent drwxr-x--x root root process drwxr-x--x cloudera-scm cloudera-scm 114-cloudera-mgmt-EVENTSERVER -rw------- root root supervisor_status [root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# When?   During installation of:   Add Cloudera Management Service Service   LOGS:   + for candidate_regex in '${JAVA_HOME_CANDIDATES[@]}' ++ ls -rvd /usr/java/jdk1.8.0_181-cloudera + for candidate in '`ls -rvd ${candidate_regex}* 2>/dev/null`' + '[' -e /usr/java/jdk1.8.0_181-cloudera/bin/java ']' + export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera + JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera + break 2 + verify_java_home + '[' -z /usr/java/jdk1.8.0_181-cloudera ']' + echo JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera + replace_conf_dir + echo CONF_DIR=/var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER + echo CMF_CONF_DIR= + EXCLUDE_CMF_FILES=('cloudera-config.sh' 'hue.sh' 'impala.sh' 'sqoop.sh' 'supervisor.conf' 'config.zip' 'proc.json' '*.log' '*.keytab' '*jceks') ++ printf '! -name %s ' cloudera-config.sh hue.sh impala.sh sqoop.sh supervisor.conf config.zip proc.json '*.log' '*.keytab' '*jceks' + find /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER -type f '!' -path '/var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER#g' '{}' ';' Can't open /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER/supervisor_status: Permission denied. + source_parcel_environment + '[' '!' -z '' ']' + '[' /usr/java/jdk1.8.0_181-cloudera '!=' '' ']' + JAVA=/usr/java/jdk1.8.0_181-cloudera/bin/java + [[ -n /opt/cloudera/cm ]] + MGMT_CLASSPATH='/opt/cloudera/cm/lib/*' + [[ -z /opt/cloudera/cm/lib/* ]] + JDBC_JARS=/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar + MGMT_CLASSPATH='/usr/share/java/mysql-connector-java.jar:/opt/cloudera/cm/lib/postgresql-42.1.4.jre7.jar:/usr/share/java/oracle-connector-java.jar:/opt/cloudera/cm/lib/*:' + DDL_DIR=/opt/cloudera/cm/schema + [[ false == \f\a\l\s\e ]] + set +x   Already added the hosts earlier via this CM 6.2 to a vanilla cluster.   Anyway to get around the above error?   Edit: Seems like supervisord creates the files as root but cloudera-scm-agent tries to read them as cloudera-scm user.  That won't work.     [root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# find /var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR -type f '!' -path '/var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name cmon.keytab '!' -name '*jceks' -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR#g' '{}' ';' [root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# echo $? 0 [root@cm-se02 114-cloudera-mgmt-EVENTSERVER]# sudo su - cloudera-scm Last login: Sun Mar 31 00:42:49 EDT 2019 on pts/0 -bash-4.2$ -bash-4.2$ -bash-4.2$ -bash-4.2$ find /var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR -type f '!' -path '/var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name cmon.keytab '!' -name '*jceks' -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR#g' '{}' ';' Can't open /var/run/cloudera-scm-agent/process/121-cloudera-mgmt-SERVICEMONITOR/supervisor_status: Permission denied. -bash-4.2$ who am i root pts/0 2019-04-14 11:06 (192.168.0.142) -bash-4.2$ whoami cloudera-scm -bash-4.2$   Additional analysis:   [root@cm-se02 cloudera-scm-agent]# sudo su - cloudera-scm Last login: Sun Apr 14 11:41:29 EDT 2019 on pts/0 -bash-4.2$ cd /var/run/cloudera-scm-agent/ -bash-4.2$ -bash-4.2$ ls -altri ls: cannot open directory .: Permission denied -bash-4.2$ cd process -bash-4.2$ pwd /var/run/cloudera-scm-agent/process -bash-4.2$ cd 114-cloudera-mgmt-EVENTSERVER -bash-4.2$ ls -altri supervisor supervisor.conf supervisor_status -bash-4.2$ ls -altri supervisor_status 714067 -rw-------. 1 root root 656 Apr 14 11:01 supervisor_status -bash-4.2$ cat supervisor_status cat: supervisor_status: Permission denied -bash-4.2$ pwd /var/run/cloudera-scm-agent/process/114-cloudera-mgmt-EVENTSERVER -bash-4.2$   Thx, TK ... View more

Hello,   I'm trying to understand any trouble I may run into if I use a shared KDC with multiple instances of Cloudera.   Let's say I build two distinct clusters and try to share a KDC and DNS between them.  Won't the principals conflict between the two clusters for things like the HDFS principal for example?  Will I run into any other issues?   What is a recommended approach to installing multiple clusters on the same Domain / KDC vs separate KDC's and separate Domains?   Regards, TC ... View more