Member since
01-27-2019
41
Posts
0
Kudos Received
0
Solutions
09-19-2019
05:50 PM
Not the best approach to getting rid of these messages but it gave me what I wanted. I set highest logging level to ERROR instead so everything else is not printed: tom@mds.xyz@cm-r01en01:~] 🙂 $ cat /etc/spark/conf/log4j.properties
log4j.rootLogger=${root.logger}
root.logger=ERROR,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
shell.log.level=ERROR
log4j.logger.org.spark-project.jetty=WARN
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=ERROR
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=ERROR
log4j.logger.org.apache.parquet=ERROR
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
log4j.logger.org.apache.spark.repl.Main=${shell.log.level}
log4j.logger.org.apache.spark.api.python.PythonGatewayServer=${shell.log.level}
tom@mds.xyz@cm-r01en01:~] 🙂 $
tom@mds.xyz@cm-r01en01:~] 🙂 $
tom@mds.xyz@cm-r01en01:~] 🙂 $ digg /etc/spark/conf/log4j.properties /etc/spark/conf/log4j.properties-original
-sh: digg: command not found
tom@mds.xyz@cm-r01en01:~] 😞 $ diff /etc/spark/conf/log4j.properties /etc/spark/conf/log4j.properties-original
2c2
< root.logger=ERROR,console
---
> root.logger=DEBUG,console
10,11c10,11
< log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=ERROR
< log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=ERROR
---
> log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
> log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
tom@mds.xyz@cm-r01en01:~] 😞 $ Now I get my spark-shell without the INFO, DEBUG or WARNING messages all over it. Still interested in a final solution if possible. I only see it fixed in Spark 3.0 . Cheers, TK
... View more
09-18-2019
04:49 AM
Including the log file of the shell session as a link: https://tinyurl.com/y2kfmke8 Cheers, TK
... View more
09-16-2019
10:20 PM
Moving back to the original question: "INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all" My configration is as follows: [root@cm-r01en01 conf]# cat log4j.properties
log4j.rootLogger=${root.logger}
root.logger=INFO,console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{2}: %m%n
shell.log.level=INFO
log4j.logger.org.spark-project.jetty=WARN
log4j.logger.org.spark-project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
log4j.logger.org.apache.parquet=ERROR
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
log4j.logger.org.apache.spark.repl.Main=${shell.log.level}
log4j.logger.org.apache.spark.api.python.PythonGatewayServer=${shell.log.level}
[root@cm-r01en01 conf]#
[root@cm-r01en01 conf]#
[root@cm-r01en01 conf]# cat spark-defaults.conf
spark.authenticate=false
spark.driver.log.dfsDir=/user/spark/driverLogs
spark.driver.log.persistToDfs.enabled=true
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.executorIdleTimeout=60
spark.dynamicAllocation.minExecutors=0
spark.dynamicAllocation.schedulerBacklogTimeout=1
spark.eventLog.enabled=true
spark.io.encryption.enabled=false
spark.network.crypto.enabled=false
spark.serializer=org.apache.spark.serializer.KryoSerializer
spark.shuffle.service.enabled=true
spark.shuffle.service.port=7337
spark.ui.enabled=true
spark.ui.killEnabled=true
spark.lineage.log.dir=/var/log/spark/lineage
spark.lineage.enabled=true
spark.master=yarn
spark.submit.deployMode=client
spark.eventLog.dir=hdfs://cm-r01nn02.mws.mds.xyz:8020/user/spark/applicationHistory
spark.yarn.historyServer.address=<a href="http://cm-r01en01.mws.mds.xyz:18088" target="_blank">http://cm-r01en01.mws.mds.xyz:18088</a>
spark.yarn.jars=local:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/spark/jars/*,local:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/spark/hive/*
spark.driver.extraLibraryPath=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/lib/native
spark.executor.extraLibraryPath=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/lib/native
spark.yarn.am.extraLibraryPath=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/lib/native
spark.yarn.config.gatewayPath=/opt/cloudera/parcels
spark.yarn.config.replacementPath={{HADOOP_COMMON_HOME}}/../../..
spark.yarn.historyServer.allowTracking=true
spark.yarn.appMasterEnv.MKL_NUM_THREADS=1
spark.executorEnv.MKL_NUM_THREADS=1
spark.yarn.appMasterEnv.OPENBLAS_NUM_THREADS=1
spark.executorEnv.OPENBLAS_NUM_THREADS=1
spark.extraListeners=com.cloudera.spark.lineage.NavigatorAppListener
spark.sql.queryExecutionListeners=com.cloudera.spark.lineage.NavigatorQueryListener
[root@cm-r01en01 conf]# I can get into the shell however the console is overwhelmed with the above error messages preventing me from doing anything useful with it. Aiming to run a few spark commands to get started learning it. Cheers, TK
... View more
09-15-2019
04:52 PM
I restarted things before capturing the second reply above to free space on the cluster. PID 13778 would be the same thing as the original PID 23190 I listed. The process always takes up 1.8+ GB. After a short while it climbs to 2.8GB.
... View more
09-12-2019
11:10 PM
Apologies. You're right. mysql connector running through the Cloudera user: [root@cm-r01nn01 ~]# top
top - 02:06:24 up 19 days, 4:04, 1 user, load average: 0.26, 0.28, 0.29
Tasks: 197 total, 1 running, 196 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.0 us, 0.5 sy, 0.0 ni, 98.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 8008640 total, 286564 free, 5060212 used, 2661864 buff/cache
KiB Swap: 4063228 total, 4063228 free, 0 used. 2205572 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3919 root 20 0 1260148 47188 8428 S 3.8 0.6 2:13.33 python2
13778 clouder+ 20 0 8263556 2.8g 17472 S 3.8 37.1 2208:20 java
3877 root 20 0 2680196 63480 9772 S 1.2 0.8 1:46.97 cmagent
4160 root 20 0 619212 28772 9028 S 1.2 0.4 0:32.63 python2
6281 httpfs 20 0 3860244 213008 29404 S 1.2 2.7 0:29.09 java
7367 hive 20 0 2118308 312892 41972 S 1.2 3.9 1:09.55 java
14318 root 20 0 172308 2404 1628 R 1.2 0.0 0:00.49 top
1 root 20 0 54952 7272 4268 S 0.0 0.1 6:44.58 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:01.41 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 1:27.50 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:04.12 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 29:36.83 rcu_sched
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:12.34 watchdog/0
12 root rt 0 0 0 0 S 0.0 0.0 0:08.09 watchdog/1
13 root rt 0 0 0 0 S 0.0 0.0 0:02.97 migration/1
14 root 20 0 0 0 0 S 0.0 0.0 1:21.07 ksoftirqd/1
16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
17 root rt 0 0 0 0 S 0.0 0.0 0:07.84 watchdog/2
18 root rt 0 0 0 0 S 0.0 0.0 0:00.64 migration/2
19 root 20 0 0 0 0 S 0.0 0.0 1:20.44 ksoftirqd/2
21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H
22 root rt 0 0 0 0 S 0.0 0.0 0:08.05 watchdog/3
23 root rt 0 0 0 0 S 0.0 0.0 0:00.39 migration/3
24 root 20 0 0 0 0 S 0.0 0.0 1:20.37 ksoftirqd/3
26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H
27 root rt 0 0 0 0 S 0.0 0.0 0:08.38 watchdog/4
28 root rt 0 0 0 0 S 0.0 0.0 0:00.51 migration/4
[root@cm-r01nn01 ~]# ps -ef|grep -Ei 13778
clouder+ 13778 1 8 Aug25 ? 1-12:48:21 /usr/java/jdk1.8.0_181-cloudera/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server -Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties -Dfile.encoding=UTF-8 -Dcmf.root.logger=INFO,LOGFILE -Dcmf.log.dir=/var/log/cloudera-scm-server -Dcmf.log.file=cloudera-scm-server.log -Dcmf.jetty.threshhold=WARN -Dcmf.schema.dir=/opt/cloudera/cm/schema -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dpython.home=/opt/cloudera/cm/python -XX:+HeapDumpOnOutOfMemoryError -Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:OnOutOfMemoryError=kill -9 %p com.cloudera.server.cmf.Main
root 14404 4844 0 02:06 pts/0 00:00:00 grep --color=auto -Ei 13778
[root@cm-r01nn01 ~]# [root@cm-r01nn01 ~]# top
top - 02:06:24 up 19 days, 4:04, 1 user, load average: 0.26, 0.28, 0.29
Tasks: 197 total, 1 running, 196 sleeping, 0 stopped, 0 zombie
%Cpu(s): 1.0 us, 0.5 sy, 0.0 ni, 98.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 8008640 total, 286564 free, 5060212 used, 2661864 buff/cache
KiB Swap: 4063228 total, 4063228 free, 0 used. 2205572 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3919 root 20 0 1260148 47188 8428 S 3.8 0.6 2:13.33 python2
13778 clouder+ 20 0 8263556 2.8g 17472 S 3.8 37.1 2208:20 java
3877 root 20 0 2680196 63480 9772 S 1.2 0.8 1:46.97 cmagent
4160 root 20 0 619212 28772 9028 S 1.2 0.4 0:32.63 python2
6281 httpfs 20 0 3860244 213008 29404 S 1.2 2.7 0:29.09 java
7367 hive 20 0 2118308 312892 41972 S 1.2 3.9 1:09.55 java
14318 root 20 0 172308 2404 1628 R 1.2 0.0 0:00.49 top
1 root 20 0 54952 7272 4268 S 0.0 0.1 6:44.58 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:01.41 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 1:27.50 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:04.12 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 29:36.83 rcu_sched
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain
11 root rt 0 0 0 0 S 0.0 0.0 0:12.34 watchdog/0
12 root rt 0 0 0 0 S 0.0 0.0 0:08.09 watchdog/1
13 root rt 0 0 0 0 S 0.0 0.0 0:02.97 migration/1
14 root 20 0 0 0 0 S 0.0 0.0 1:21.07 ksoftirqd/1
16 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:0H
17 root rt 0 0 0 0 S 0.0 0.0 0:07.84 watchdog/2
18 root rt 0 0 0 0 S 0.0 0.0 0:00.64 migration/2
19 root 20 0 0 0 0 S 0.0 0.0 1:20.44 ksoftirqd/2
21 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:0H
22 root rt 0 0 0 0 S 0.0 0.0 0:08.05 watchdog/3
23 root rt 0 0 0 0 S 0.0 0.0 0:00.39 migration/3
24 root 20 0 0 0 0 S 0.0 0.0 1:20.37 ksoftirqd/3
26 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:0H
27 root rt 0 0 0 0 S 0.0 0.0 0:08.38 watchdog/4
28 root rt 0 0 0 0 S 0.0 0.0 0:00.51 migration/4
[root@cm-r01nn01 ~]# ps -ef|grep -Ei 13778
clouder+ 13778 1 8 Aug25 ? 1-12:48:21 /usr/java/jdk1.8.0_181-cloudera/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server -Dlog4j.configuration=file:/etc/cloudera-scm-server/log4j.properties -Dfile.encoding=UTF-8 -Dcmf.root.logger=INFO,LOGFILE -Dcmf.log.dir=/var/log/cloudera-scm-server -Dcmf.log.file=cloudera-scm-server.log -Dcmf.jetty.threshhold=WARN -Dcmf.schema.dir=/opt/cloudera/cm/schema -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true -Dpython.home=/opt/cloudera/cm/python -XX:+HeapDumpOnOutOfMemoryError -Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:OnOutOfMemoryError=kill -9 %p com.cloudera.server.cmf.Main
root 14404 4844 0 02:06 pts/0 00:00:00 grep --color=auto -Ei 13778
[root@cm-r01nn01 ~]# systemctl status cloudera-scm-agent
â cloudera-scm-agent.service - Cloudera Manager Agent Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2019-09-13 00:55:44 EDT; 1h 11min ago
Main PID: 3877 (cmagent)
CGroup: /system.slice/cloudera-scm-agent.service
ââ3877 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agent
Sep 13 00:55:55 cm-r01nn01.mws.mds.xyz cm[3877]: warnings.warn(msg, RuntimeWarning)
Sep 13 01:10:58 cm-r01nn01.mws.mds.xyz cm[3877]: 2591-kms-KMS: added process group
Sep 13 01:10:58 cm-r01nn01.mws.mds.xyz cm[3877]: 2592-zookeeper-server: added process group
Sep 13 01:12:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2598-hdfs-HTTPFS: added process group
Sep 13 01:12:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2600-hdfs-SECONDARYNAMENODE: added process group
Sep 13 01:17:05 cm-r01nn01.mws.mds.xyz cm[3877]: [2019-09-13 01:17:05,746 pyinotify ERROR] add_watch: cannot watch /var/log/hive/audit WD=-1, Er...(ENOENT)
Sep 13 01:18:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2649-hive-HIVESERVER2: added process group
Sep 13 01:18:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2650-hive-WEBHCAT: added process group
Sep 13 01:54:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2649-hive-HIVESERVER2: stopped
Sep 13 01:54:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2649-hive-HIVESERVER2: removed process group
Hint: Some lines were ellipsized, use -l to show in full.
[root@cm-r01nn01 ~]# systemctl status cloudera-scm-agent -l
â cloudera-scm-agent.service - Cloudera Manager Agent Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2019-09-13 00:55:44 EDT; 1h 12min ago
Main PID: 3877 (cmagent)
CGroup: /system.slice/cloudera-scm-agent.service
ââ3877 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agent
Sep 13 00:55:55 cm-r01nn01.mws.mds.xyz cm[3877]: warnings.warn(msg, RuntimeWarning)
Sep 13 01:10:58 cm-r01nn01.mws.mds.xyz cm[3877]: 2591-kms-KMS: added process group
Sep 13 01:10:58 cm-r01nn01.mws.mds.xyz cm[3877]: 2592-zookeeper-server: added process group
Sep 13 01:12:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2598-hdfs-HTTPFS: added process group
Sep 13 01:12:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2600-hdfs-SECONDARYNAMENODE: added process group
Sep 13 01:17:05 cm-r01nn01.mws.mds.xyz cm[3877]: [2019-09-13 01:17:05,746 pyinotify ERROR] add_watch: cannot watch /var/log/hive/audit WD=-1, Errno=No such file or directory (ENOENT)
Sep 13 01:18:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2649-hive-HIVESERVER2: added process group
Sep 13 01:18:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2650-hive-WEBHCAT: added process group
Sep 13 01:54:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2649-hive-HIVESERVER2: stopped
Sep 13 01:54:01 cm-r01nn01.mws.mds.xyz cm[3877]: 2649-hive-HIVESERVER2: removed process group
[root@cm-r01nn01 ~]#
... View more
09-12-2019
09:53 PM
Is there anywhere to reduce the cloudera-scm-agent memory usage? Currently it's consuming 1.9G:
top - 00:49:46 up 10 days, 11:06, 2 users, load average: 0.62, 0.43, 0.50
Tasks: 195 total, 1 running, 194 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.5 us, 0.2 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 8008644 total, 3828140 free, 3228256 used, 952248 buff/cache
KiB Swap: 4063228 total, 3974644 free, 88584 used. 4180168 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24743 clouder+ 20 0 7107092 278440 5388 S 16.6 3.5 1329:59 java
13611 root 20 0 1260404 41488 2252 S 2.7 0.5 1:27.75 python2
23190 clouder+ 20 0 8526144 1.9g 47492 S 1.3 25.5 2470:33 java
Even with a stopped cluster.
Anyway to tweak this on smaller POC clusters?
Would anyone be able to ballpark memory settings for small Cloudera clusters and services?
Need to restart the agent and clear caches to free up memory. Wondering if that could be reduced instead.
Cheers, TK
... View more
Labels:
09-03-2019
09:50 PM
I've adjusted the auth_to_local rules as follows: RULE:[2:$1@$0](HTTP@\QMWS.MDS.XYZ\E$)s/@\QMWS.MDS.XYZ\E$//
RULE:[1:$1@$0](.*@\QMWS.MDS.XYZ\E$)s/@\QMWS.MDS.XYZ\E$///L
RULE:[2:$1@$0](.*@\QMWS.MDS.XYZ\E$)s/@\QMWS.MDS.XYZ\E$///L
RULE:[2:$1@$0](HTTP@\Qmws.mds.xyz\E$)s/@\Qmws.mds.xyz\E$//
RULE:[1:$1@$0](.*@\Qmws.mds.xyz\E$)s/@\Qmws.mds.xyz\E$///L
RULE:[2:$1@$0](.*@\Qmws.mds.xyz\E$)s/@\Qmws.mds.xyz\E$///L
RULE:[2:$1@$0](HTTP@\QMDS.XYZ\E$)s/@\QMDS.XYZ\E$//
RULE:[1:$1@$0](.*@\QMDS.XYZ\E$)s/@\QMDS.XYZ\E$///L
RULE:[2:$1@$0](.*@\QMDS.XYZ\E$)s/@\QMDS.XYZ\E$///L
RULE:[2:$1@$0](HTTP@\Qmds.xyz\E$)s/@\Qmds.xyz\E$//
RULE:[1:$1@$0](.*@\Qmds.xyz\E$)s/@\Qmds.xyz\E$///L
RULE:[2:$1@$0](.*@\Qmds.xyz\E$)s/@\Qmds.xyz\E$///L
DEFAULT And now when I create the folder in this manner, spark-shell starts using the below folder: drwxr-xr-x - tom tom 0 2019-09-04 00:45 /user/tom Since I have multiple users from multiple domains, collisions can occur if same user exists in two different domains. So I would be curious if you know how to create the folders in this manner: /user/<DOMAIN>/<USER> to avoid potential conflict with same named user but in different domains. Cheers, TK
... View more
09-03-2019
08:48 PM
Odd then. This format worked: drwxr-xr-x - tom@MDS.XYZ tom@MDS.XYZ 0 2019-09-03 21:54 /user/tom@MDS.XYZ Perhaps I should be expecting something in the settings?
... View more
09-03-2019
08:19 PM
What I'm getting now is the below and wondering what the solution might be, having tried the ones on the community so far without success: INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all This message floods the spark-shell console rendering it unusable. What I did so far to try and check it: 1) Reverse lookups work. 2) Forward lookups work. 3) UID of CM Agent are unique. 4) RHEL 7 UID's are unique. Looks like it might be related to this bug so I may just have to wait it out or grab a copy of the latest Spark somehow to fix it? https://issues.apache.org/jira/browse/SPARK-28005 Cheers, TK
... View more
09-03-2019
07:02 PM
I had the directory /user/tom and tried the following permissions: tom:tom
tom@mds.xyz:tom@mds.xyz
tom@MDS.XYZ:tom@MDS.XYZ No luck. Till I saw this message: 19/09/03 21:51:53 WARN fs.TrashPolicyDefault: Can't create trash directory: hdfs://cm-r01nn02.mws.mds.xyz:8020/user/tom@MDS.XYZ/.Trash/Current/user/mds.xyz/tom
org.apache.hadoop.security.AccessControlException: Permission denied: user=tom@MDS.XYZ, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x Telling me the right folder was supposed to be /user/tom@MDS.XYZ. So that's what I set and spark-shell now works. It really has to do with this issue: 19/09/03 21:51:33 INFO util.KerberosName: No auth_to_local rules applied to tom@MDS.XYZ And I really need to define auth_to_local to create the folders in this manner: /user/domain/user But I'm not sure how just yet. Cheers, TK
... View more
09-02-2019
07:37 PM
Just to clarify, did you mean you "can" or "can't" stop client side gateway roles?
... View more
09-02-2019
07:35 PM
I've upped the memory to get over the issue stated above. Now I get: Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=tom@MDS.XYZ, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x How do I configured Spark to write to individual user folders such as /user/tom? Cheers, TK
... View more
09-02-2019
07:18 AM
Hey Eric, Thanks! Yes, I've added the Hive and Spark gateway role to the host. The Spark gateways are distributed to the same nodes. However, since your comment, I notice both Hive and Spark gateways are offline. Can't start them as of this writing. Getting: Command Start is not currently available for execution. whenever I try to start the role. So definitely an issue there. Kerberos credentials appear ok. I can regenerate them without an issue. Running kinit using the hdfs.keytab works as expected. On a closer look, I do get this error which I'll try to fix up after this comment. 19/09/02 09:56:42 ERROR repl.Main: Failed to initialize Spark session.
java.lang.IllegalArgumentException: Required executor memory (1024), overhead (384 MB), and PySpark memory (0 MB) is above the max threshold (256 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'. This is a fairly small cluster for POC type of work so I would rather tweak Spark memory requirements rather than increase max memory per container. Not able to figure that out yet. Cheers, TK
... View more
08-31-2019
08:38 AM
Hey All,
I'm trying to run spark-shell for the first time on a CM / CDH 6.3 installation. But getting the below instead.
19/08/31 11:05:24 DEBUG ipc.Client: The ping interval is 60000 ms.
19/08/31 11:05:24 DEBUG ipc.Client: Connecting to cm-r01nn02.mws.mds.xyz/192.168.0.133:8032
19/08/31 11:05:24 DEBUG security.UserGroupInformation: PrivilegedAction as:root (auth:SIMPLE) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
19/08/31 11:05:24 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE
19/08/31 11:05:24 DEBUG security.SaslRpcClient: Get token info proto:interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB info:org.apache.hadoop.yarn.security.client.ClientRMSecurityInfo$2@238c63df
19/08/31 11:05:24 DEBUG client.RMDelegationTokenSelector: Looking for a token with service 192.168.0.133:8032
19/08/31 11:05:24 DEBUG security.SaslRpcClient: tokens aren't supported for this protocol or user doesn't have one
19/08/31 11:05:24 DEBUG security.SaslRpcClient: client isn't using kerberos
19/08/31 11:05:24 DEBUG security.UserGroupInformation: PrivilegedActionException as:root (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/31 11:05:24 DEBUG security.UserGroupInformation: PrivilegedAction as:root (auth:SIMPLE) from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:719)
19/08/31 11:05:24 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/31 11:05:24 DEBUG security.UserGroupInformation: PrivilegedActionException as:root (auth:SIMPLE) cause:java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/31 11:05:24 DEBUG ipc.Client: closing ipc connection to cm-r01nn02.mws.mds.xyz/192.168.0.133:8032: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:756)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:719)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560)
at org.apache.hadoop.ipc.Client.call(Client.java:1391)
at org.apache.hadoop.ipc.Client.call(Client.java:1355)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy16.getClusterMetrics(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:251)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy17.getClusterMetrics(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:604)
at org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:169)
at org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:169)
at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:57)
at org.apache.spark.deploy.yarn.Client.logInfo(Client.scala:62)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:168)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:60)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:186)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:511)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
at $line3.$read$$iw$$iw.<init>(<console>:15)
at $line3.$read$$iw.<init>(<console>:43)
at $line3.$read.<init>(<console>:45)
at $line3.$read$.<init>(<console>:49)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.$print$lzycompute(<console>:7)
at $line3.$eval$.$print(<console>:6)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:793)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1054)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:645)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:644)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:644)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:576)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:572)
at scala.tools.nsc.interpreter.IMain$$anonfun$quietRun$1.apply(IMain.scala:231)
at scala.tools.nsc.interpreter.IMain$$anonfun$quietRun$1.apply(IMain.scala:231)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:221)
at scala.tools.nsc.interpreter.IMain.quietRun(IMain.scala:231)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:109)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:109)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:109)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:109)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:109)
at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:91)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:108)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1$1.apply$mcV$sp(SparkILoop.scala:211)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1$1.apply(SparkILoop.scala:199)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1$1.apply(SparkILoop.scala:199)
at scala.tools.nsc.interpreter.ILoop$$anonfun$mumly$1.apply(ILoop.scala:189)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:221)
at scala.tools.nsc.interpreter.ILoop.mumly(ILoop.scala:186)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1(SparkILoop.scala:199)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$startup$1$1.apply(SparkILoop.scala:267)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$startup$1$1.apply(SparkILoop.scala:247)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.withSuppressedSettings$1(SparkILoop.scala:235)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.startup$1(SparkILoop.scala:247)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:282)
at org.apache.spark.repl.SparkILoop.runClosure(SparkILoop.scala:159)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:182)
at org.apache.spark.repl.Main$.doMain(Main.scala:78)
at org.apache.spark.repl.Main$.main(Main.scala:58)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:614)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:410)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
... 95 more
19/08/31 11:05:24 DEBUG ipc.Client: IPC Client (483582792) connection to cm-r01nn02.mws.mds.xyz/192.168.0.133:8032 from root: closed
19/08/31 11:05:24 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "cm-r01en01.mws.mds.xyz/192.168.0.140"; destination host is: "cm-r01nn02.mws.mds.xyz":8032; , while invoking ApplicationClientProtocolPBClientImpl.getClusterMetrics over null after 6 failover attempts. Trying to failover after sleeping for 19516ms.
19/08/31 11:05:24 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:25 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:26 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:27 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:28 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:29 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:30 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:31 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:32 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:33 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:34 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:35 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:36 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:37 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:38 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:39 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:40 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:41 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:42 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:43 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:43 DEBUG ipc.Client: The ping interval is 60000 ms.
19/08/31 11:05:43 DEBUG ipc.Client: Connecting to cm-r01nn02.mws.mds.xyz/192.168.0.133:8032
19/08/31 11:05:43 DEBUG security.UserGroupInformation: PrivilegedAction as:root (auth:SIMPLE) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
19/08/31 11:05:43 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE
19/08/31 11:05:43 DEBUG security.SaslRpcClient: Get token info proto:interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB info:org.apache.hadoop.yarn.security.client.ClientRMSecurityInfo$2@321558f8
19/08/31 11:05:43 DEBUG client.RMDelegationTokenSelector: Looking for a token with service 192.168.0.133:8032
19/08/31 11:05:43 DEBUG security.SaslRpcClient: tokens aren't supported for this protocol or user doesn't have one
19/08/31 11:05:43 DEBUG security.SaslRpcClient: client isn't using kerberos
19/08/31 11:05:43 DEBUG security.UserGroupInformation: PrivilegedActionException as:root (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/31 11:05:43 DEBUG security.UserGroupInformation: PrivilegedAction as:root (auth:SIMPLE) from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:719)
19/08/31 11:05:43 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/31 11:05:43 DEBUG security.UserGroupInformation: PrivilegedActionException as:root (auth:SIMPLE) cause:java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/31 11:05:43 DEBUG ipc.Client: closing ipc connection to cm-r01nn02.mws.mds.xyz/192.168.0.133:8032: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:756)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:719)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560)
at org.apache.hadoop.ipc.Client.call(Client.java:1391)
at org.apache.hadoop.ipc.Client.call(Client.java:1355)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy16.getClusterMetrics(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:251)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy17.getClusterMetrics(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:604)
at org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:169)
at org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:169)
at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:57)
at org.apache.spark.deploy.yarn.Client.logInfo(Client.scala:62)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:168)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:60)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:186)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:511)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
at $line3.$read$$iw$$iw.<init>(<console>:15)
at $line3.$read$$iw.<init>(<console>:43)
at $line3.$read.<init>(<console>:45)
at $line3.$read$.<init>(<console>:49)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.$print$lzycompute(<console>:7)
at $line3.$eval$.$print(<console>:6)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:793)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1054)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:645)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:644)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:644)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:576)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:572)
at scala.tools.nsc.interpreter.IMain$$anonfun$quietRun$1.apply(IMain.scala:231)
at scala.tools.nsc.interpreter.IMain$$anonfun$quietRun$1.apply(IMain.scala:231)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:221)
at scala.tools.nsc.interpreter.IMain.quietRun(IMain.scala:231)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:109)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:109)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:109)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:109)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:109)
at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:91)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:108)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1$1.apply$mcV$sp(SparkILoop.scala:211)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1$1.apply(SparkILoop.scala:199)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1$1.apply(SparkILoop.scala:199)
at scala.tools.nsc.interpreter.ILoop$$anonfun$mumly$1.apply(ILoop.scala:189)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:221)
at scala.tools.nsc.interpreter.ILoop.mumly(ILoop.scala:186)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1(SparkILoop.scala:199)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$startup$1$1.apply(SparkILoop.scala:267)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$startup$1$1.apply(SparkILoop.scala:247)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.withSuppressedSettings$1(SparkILoop.scala:235)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.startup$1(SparkILoop.scala:247)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:282)
at org.apache.spark.repl.SparkILoop.runClosure(SparkILoop.scala:159)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:182)
at org.apache.spark.repl.Main$.doMain(Main.scala:78)
at org.apache.spark.repl.Main$.main(Main.scala:58)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:614)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:410)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
... 95 more
19/08/31 11:05:43 DEBUG ipc.Client: IPC Client (483582792) connection to cm-r01nn02.mws.mds.xyz/192.168.0.133:8032 from root: closed
19/08/31 11:05:43 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "cm-r01en01.mws.mds.xyz/192.168.0.140"; destination host is: "cm-r01nn02.mws.mds.xyz":8032; , while invoking ApplicationClientProtocolPBClientImpl.getClusterMetrics over null after 7 failover attempts. Trying to failover after sleeping for 33704ms.
19/08/31 11:05:44 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:45 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:46 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:47 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:48 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:49 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:50 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:51 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:52 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:53 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:54 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:55 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:56 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:57 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:58 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:05:59 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:06:00 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:06:01 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:06:02 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:06:03 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:06:04 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:06:05 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:06:06 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:06:07 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:06:08 INFO storage.DiskBlockManager: Shutdown hook called
19/08/31 11:06:08 INFO util.ShutdownHookManager: Shutdown hook called
19/08/31 11:06:08 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-9a9338cb-f16b-48e0-b0cd-7ddfcc148a13/repl-52ba4c53-3478-4ead-93e7-d20ecbd2e866
19/08/31 11:06:08 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-9a9338cb-f16b-48e0-b0cd-7ddfcc148a13/userFiles-5f218430-30bb-4a7e-87df-7ee235183578
19/08/31 11:06:08 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-9a9338cb-f16b-48e0-b0cd-7ddfcc148a13
19/08/31 11:06:08 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-79f3eecb-69d4-4b21-85dc-6746fc33f65c
19/08/31 11:06:08 DEBUG ipc.Client: stopping client from cache: org.apache.hadoop.ipc.Client@19656e21
19/08/31 11:06:08 DEBUG util.ShutdownHookManager: Completed shutdown in 0.062 seconds; Timeouts: 0
19/08/31 11:06:08 DEBUG util.ShutdownHookManager: ShutdownHookManger completed shutdown.
[root@cm-r01en01 process]# dig -x 192.168.0.140
; <<>> DiG 9.9.4-RedHat-9.9.4-73.el7_6 <<>> -x 192.168.0.140
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39821
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;140.0.168.192.in-addr.arpa. IN PTR
;; ANSWER SECTION:
140.0.168.192.in-addr.arpa. 1200 IN PTR cm-r01en01.mws.mds.xyz.
;; AUTHORITY SECTION:
0.168.192.in-addr.arpa. 86400 IN NS idmipa03.mws.mds.xyz.
0.168.192.in-addr.arpa. 86400 IN NS idmipa04.mws.mds.xyz.
;; ADDITIONAL SECTION:
idmipa03.mws.mds.xyz. 1200 IN A 192.168.0.154
idmipa04.mws.mds.xyz. 1200 IN A 192.168.0.155
;; Query time: 1 msec
;; SERVER: 192.168.0.154#53(192.168.0.154)
;; WHEN: Sat Aug 31 11:06:18 EDT 2019
;; MSG SIZE rcvd: 169
[root@cm-r01en01 process]# dig -x 192.168.0.133
; <<>> DiG 9.9.4-RedHat-9.9.4-73.el7_6 <<>> -x 192.168.0.133
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11817
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;133.0.168.192.in-addr.arpa. IN PTR
;; ANSWER SECTION:
133.0.168.192.in-addr.arpa. 1200 IN PTR cm-r01nn02.mws.mds.xyz.
;; AUTHORITY SECTION:
0.168.192.in-addr.arpa. 86400 IN NS idmipa04.mws.mds.xyz.
0.168.192.in-addr.arpa. 86400 IN NS idmipa03.mws.mds.xyz.
;; ADDITIONAL SECTION:
idmipa03.mws.mds.xyz. 1200 IN A 192.168.0.154
idmipa04.mws.mds.xyz. 1200 IN A 192.168.0.155
;; Query time: 1 msec
;; SERVER: 192.168.0.154#53(192.168.0.154)
;; WHEN: Sat Aug 31 11:26:10 EDT 2019
;; MSG SIZE rcvd: 169
[root@cm-r01en01 process]#
I try the same as a non previlidged AD / FreeIPA user but with same results:
19/08/31 11:33:07 DEBUG ipc.Client: The ping interval is 60000 ms.
19/08/31 11:33:07 DEBUG ipc.Client: Connecting to cm-r01nn02.mws.mds.xyz/192.168.0.133:8032
19/08/31 11:33:07 DEBUG security.UserGroupInformation: PrivilegedAction as:tom@mds.xyz (auth:SIMPLE) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
19/08/31 11:33:07 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE
19/08/31 11:33:07 DEBUG security.SaslRpcClient: Get token info proto:interface org.apache.hadoop.yarn.api.ApplicationClientProtocolPB info:org.apache.hadoop.yarn.security.client.ClientRMSecurityInfo$2@6d4df1d2
19/08/31 11:33:07 DEBUG client.RMDelegationTokenSelector: Looking for a token with service 192.168.0.133:8032
19/08/31 11:33:07 DEBUG security.SaslRpcClient: tokens aren't supported for this protocol or user doesn't have one
19/08/31 11:33:07 DEBUG security.SaslRpcClient: client isn't using kerberos
19/08/31 11:33:07 DEBUG security.UserGroupInformation: PrivilegedActionException as:tom@mds.xyz (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/31 11:33:07 DEBUG security.UserGroupInformation: PrivilegedAction as:tom@mds.xyz (auth:SIMPLE) from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:719)
19/08/31 11:33:07 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/31 11:33:07 DEBUG security.UserGroupInformation: PrivilegedActionException as:tom@mds.xyz (auth:SIMPLE) cause:java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
19/08/31 11:33:07 DEBUG ipc.Client: closing ipc connection to cm-r01nn02.mws.mds.xyz/192.168.0.133:8032: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:756)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:719)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:812)
at org.apache.hadoop.ipc.Client$Connection.access$3600(Client.java:410)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1560)
at org.apache.hadoop.ipc.Client.call(Client.java:1391)
at org.apache.hadoop.ipc.Client.call(Client.java:1355)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
at com.sun.proxy.$Proxy16.getClusterMetrics(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getClusterMetrics(ApplicationClientProtocolPBClientImpl.java:251)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:422)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:165)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:157)
at org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:95)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:359)
at com.sun.proxy.$Proxy17.getClusterMetrics(Unknown Source)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getYarnClusterMetrics(YarnClientImpl.java:604)
at org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:169)
at org.apache.spark.deploy.yarn.Client$$anonfun$submitApplication$1.apply(Client.scala:169)
at org.apache.spark.internal.Logging$class.logInfo(Logging.scala:57)
at org.apache.spark.deploy.yarn.Client.logInfo(Client.scala:62)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:168)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:60)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:186)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:511)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2549)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:944)
at org.apache.spark.sql.SparkSession$Builder$$anonfun$7.apply(SparkSession.scala:935)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:935)
at org.apache.spark.repl.Main$.createSparkSession(Main.scala:106)
at $line3.$read$$iw$$iw.<init>(<console>:15)
at $line3.$read$$iw.<init>(<console>:43)
at $line3.$read.<init>(<console>:45)
at $line3.$read$.<init>(<console>:49)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.$print$lzycompute(<console>:7)
at $line3.$eval$.$print(<console>:6)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:793)
at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1054)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:645)
at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:644)
at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:644)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:576)
at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:572)
at scala.tools.nsc.interpreter.IMain$$anonfun$quietRun$1.apply(IMain.scala:231)
at scala.tools.nsc.interpreter.IMain$$anonfun$quietRun$1.apply(IMain.scala:231)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:221)
at scala.tools.nsc.interpreter.IMain.quietRun(IMain.scala:231)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:109)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1$$anonfun$apply$mcV$sp$1.apply(SparkILoop.scala:109)
at scala.collection.immutable.List.foreach(List.scala:392)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply$mcV$sp(SparkILoop.scala:109)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:109)
at org.apache.spark.repl.SparkILoop$$anonfun$initializeSpark$1.apply(SparkILoop.scala:109)
at scala.tools.nsc.interpreter.ILoop.savingReplayStack(ILoop.scala:91)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:108)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1$1.apply$mcV$sp(SparkILoop.scala:211)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1$1.apply(SparkILoop.scala:199)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1$1.apply(SparkILoop.scala:199)
at scala.tools.nsc.interpreter.ILoop$$anonfun$mumly$1.apply(ILoop.scala:189)
at scala.tools.nsc.interpreter.IMain.beQuietDuring(IMain.scala:221)
at scala.tools.nsc.interpreter.ILoop.mumly(ILoop.scala:186)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.org$apache$spark$repl$SparkILoop$$anonfun$$loopPostInit$1(SparkILoop.scala:199)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$startup$1$1.apply(SparkILoop.scala:267)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$startup$1$1.apply(SparkILoop.scala:247)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.withSuppressedSettings$1(SparkILoop.scala:235)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.startup$1(SparkILoop.scala:247)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:282)
at org.apache.spark.repl.SparkILoop.runClosure(SparkILoop.scala:159)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:182)
at org.apache.spark.repl.Main$.doMain(Main.scala:78)
at org.apache.spark.repl.Main$.main(Main.scala:58)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:173)
at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:390)
at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:614)
at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:410)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:799)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:795)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
... 95 more
19/08/31 11:33:07 DEBUG ipc.Client: IPC Client (1263257405) connection to cm-r01nn02.mws.mds.xyz/192.168.0.133:8032 from tom@mds.xyz: closed
19/08/31 11:33:07 INFO retry.RetryInvocationHandler: java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "cm-r01en01.mws.mds.xyz/192.168.0.140"; destination host is: "cm-r01nn02.mws.mds.xyz":8032; , while invoking ApplicationClientProtocolPBClientImpl.getClusterMetrics over null after 1 failover attempts. Trying to failover after sleeping for 17516ms.
19/08/31 11:33:07 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:08 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:09 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:10 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:11 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:12 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:13 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:14 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:15 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:16 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:17 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:18 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:19 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:20 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:21 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:22 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:23 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:24 INFO yarn.SparkRackResolver: Got an error when resolving hostNames. Falling back to /default-rack for all
19/08/31 11:33:25 DEBUG ipc.Client: The ping interval is 60000 ms.
19/08/31 11:33:25 DEBUG ipc.Client: Connecting to cm-r01nn02.mws.mds.xyz/192.168.0.133:8032
19/08/31 11:33:25 DEBUG security.UserGroupInformation: PrivilegedAction as:tom@mds.xyz (auth:SIMPLE) from:org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:795)
19/08/31 11:33:25 DEBUG security.SaslRpcClient: Sending sasl message state: NEGOTIATE
Has anyone seen the same and could suggest what to do to move forward with this?
A few points: 1) Reverse and forward lookups work fine from the OS side. 2) Kerberos credentials generate without issue. Cheers, TK
... View more
Labels:
08-17-2019
07:07 PM
I'm receiving the following message from my CDH 6.2 installation. I do see some older posts explaining the issue however as they are old posts, I don't believe they are relevent to this version. Would anyone know the specifics of what this check does or looks for on RHEL 7 hosts that's specific to Cloudera versions? Mismatched CDH versions: host has NONE but role expects 6 Cheers, TK
... View more
08-14-2019
08:39 PM
Currently stuck on this error: Run a set of services for the first time
Completed only 8/9 steps. First failure: Failed to execute command Install Oozie ShareLib on service Oozie
Aug 14, 11:32:18 PM 4.9m
Execute 8 steps in sequence
Completed only 8/9 steps. First failure: Failed to execute command Install Oozie ShareLib on service Oozie
Aug 14, 11:32:23 PM 4.8m
Execute 9 steps in parallel
Completed only 8/9 steps. First failure: Failed to execute command Install Oozie ShareLib on service Oozie
Aug 14, 11:32:23 PM 4.8m
Installing the Oozie ShareLib in HDFS
Failed to execute command Install Oozie ShareLib on service Oozie
Oozie
Aug 14, 11:32:23 PM 4.8m
Install Oozie ShareLib
Failed to install Oozie ShareLib.
Oozie
Aug 14, 11:32:23 PM 4.8m
Upload Oozie ShareLib
Command aborted because of exception: Command timed-out after 270 seconds
Oozie Server (cm-r01en01)
Aug 14, 11:32:37 PM 4.6m
Upload Oozie ShareLib
Command aborted because of exception: Command timed-out after 270 seconds
Oozie Server (cm-r01en01)
Aug 14, 11:32:37 PM 4.6m
$> oozie/oozie.sh ["install-sharelib","oozie-sharelib-yarn","hdfs://cm-r01nn02.mws.mds.xyz:8020","775","/user/oozie","8"]
stdout
stderr
Role Log
Wed Aug 14 23:32:58 EDT 2019
JAVA_HOME=/usr/java/latest
using 6 as CDH_VERSION
CONF_DIR=/var/run/cloudera-scm-agent/process/166-oozie-OOZIE-SERVER-upload-sharelib
CMF_CONF_DIR=
Found Hadoop that supports Erasure Coding. Trying to disable Erasure Coding for path: /user/oozie/share/lib
Done
the destination path for sharelib is: /user/oozie/share/lib/lib_20190814233305
Running 1738 copy tasks on 8 threads
Full log file
$> oozie/oozie.sh ["install-sharelib","oozie-sharelib-yarn","hdfs://cm-r01nn02.mws.mds.xyz:8020","775","/user/oozie","8"]
stdout
stderr
Role Log
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.
Full log file Same effect as the earlier error: nothing else installs after. Not sure if it's significant however originaly had RPM's installed not parcels. Reverted to parcels since the installer failed even detecting a PostgreSQL DB Cluster when RPM's were used (Had a separate post on that here). Cheers, TK
... View more
08-14-2019
06:59 PM
This affects the Oozie setup and configuration. The other services are not installed since this error breaks the installation chain. This is the list of the steps and where it get's stuck on: Completed 1 of 1 step(s).
Show All Steps Show Only Failed Steps Show Only Running Steps
Run a set of services for the first time
Completed only 8/9 steps. First failure: Failed to execute command Install Oozie ShareLib on service Oozie
Aug 14, 12:30:25 AM 5m
Execute 8 steps in sequence
Completed only 8/9 steps. First failure: Failed to execute command Install Oozie ShareLib on service Oozie
Aug 14, 12:30:30 AM 4.9m
Execute 9 steps in parallel
Completed only 8/9 steps. First failure: Failed to execute command Install Oozie ShareLib on service Oozie
Aug 14, 12:30:30 AM 4.9m
Installing the Oozie ShareLib in HDFS
Failed to execute command Install Oozie ShareLib on service Oozie
Oozie
Aug 14, 12:30:30 AM 4.9m
Install Oozie ShareLib
Failed to install Oozie ShareLib.
Oozie
Aug 14, 12:30:31 AM 4.9m
Upload Oozie ShareLib
Command aborted because of exception: Command timed-out after 270 seconds
Oozie Server (cm-r01en01)
Aug 14, 12:30:50 AM 4.6m
Upload Oozie ShareLib
Command aborted because of exception: Command timed-out after 270 seconds
Oozie Server (cm-r01en01)
Aug 14, 12:30:50 AM 4.6m
$>
oozie/oozie.sh ["install-sharelib","oozie-sharelib-yarn","hdfs://cm-r01nn02.mws.mds.xyz:8020","775","/user/oozie","8"]
stdout
stderr
Role Log
log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender].
java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395)
at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:955)
at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89)
at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58)
at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:947)
at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:473)
at org.apache.oozie.service.XLogService.init(XLogService.java:178)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.<init>(Services.java:111)
at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194)
at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97)
log4j:ERROR Could not instantiate appender named "redactor1".
... View more
08-14-2019
04:56 AM
Hey All,
Getting the following on a brand new 6.2 installation and wondering what is the 'proper' fix for this.
I can start symlinking to get things working but perhaps I haven't deployed a package or taken another step I should have taken earlier?
+ /usr/java/latest/bin/java -Xms52428800 -Xmx52428800 -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 -XX:+DisableExplicitGC -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/oozie_oozie-OOZIE_SERVER-901d5713a53510380392378fa81b483d_pid18817.hprof -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh -Doozie.home.dir=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie -Doozie.config.dir=/run/cloudera-scm-agent/process/152-oozie-OOZIE-SERVER-upload-sharelib -Doozie.log.dir=/var/log/oozie -Doozie.log.file=oozie-cmf-oozie-OOZIE_SERVER-cm-r01en01.mws.mds.xyz.log.out -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=cm-r01en01.mws.mds.xyz -Doozie.http.port=11000 -Djava.net.preferIPv4Stack=true -Doozie.admin.port= -Dderby.stream.error.file=/var/log/oozie/derby.log -Doozie.instance.id=cm-r01en01.mws.mds.xyz -Djava.library.path=/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/hadoop/lib/native -cp ':/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/accessors-smart-1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/activation-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/aggdesigner-algorithm-6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/ant-1.6.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/antlr-2.7.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/antlr-runtime-3.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/aopalliance-1.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/aopalliance-repackaged-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/apache-jsp-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/apache-jstl-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/apache-log4j-extras-1.2.17.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/asm-5.0.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/asm-commons-6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/asm-tree-6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/audience-annotations-0.5.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/automaton-1.11-8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/avro.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-anim-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-awt-util-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-bridge-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-codec-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-constants-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-css-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-dom-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-ext-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-gvt-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-i18n-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-parser-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-rasterizer-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-script-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-svg-dom-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-svggen-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-svgrasterizer-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-transcoder-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-util-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/batik-xml-1.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/bcpkix-jdk15on-1.60.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/bcprov-jdk15on-1.60.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/bonecp-0.8.0.RELEASE.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/c3p0-0.9.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-beanutils-1.9.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-cli-1.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-codec-1.9.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-collections-3.2.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-compress-1.4.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-configuration2-2.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-crypto-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-daemon-1.0.13.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-dbcp-1.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-el-1.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-exec-1.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-httpclient-3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-io-2.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-lang-2.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-lang3-3.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-logging-1.1.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-math3-3.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-net-3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/commons-pool-1.5.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/curator-client-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/curator-framework-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/curator-recipes-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/curator-x-discovery-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/datanucleus-api-jdo-4.2.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/datanucleus-core-4.1.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/datanucleus-rdbms-4.1.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/disruptor-3.3.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/dozer-5.5.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/ecj-4.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/ehcache-3.3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/ehcache-core-2.6.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/fastutil-6.5.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/findbugs-annotations-1.3.9-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/fst-2.50.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/geronimo-jms_1.1_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/geronimo-jpa_2.0_spec-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/geronimo-jta_1.1_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/gmetric4j-1.0.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/graphviz-java-0.7.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/gson-2.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/guava-11.0.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/guice-3.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/guice-assistedinject-3.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/guice-servlet-4.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-annotations.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-auth.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-client-3.0.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-hdfs-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-hdfs.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-mapreduce-client-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-mapreduce-client-core.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-mapreduce-client-jobclient.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-api.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-registry.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-server-applicationhistoryservice.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-server-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-server-resourcemanager.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hadoop-yarn-server-web-proxy.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-client-2.1.0-cdh6.2.0-tests.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-hadoop2-compat.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-hadoop-compat.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-http.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-mapreduce.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-metrics-api.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-metrics.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-procedure.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-protocol.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-protocol-shaded.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-replication.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-server.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-shaded-miscellaneous.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-shaded-netty.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-shaded-protobuf.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hbase-zookeeper.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/HikariCP-2.6.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/HikariCP-java7-2.4.12.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-classification.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-hcatalog-pig-adapter.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-jdbc.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-llap-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-llap-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-llap-server.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-llap-tez.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-metastore.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-serde.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-service.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-service-rpc.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-shims-0.23.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-shims-common.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-shims.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hive-shims-scheduler.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hk2-api-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hk2-locator-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hk2-utils-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/hsqldb-1.8.0.10.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/htrace-core4-4.1.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/httpclient-4.5.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/httpcore-4.4.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/j2v8_linux_x86_64-4.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/j2v8_macosx_x86_64-4.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/j2v8_win32_x86-4.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/j2v8_win32_x86_64-4.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-annotations-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-core-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-core-asl-1.9.13.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-databind-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-jaxrs-1.9.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-jaxrs-base-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-jaxrs-json-provider-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-mapper-asl-1.9.13.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-module-jaxb-annotations-2.9.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jackson-xc-1.9.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jamon-runtime-2.3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jansi-1.9.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jasper-compiler-5.5.23.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jasper-runtime-5.5.23.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/JavaEWAH-1.1.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javaparser-1.0.11.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javassist-3.20.0-GA.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/java-util-1.9.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.annotation-api-1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.el-3.0.1-b11.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.inject-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.inject-2.5.0-b32.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.jdo-3.2.0-m3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.servlet-api-3.1.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.servlet.jsp-2.3.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.servlet.jsp-api-2.3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javax.ws.rs-api-2.0.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/javolution-5.5.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb2-basics-1.11.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb2-basics-runtime-1.11.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb2-basics-tools-1.11.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb-api-2.2.11.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jaxb-impl-2.2.3-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jcip-annotations-1.0-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jcl-over-slf4j-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jcodings-1.0.18.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jcommander-1.30.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jdo-api-3.0.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jdom-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-client-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-client-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-common-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-container-servlet-core-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-core-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-guava-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-guice-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-json-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-media-jaxb-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-server-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-server-2.25.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jersey-servlet-1.19.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jettison-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-annotations-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-http-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-io-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-jaas-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-jndi-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-plus-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-runner-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-schemas-3.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-security-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-server-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-servlet-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-util-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-util-ajax-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-webapp-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jetty-xml-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jline-2.12.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jms-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/joda-time-2.9.9.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/joni-2.1.11.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jpam-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jsch-0.1.54.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/json-io-2.5.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/json-simple-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/json-smart-2.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jsp-api-2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jsr305-3.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jsr311-api-1.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jta-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jul-to-slf4j-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jython-standalone-2.7.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/jzlib-1.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-admin-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-client-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-common-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-core-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-crypto-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-identity-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-server-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-simplekdc-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerb-util-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-asn1-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-config-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-pkix-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-util-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kerby-xdr-1.0.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/kryo-2.22.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/leveldbjni-all-1.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/libfb303-0.9.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/libthrift-0.9.3-1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/log4j-1.2.17.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/log4j-api-2.8.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/log4j-slf4j-impl-2.8.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/mail-1.4.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-core-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-ganglia-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-graphite-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-json-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/metrics-jvm-3.1.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/mssql-jdbc-6.2.1.jre7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/nashorn-promise-0.1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/netty-3.10.6.Final.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/netty-all-4.1.17.Final.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/nimbus-jose-jwt-4.41.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/objenesis-1.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/okhttp-2.7.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/okio-1.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-client-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-client.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-core-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-core.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-fluent-job-api-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-fluent-job-api.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-sharelib-hcatalog-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-sharelib-hcatalog.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-sharelib-oozie-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-sharelib-oozie.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-tools-5.1.0-cdh6.2.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/oozie-tools.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/opencsv-2.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-jdbc-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-kernel-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-lib-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-persistence-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/openjpa-persistence-jdbc-2.4.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/org.eclipse.jgit-5.0.1.201806211838-r.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/osgi-resource-locator-1.0.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/paranamer-2.8.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/pig.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/postgresql-9.0-801.jdbc4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/protobuf-java-2.5.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/quartz-2.1.7.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/re2j-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/serializer-2.7.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/serp-1.15.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/servlet-api-2.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/slf4j-api-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/slf4j-log4j12-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/slf4j-simple-1.7.25.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/slider-core-0.90.2-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/snappy-java-1.1.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/stax2-api-3.1.4.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/stringtemplate-3.2.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/taglibs-standard-impl-1.2.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/taglibs-standard-spec-1.2.5.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/tephra-api-0.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/tephra-core-0.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/tephra-hbase-compat-1.0-0.6.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/transaction-api-1.1.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-api-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-common-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-core-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-discovery-api-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-discovery-core-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/twill-zookeeper-0.6.0-incubating.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/validation-api-1.1.0.Final.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-api-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-client-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-common-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-server-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/websocket-servlet-9.3.25.v20180904.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/woodstox-core-5.0.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xalan-2.7.2.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xbean-asm5-shaded-3.17.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xercesImpl-2.11.0.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xml-apis-1.4.01.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xml-apis-ext-1.3.04.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xmlenc-0.52.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xmlgraphics-commons-2.3.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/xz-1.6.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libtools/zookeeper.jar:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/libext/*.jar' org.apache.oozie.tools.OozieSharelibCLI create -fs hdfs://cm-r01nn02.mws.mds.xyz:8020 -locallib /opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/lib/oozie/oozie-sharelib-yarn -concurrency 8
log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender].
java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.apache.oozie.service.XLogService.init(XLogService.java:149)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.<init>(Services.java:111)
at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194)
at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97)
log4j:ERROR Could not instantiate appender named "redactorForRootLogger".
log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender].
java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.apache.oozie.service.XLogService.init(XLogService.java:149)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.<init>(Services.java:111)
at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194)
at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97)
log4j:ERROR Could not instantiate appender named "redactor2".
log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender].
java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.apache.oozie.service.XLogService.init(XLogService.java:149)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.<init>(Services.java:111)
at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194)
at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97)
log4j:ERROR Could not instantiate appender named "redactor1".
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.0-1.cdh6.2.0.p0.967373/jars/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" target="_blank">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender].
java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395)
at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:955)
at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89)
at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58)
at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:947)
at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:473)
at org.apache.oozie.service.XLogService.init(XLogService.java:178)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.<init>(Services.java:111)
at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194)
at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97)
log4j:ERROR Could not instantiate appender named "redactorForRootLogger".
log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender].
java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395)
at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:955)
at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89)
at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58)
at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:947)
at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:473)
at org.apache.oozie.service.XLogService.init(XLogService.java:178)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.<init>(Services.java:111)
at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194)
at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97)
log4j:ERROR Could not instantiate appender named "redactor2".
log4j:ERROR Could not instantiate class [org.cloudera.log4j.redactor.RedactorAppender].
java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768)
at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:395)
at org.apache.log4j.PropertyWatchdog.doOnChange(PropertyConfigurator.java:955)
at org.apache.log4j.helpers.FileWatchdog.checkAndConfigure(FileWatchdog.java:89)
at org.apache.log4j.helpers.FileWatchdog.<init>(FileWatchdog.java:58)
at org.apache.log4j.PropertyWatchdog.<init>(PropertyConfigurator.java:947)
at org.apache.log4j.PropertyConfigurator.configureAndWatch(PropertyConfigurator.java:473)
at org.apache.oozie.service.XLogService.init(XLogService.java:178)
at org.apache.oozie.service.Services.setServiceInternal(Services.java:386)
at org.apache.oozie.service.Services.<init>(Services.java:111)
at org.apache.oozie.tools.OozieSharelibCLI.run(OozieSharelibCLI.java:194)
at org.apache.oozie.tools.OozieSharelibCLI.main(OozieSharelibCLI.java:97)
log4j:ERROR Could not instantiate appender named "redactor1".
[14/Aug/2019 00:35:33 +0000] 18806 MainThread redactor INFO Killing with SIGTERM
Cheers, TK
... View more
Labels:
08-14-2019
04:49 AM
Switching to parcels worked. I could have just deployed things manually with rpm's but parcels make this much easier and so I switched to them. Posted a new issue however with Oozie. (Separate thread)
... View more
08-11-2019
09:49 PM
Spoke too soon. Manual install didn't do the trick. I'm actually uncertain which nodes the error message is coming from so unsure where to install Hue now. Will pick it up tomorrow. Cheers, TK
... View more
08-11-2019
09:43 PM
Having the same issue with HUE now even though the database connection works fine: ==> cloudera-scm-server.log <==
2019-08-12 00:18:37,663 INFO CommandPusher:com.cloudera.cmf.service.AbstractOneOffHostCommand: Unsuccessful 'HueTestDatabaseConnection'
2019-08-12 00:18:37,665 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: Command exited with code: 127
2019-08-12 00:18:37,665 INFO CommandPusher:com.cloudera.cmf.service.AbstractDbConnectionTestCommand: + find /var/run/cloudera-scm-agent/process/69-HUE-test-db-connection -type f '!' -path '/var/run/cloudera-scm-agent/process/69-HUE-test-db-connection/logs/*' '!' -name cloudera-config.sh '!' -name hue.sh '!' -name impala.sh '!' -name sqoop.sh '!' -name supervisor.conf '!' -name config.zip '!' -name proc.json '!' -name '*.log' '!' -name '*.keytab' '!' -name creds.localjceks -exec perl -pi -e 's#\{\{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/69-HUE-test-db-connection#g' '{}' ';'
Can't open /var/run/cloudera-scm-agent/process/69-HUE-test-db-connection/supervisor_status: Permission denied.
+ make_scripts_executable
+ find /var/run/cloudera-scm-agent/process/69-HUE-test-db-connection -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';'
+ '[' is_db_alive == beeswax_server ']'
+ set_classpath_in_var HADOOP_EXTRA_CLASSPATH_STRING
+ '[' -z HADOOP_EXTRA_CLASSPATH_STRING ']'
+ [[ -n /opt/cloudera/cm ]]
++ find /opt/cloudera/cm/lib/plugins -maxdepth 1 -name '*.jar'
++ tr '\n' :
+ ADD_TO_CP=/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar:/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar:
+ [[ -n '' ]]
+ eval 'OLD_VALUE=$HADOOP_EXTRA_CLASSPATH_STRING'
++ OLD_VALUE=
+ NEW_VALUE=/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar:/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar:
+ export HADOOP_EXTRA_CLASSPATH_STRING=/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar:/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar
+ HADOOP_EXTRA_CLASSPATH_STRING=/opt/cloudera/cm/lib/plugins/event-publish-6.2.0-shaded.jar:/opt/cloudera/cm/lib/plugins/tt-instrumentation-6.2.0.jar
+ HUE=/usr/lib/hue/build/env/bin/hue
+ [[ is_db_alive == runcpserver ]]
+ [[ is_db_alive == kt_renewer ]]
+ '[' dumpdata = is_db_alive ']'
+ '[' syncdb = is_db_alive ']'
+ '[' ldaptest = is_db_alive ']'
+ exec /usr/lib/hue/build/env/bin/hue is_db_alive
/opt/cloudera/cm-agent/service/hue/hue.sh: line 293: /usr/lib/hue/build/env/bin/hue: No such file or directory
2019-08-12 00:18:37,665 ERROR CommandPusher:com.cloudera.cmf.model.DbCommand: Command 1334(HueTestDatabaseConnection) has completed. finalstate:FINISHED, success:false, msg:Unexpected error. Unable to verify database connection.
==> cmf-server-nio.log <==
==> cloudera-scm-server.log <== I'll try to apply my solution but this is preventing cluster services installation: Hue Unexpected error. Unable to verify database connection. This causes a grayed out continue button, which makes it a real issue. I do see a missing package on the host where I'm running CM however I've forgotten how I configured the second namenode where the package exists now thought I'm sure I had installed all packages required on both nodes. 😞 Or perhaps something else should have installed the required packages? Can't remember exactly. Manually installed: hue-4.2.0+cdh6.2.0-967373.el7.x86_64 Dependencies Resolved
===========================================================================================================================
Package Arch Version Repository Size
===========================================================================================================================
Installing:
hue x86_64 4.2.0+cdh6.2.0-967373.el7 cloudera-cdh 78 M
Installing for dependencies:
avro-libs noarch 1.8.2+cdh6.2.0-967373.el7 cloudera-cdh 41 M
bigtop-jsvc x86_64 1.0.10+cdh6.2.0-967373.el7 cloudera-cdh 29 k
hadoop x86_64 3.0.0+cdh6.2.0-967373.el7 cloudera-cdh 103 M
hadoop-client x86_64 3.0.0+cdh6.2.0-967373.el7 cloudera-cdh 25 k
hadoop-hdfs x86_64 3.0.0+cdh6.2.0-967373.el7 cloudera-cdh 44 M
hadoop-mapreduce x86_64 3.0.0+cdh6.2.0-967373.el7 cloudera-cdh 20 M
hadoop-yarn x86_64 3.0.0+cdh6.2.0-967373.el7 cloudera-cdh 17 M
hive noarch 2.1.1+cdh6.2.0-967373.el7 cloudera-cdh 182 M
hive-jdbc noarch 2.1.1+cdh6.2.0-967373.el7 cloudera-cdh 42 M
libtidy x86_64 5.4.0-1.el7 epel 174 k
parquet noarch 1.9.0+cdh6.2.0-967373.el7 cloudera-cdh 80 M
parquet-format noarch 2.3.1+cdh6.2.0-967373.el7 cloudera-cdh 753 k
sentry noarch 2.1.0+cdh6.2.0-967373.el7 cloudera-cdh 101 M
solr noarch 7.4.0+cdh6.2.0-967373.el7 cloudera-cdh 64 M
zookeeper x86_64 3.4.5+cdh6.2.0-967373.el7 cloudera-cdh 4.0 M
Transaction Summary
===========================================================================================================================
Install 1 Package (+15 Dependent packages) Shouldn't Cloudera distribute the packages? Just curious. Or perhaps I forgot a step somewhere? Cheers, TK
... View more
07-25-2019
10:58 PM
So I've moved the HAproxy and Keepalived to another set of hosts. Same issue with certs however I did try to replace the cm-r01nn01 and cm-r01nn02 (Name Node's) with the SAN SSL Certificate that I've generated. Here is the writeup: ==> /var/log/cloudera-scm-agent/cloudera-scm-agent.log <==
self.cfg.max_cert_depth)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
self.conn.connect()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
sock.connect((self.host, self.port))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
ret = self.connect_ssl()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
return m2.ssl_connect(self.ssl, self._timeout)
SSLError: certificate verify failed Replace the certificate cm-r01nn01 and cm-r01nn02 with the SAN SSL Cert one as follows: [root@cm-r01nn01 yum.repos.d]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"
cm-r01wn07.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01wn04.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01wn01.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01wn08.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01nn01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
cm-r01wn05.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01wn02.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01en01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
cm-r01wn06.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
cm-r01wn03.mws.mds.xyz, Jul 13, 2019, PrivateKeyEntry,
srv-c01.mws.mds.xyz, Jul 18, 2019, PrivateKeyEntry,
cm-r01en02.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
cm-c01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 yum.repos.d]#
[root@cm-r01nn01 yum.repos.d]#
[root@cm-r01nn01 yum.repos.d]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"|grep -Ei r01nn01
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
cm-r01nn01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
[root@cm-r01nn01 yum.repos.d]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn01.mws.mds.xyz ^C
[root@cm-r01nn01 yum.repos.d]#
[root@cm-r01nn01 yum.repos.d]#
[root@cm-r01nn01 yum.repos.d]#
[root@cm-r01nn01 yum.repos.d]# cd /root/srv-c01/
[root@cm-r01nn01 srv-c01]# ls -altri
total 40
201326721 dr-xr-x---. 11 root root 4096 Jul 17 22:04 ..
134330464 drwxr-xr-x. 2 root root 4096 Jul 18 22:39 1
2230259 -rw-r--r--. 1 root root 2422 Jul 18 22:46 srv-c01.mws.mds.xyz.keystore.jks
2230300 -rw-r--r--. 1 root root 2801 Jul 18 22:49 srv-c01.mws.mds.xyz.keystore.p12
2230651 -rw-r--r--. 1 root root 1763 Jul 18 22:49 srv-c01.mws.mds.xyz.cert.pem
2230652 -rw-r--r--. 1 root root 1860 Jul 18 22:49 srv-c01.mws.mds.xyz.key.pem
2230653 -rw-r--r--. 1 root root 1505 Jul 18 22:51 srv-c01.mws.mds.xyz.pem
2230654 -rw-r--r--. 1 root root 1679 Jul 18 23:00 srv-c01.mws.mds.xyz.key.nopass.pem
2230257 drwxr-xr-x. 3 root root 4096 Jul 18 23:00 .
2230655 -rw-r--r--. 1 root root 3442 Jul 18 23:00 srv-c01.mws.mds.xyz-haproxy.pem
[root@cm-r01nn01 srv-c01]# history|grep srv-c01.mws.mds.xyz.pem|grep openssl|tail
876 openssl x509 -in ./srv-c01.mws.mds.xyz.pem -noout -text
1029 history|grep srv-c01.mws.mds.xyz.pem|grep openssl|tail
[root@cm-r01nn01 srv-c01]# openssl x509 -in ./srv-c01.mws.mds.xyz.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 1594172762 (0x5f05255a)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, ST=California, L=Los Angeles, O=MDS, OU=MDS, CN=srv-c01.mws.mds.xyz
Validity
Not Before: Jul 19 02:46:18 2019 GMT
Not After : Jul 16 02:46:18 2029 GMT
Subject: C=US, ST=California, L=Los Angeles, O=MDS, OU=MDS, CN=srv-c01.mws.mds.xyz
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:c5:a9:00:83:12:9e:02:86:32:4e:2b:a7:c6:1a:
6b:9d:e3:56:00:53:22:01:d8:db:83:cd:14:79:6a:
85:27:20:f6:5d:86:0e:0b:af:df:46:dd:c3:23:72:
f0:bf:38:3e:cd:9f:92:e6:65:81:7b:26:32:50:fc:
81:0e:7b:dd:b4:61:6f:a7:56:ec:c8:fe:89:72:ec:
e5:e0:63:61:92:77:0b:36:41:98:93:14:6d:53:a0:
24:fb:fb:77:40:98:5b:2f:d2:3c:65:4f:8b:65:33:
e5:db:14:ce:01:d2:4f:9f:e4:c6:c8:35:50:09:a2:
f3:48:0a:ac:06:fd:66:42:30:10:a4:e7:fa:a8:2b:
0b:2b:ef:ce:83:82:4e:0d:86:34:ce:0c:8d:0c:a2:
f5:88:4d:38:9f:3b:dd:2e:6e:e3:8c:60:69:da:8d:
a4:d4:db:d5:cd:26:91:95:ca:a2:47:de:3c:f3:8f:
52:b8:e5:b0:09:26:af:77:fb:a3:5b:40:f6:e8:1b:
66:d7:b7:1b:da:2c:6c:34:99:76:de:c4:9b:80:69:
25:d5:12:2f:cb:9b:c5:d2:7e:15:a7:50:5f:54:5c:
9d:6b:8c:c0:9c:03:3f:96:f3:8a:2c:a6:05:ec:a4:
d3:83:84:61:13:da:57:6d:e8:8c:93:d9:40:38:24:
96:c9
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication, Code Signing, E-mail Protection, Time Stamping, OCSP Signing
X509v3 Subject Alternative Name:
DNS:srv-c01.mws.mds.xyz, DNS:cm-r01nn01.mws.mds.xyz, DNS:cm-r01nn02.mws.mds.xyz
X509v3 Subject Key Identifier:
F6:EA:97:6F:82:20:84:75:E9:63:71:2F:16:D6:41:8B:64:05:07:0D
Signature Algorithm: sha256WithRSAEncryption
4f:35:6d:18:dc:5c:4a:65:db:8c:62:75:0b:f8:da:2b:14:72:
22:f7:3a:ba:15:17:58:41:46:3b:6b:6e:40:db:6b:be:e5:07:
82:d1:37:0a:d6:4e:96:14:f6:87:ca:ff:d3:5f:a9:94:de:81:
e7:a1:28:94:0a:19:0b:f4:dc:ed:0a:a5:77:78:20:53:3f:3f:
03:54:67:a0:c4:a1:de:49:7d:e8:fc:2d:76:bd:7b:a5:98:cd:
45:7e:ba:21:79:e2:91:7d:f3:e9:d6:5d:b7:91:34:30:3a:e4:
3a:38:e9:33:9b:26:2e:3e:6c:c9:3d:5d:48:81:cb:35:2f:ff:
7a:ff:22:c2:f8:b5:a2:01:d0:54:7f:f2:08:33:89:78:80:af:
72:2d:d7:df:61:f0:4a:7f:d2:19:0d:c6:0c:51:ee:4e:c1:ed:
8d:8b:4f:82:17:47:6b:03:1a:f2:8b:00:cc:17:8a:75:ca:72:
c0:a4:a7:12:87:32:16:89:15:2c:80:d1:07:fd:37:e8:bf:f5:
87:6b:a2:dd:9d:a4:c4:2c:68:f8:d9:15:dd:3c:40:6d:8b:e0:
6d:c4:87:6d:39:a9:6b:91:f6:0a:bc:7c:63:e7:f0:37:cb:7a:
5f:35:6c:5c:f9:bb:cb:58:1a:b9:9c:49:ab:24:ac:2a:c9:2d:
3f:b2:2f:68
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"|grep -Ei r01nn01
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
cm-r01nn01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"|grep -Ei r01nn01
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
cm-r01nn01.mws.mds.xyz, Jul 12, 2019, PrivateKeyEntry,
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn01.mws.mds.xyz
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn02.mws.mds.xyz
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore srv-c01.mws.mds.xyz.keystore.p12 -deststoretype PKCS12 -destalias cm-r01nn01.mws.mds.xyz -srcalias srv-c01.mws.mds.xyz -deststorepass srv-c01.mws.mds.xyz -destkeypass srv-c01.mws.mds.xyz
Importing keystore srv-c01.mws.mds.xyz.keystore.jks to srv-c01.mws.mds.xyz.keystore.p12...
Enter source keystore password:
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv"|grep -Ei r01nn
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn01.mws.mds.xyz
keytool error: java.lang.Exception: Alias <cm-r01nn01.mws.mds.xyz> does not exist
[root@cm-r01nn01 srv-c01]# keytool -list -keystore srv-c01.mws.mds.xyz.keystore.p12
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 2 entries
srv-c01.mws.mds.xyz, Jul 18, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): 01:2D:6C:36:E6:7D:76:2B:45:66:91:7F:E8:B9:C2:61:A4:22:FB:D6
cm-r01nn01.mws.mds.xyz, Jul 26, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): 01:2D:6C:36:E6:7D:76:2B:45:66:91:7F:E8:B9:C2:61:A4:22:FB:D6
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias cm-r01nn01.mws.mds.xyz
keytool error: java.lang.Exception: Alias <cm-r01nn01.mws.mds.xyz> does not exist
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -destalias cm-r01nn01.mws.mds.xyz
Enter alias name: cm-r01nn01.mws.mds.xyz
keytool error: java.lang.Exception: Alias <cm-r01nn01.mws.mds.xyz> does not exist
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass cm-r01nn01.mws.mds.xyz -alias cm-r01nn01.mws.mds.xyz
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass srv-c01.mws.mds.xyz -alias cm-r01nn01.mws.mds.xyz
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect
[root@cm-r01nn01 srv-c01]# keytool -delete -keystore srv-c01.mws.mds.xyz.keystore.p12 -storepass srv-c01.mws.mds.xyz -alias cm-r01nn01.mws.mds.xyz [root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -list -keystore srv-c01.mws.mds.xyz.keystore.p12 Enter keystore password:
keytool error: java.io.IOException: keystore password was incorrect
[root@cm-r01nn01 srv-c01]# keytool -list -keystore srv-c01.mws.mds.xyz.keystore.p12
Enter keystore password:
Keystore type: PKCS12
Keystore provider: SUN
Your keystore contains 1 entry
srv-c01.mws.mds.xyz, Jul 18, 2019, PrivateKeyEntry,
Certificate fingerprint (SHA1): 01:2D:6C:36:E6:7D:76:2B:45:66:91:7F:E8:B9:C2:61:A4:22:FB:D6
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# history|grep import|grep srv-c01.mws.mds.xyz.keystore.p12
699 keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore srv-c01.mws.mds.xyz.keystore.p12 -deststoretype PKCS12 -srcalias srv-c01.mws.mds.xyz -deststorepass srv-c01.mws.mds.xyz
869 keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore srv-c01.mws.mds.xyz.keystore.p12 -deststoretype PKCS12 -srcalias srv-c01.mws.mds.xyz -deststorepass srv-c01.mws.mds.xyz -destkeypass srv-c01.mws.mds.xyz
1034 keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore srv-c01.mws.mds.xyz.keystore.p12 -deststoretype PKCS12 -destalias cm-r01nn01.mws.mds.xyz -srcalias srv-c01.mws.mds.xyz -deststorepass srv-c01.mws.mds.xyz -destkeypass srv-c01.mws.mds.xyz
1044 history|grep import|grep srv-c01.mws.mds.xyz.keystore.p12
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -srcalias srv-c01.mws.mds.xyz -deststorepass changeit -srcstorepass srv-c01.mws.mds.xyz -destalias cm-r01nn01.mws.mds.xyz
Importing keystore srv-c01.mws.mds.xyz.keystore.jks to /etc/pki/ca-trust/extracted/java/jssecacerts...
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -srcalias srv-c01.mws.mds.xyz -deststorepass changeit -srcstorepass srv-c01.mws.mds.xyz -destalias cm-r01nn02.mws.mds.xyz
Importing keystore srv-c01.mws.mds.xyz.keystore.jks to /etc/pki/ca-trust/extracted/java/jssecacerts...
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]#
[root@cm-r01nn01 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit -alias srv-c01.mws.mds.xyz -alias cm-r01nn01.mws.mds.xyz -v
Alias name: cm-r01nn01.mws.mds.xyz
Creation date: Jul 26, 2019
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=srv-c01.mws.mds.xyz, OU=MDS, O=MDS, L=Los Angeles, ST=California, C=US
Issuer: CN=srv-c01.mws.mds.xyz, OU=MDS, O=MDS, L=Los Angeles, ST=California, C=US
Serial number: 5f05255a
Valid from: Thu Jul 18 22:46:18 EDT 2019 until: Sun Jul 15 22:46:18 EDT 2029
Certificate fingerprints:
MD5: A7:C0:9E:E4:CC:DC:7E:4B:3A:96:CF:11:58:6C:86:D2
SHA1: 01:2D:6C:36:E6:7D:76:2B:45:66:91:7F:E8:B9:C2:61:A4:22:FB:D6
SHA256: 62:11:52:4E:7C:1E:03:11:20:CA:41:7E:5D:22:64:F8:CE:CC:85:C6:07:06:A9:21:FE:25:F3:71:DD:20:00:49
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Extensions:
#1: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
clientAuth
codeSigning
emailProtection
timeStamping
OCSPSigning
]
#2: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: srv-c01.mws.mds.xyz
DNSName: cm-r01nn01.mws.mds.xyz
DNSName: cm-r01nn02.mws.mds.xyz
]
#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: F6 EA 97 6F 82 20 84 75 E9 63 71 2F 16 D6 41 8B ...o. .u.cq/..A.
0010: 64 05 07 0D d...
]
]
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01nn01 srv-c01]#
[root@cm-r01wn08 srv-c01]#
[root@cm-r01wn08 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -srcalias srv-c01.mws.mds.xyz -deststorepass changeit -srcstorepass srv-c01.mws.mds.xyz -destalias cm-r01nn01.mws.mds.xyz
Importing keystore srv-c01.mws.mds.xyz.keystore.jks to /etc/pki/ca-trust/extracted/java/jssecacerts...
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01wn08 srv-c01]# keytool -importkeystore -srckeystore srv-c01.mws.mds.xyz.keystore.jks -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -srcalias srv-c01.mws.mds.xyz -deststorepass changeit -srcstorepass srv-c01.mws.mds.xyz -destalias cm-r01nn02.mws.mds.xyz
Importing keystore srv-c01.mws.mds.xyz.keystore.jks to /etc/pki/ca-trust/extracted/java/jssecacerts...
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01wn08 srv-c01]#
[root@cm-r01wn08 srv-c01]#
[root@cm-r01wn08 srv-c01]#
[root@cm-r01wn08 srv-c01]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts -storepass changeit | grep -Ei "cm|srv" srv-c01.mws.mds.xyz, Jul 21, 2019, PrivateKeyEntry,
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01wn08.mws.mds.xyz, Jul 10, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Jul 26, 2019, PrivateKeyEntry,
cm-r01nn01.mws.mds.xyz, Jul 26, 2019, PrivateKeyEntry,
Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/jssecacerts -destkeystore /etc/pki/ca-trust/extracted/java/jssecacerts -deststoretype pkcs12".
[root@cm-r01wn08 srv-c01]#
[root@cm-r01wn08 srv-c01]# But no luck. Again, is there any way to increase the debug level on the cloudera-scm-agent so it will print further information detailing what private and public key files it's opening up? Thx, TK
... View more
07-21-2019
10:01 PM
Received the subject error when I replaced the certs with the SAN one that contained 3 hosts: keytool -genkeypair -alias cm-c01.mws.mds.xyz -keyalg RSA -keysize 2048 -dname "cn=cm-c01.mws.mds.xyz,OU=MDS,O=MDS,L=Los Angeles,ST=California,C=US" -keypass cm-c01.mws.mds.xyz -keystore cm-c01.mws.mds.xyz.keystore.jks -storepass cm-c01.mws.mds.xyz -validity 3650 -ext EKU=serverAuth,clientAuth,codeSigning,emailProtection,timeStamping,OCSPSigning -ext san=dns:cm-c01.mws.mds.xyz,dns:cm-r01nn01.mws.mds.xyz,dns:cm-r01nn02.mws.mds.xyz Updated jssecerts (in path above) with the new cert as well. Same issue. I'm running the load balancer and haproxy on the cm-r01nn01/02 . This is a problem since Cloudera opens up ports such as 7180, 7182 and 7183 on all available interfaces. So if I have a VIP running on the same host, Cloudera services try and do bind to it. Can't really have HAproxy running on port 7183 on the LB VIP if Cloudera services are already bound to the same port. Tried the Cloudera Manager Hostname Override but only got these errors: 12:25:32.076 AM WARN BasicScmProxy
Exception while getting fetch configDefaults hash: none
java.io.IOException: HTTPS hostname wrong: should be <srv-c01.mws.mds.xyz>
at sun.net.www.protocol.https.HttpsClient.checkURLSpoofing(HttpsClient.java:649)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:573)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:259)
at com.cloudera.cmf.BasicScmProxy.authenticate(BasicScmProxy.java:277)
at com.cloudera.cmf.BasicScmProxy.fetch(BasicScmProxy.java:607)
at com.cloudera.cmf.BasicScmProxy.getFragmentAndHash(BasicScmProxy.java:696)
at com.cloudera.cmf.DescriptorAndFragments.newDescriptorAndFragments(DescriptorAndFragments.java:65)
at com.cloudera.cmon.firehose.Main.main(Main.java:396) Need to spend more time reading on the LB's and CM / CDH. Thx, TK
... View more
07-17-2019
06:47 PM
Breakdown of what I'm getting with different configs: 1) frontend srvin
log 127.0.0.1 local0 debug
bind srv-c01:17182 ssl crt /etc/haproxy/certs/srv-c01.mws.mds.xyz-haproxy.pem no-sslv3
option tcplog
default_backend srvback
backend srvback
log /dev/log local0 debug
mode tcp
option tcplog
balance roundrobin
server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7182 check
server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7182 check
Results in: [17/Jul/2019 21:12:12 +0000] 25588 MainThread agent ERROR Heartbeating to srv-c01.mws.mds.xyz:17182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1396, in _send_heartbeat
response = self.requestor.request('heartbeat', heartbeat_data)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
return self.issue_request(call_request, message_name, request_datum)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
call_response = self.transceiver.transceive(call_request)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
result = self.read_framed_message()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 489, in read_framed_message
framed_message = response_reader.read_framed_message()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 417, in read_framed_message
raise ConnectionClosedException("Reader read 0 bytes.")
ConnectionClosedException: Reader read 0 bytes. 2) frontend srvin
log 127.0.0.1 local0 debug
bind srv-c01:17182
option tcplog
default_backend srvback
backend srvback
log /dev/log local0 debug
mode tcp
option tcplog
balance roundrobin
server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7182 check
server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7182 check Results in: [17/Jul/2019 21:15:23 +0000] 25588 MainThread agent ERROR Heartbeating to srv-c01.mws.mds.xyz:17182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
self.cfg.max_cert_depth)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
self.conn.connect()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
sock.connect((self.host, self.port))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 313, in connect
if not check(self.get_peer_cert(), self.addr[0]):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Checker.py", line 125, in __call__
fieldName='subjectAltName')
WrongHost: Peer certificate subjectAltName does not match host, expected srv-c01.mws.mds.xyz, got DNS:cm-r01nn01.mws.mds.xyz 3) frontend srvin
log 127.0.0.1 local0 debug
bind srv-c01:17182 ssl crt /etc/haproxy/certs/srv-c01.mws.mds.xyz-haproxy.pem no-sslv3
default_backend srvback
backend srvback
log /dev/log local0 debug
mode http
balance roundrobin
cookie srv-c01 insert indirect nocache
server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 cookie cm-r01nn01.mws.mds.xyz sni req.hdr(host)
server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 cookie cm-r01nn02.mws.mds.xyz sni req.hdr(host)
Results in: parsing [/etc/haproxy/haproxy.cfg:69] : 'server cm-r01nn02.mws.mds.xyz' unknown keyword 'sni'. Config uses a pem file with certs from the two nodes + the VIP ( concatenated from srv-c01, cm-r01nn01 and cm-r01nn02 ). The certs appear to work ok. The traffic isn't passing through, however. Config looks like this: [root@cm-r01wn08 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -v "#" | sed -e "/^$/d"
[General]
server_host=srv-c01.mws.mds.xyz
server_port=17182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=DEBUG
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/pki/cluster-vip.pem
client_key_file=/opt/cloudera/security/pki/client-key.pem
client_keypw_file=/opt/cloudera/security/pki/agent.pw
client_cert_file=/opt/cloudera/security/pki/client-cert.pem
[Hadoop]
[Cloudera]
[JDBC]
[Cgroup_Paths]
[root@cm-r01wn08 pki]# I would like to keep it pointed to the VIP because this guarantees the config will be identical on all hosts and ready for any kind of failover. Thx, TK
... View more
07-17-2019
03:42 PM
Tried. Certificates appeared fine ( were recognized ). The issue appears to be between the Load Balancer VIP and HAproxy right now since I get this: [17/Jul/2019 03:56:22 +0000] 20834 MainThread agent ERROR Heartbeating to srv-c01.mws.mds.xyz:17182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1396, in _send_heartbeat
response = self.requestor.request('heartbeat', heartbeat_data)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 141, in request
return self.issue_request(call_request, message_name, request_datum)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 254, in issue_request
call_response = self.transceiver.transceive(call_request)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 483, in transceive
result = self.read_framed_message()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 489, in read_framed_message
framed_message = response_reader.read_framed_message()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/avro/ipc.py", line 417, in read_framed_message
raise ConnectionClosedException("Reader read 0 bytes.")
ConnectionClosedException: Reader read 0 bytes. This is really telling me, and please keep me honest, that the traffic isn't being passed between the VIP and the backend servers defined in the HAproxy config (see below). Currently trying to solve the above by incorporating what this page mentions, however, I need a newer HAproxy version since mine doesn't support SNI. My latest HAproxy config is as follows to try and solve the above issue by setting up HAproxy in TLS bridging mode: frontend srvin
log 127.0.0.1 local0 debug
bind srv-c01:17182 ssl crt /etc/haproxy/certs/srv-c01.mws.mds.xyz-haproxy.pem no-sslv3
default_backend srvback
backend srvback
log /dev/log local0 debug
mode http
balance roundrobin
cookie srv-c01 insert indirect nocache
server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 cookie cm-r01nn01.mws.mds.xyz sni req.hdr(host)
server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3 cookie cm-r01nn02.mws.mds.xyz sni req.hdr(host)
Current error is as follows but that's an HAproxy problem now that I'm following up on separately: [ALERT] 197/040530 (7560) : parsing [/etc/haproxy/haproxy.cfg:69] : 'server cm-r01nn02.mws.mds.xyz' unknown keyword 'sni'. Question I had above is, is there any other page other than the following that demonstrates the use of a VIP w/ HAproxy and TLS config when load balancing Cloudera services? Thx, TK
... View more
07-16-2019
08:18 PM
How do I get SSL to work properly with CM and port 7182 through a VIP provided by a Load Balancer? A visualization: https://ibb.co/hY0GsVY
... View more
07-15-2019
09:59 PM
Thanks very much again for taking the time and explain here. I've got this part working as well. In the meantime, I'm looking to enable high availability on the cluster. Have a few questions in this regard. 1) HAproxy is given as an example. I've used Haproxy + Keepalived for the CMS (7183) and a custom DNS entry, cm-c01.mws.mds.xyz to point to the cluster VIP. Everything works, including the certs for the UI. UI correctly displays the SSL certs for cm-c01 rather then the constituent hosts providing the backend. 2) I tried the same process with the Agent Avro port 7182 (If I'm calling it that correctly). I've set up a VIP, configured HAproxy and the proper SSL certs for srv-c01.mws.mds.xyz. This doesn't work. I've imported the right certs into the jssecerts file as well. HAproxy config: frontend cmin
bind cm-c01:443 ssl crt /etc/haproxy/certs/cm-c01.mws.mds.xyz-haproxy.pem no-sslv3
default_backend cmback
backend cmback
mode http
balance roundrobin
server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7183 ssl check verify none port 7183 inter 12000 rise 3 fall 3
server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7183 ssl check verify none port 7183 inter 12000 rise 3 fall 3
frontend srvin
log 127.0.0.1 local0 debug
bind srv-c01:17182 ssl crt /etc/haproxy/certs/srv-c01.mws.mds.xyz-haproxy.pem no-sslv3
default_backend srvback
backend srvback
log /dev/log local0 debug
mode http
balance roundrobin
server cm-r01nn01.mws.mds.xyz cm-r01nn01.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3
server cm-r01nn02.mws.mds.xyz cm-r01nn02.mws.mds.xyz:7182 ssl check verify none port 7182 inter 12000 rise 3 fall 3
Each PEM file has a private and public key. How could I get this working with Self-Signed SSL certs? Do I need to revert the HAproxy config to a tcp pass-through? If so, how will I handle the case when it fails over and needs the cert file of the second host? Do I combine the public pem file certs? Thx, TK
... View more
07-11-2019
08:47 PM
The error messages I get from the agent when I attempt (3): ==> /var/log/cloudera-scm-agent/status-stderr.log <==
[11/Jul/2019:23:37:40] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cheroot/server.py", line 1339, in start
self.tick()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cheroot/server.py", line 1408, in tick
s, ssl_env = self.ssl_adapter.wrap(s)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/status_server.py", line 1048, in wrap
ssl.accept_ssl()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 258, in accept_ssl
return m2.ssl_accept(self.ssl, self._timeout)
SSLError: unexpected eof [11/Jul/2019 23:37:43 +0000] 8193 MainThread agent ERROR Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
self.cfg.max_cert_depth)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
self.conn.connect()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
sock.connect((self.host, self.port))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
ret = self.connect_ssl()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
return m2.ssl_connect(self.ssl, self._timeout)
SSLError: sslv3 alert certificate unknown The pressing question I have above all others is how do I get the agent to print out enough logging to tell me WHICH certificate it attempted to load so I know the context under which the above is thrown? Right now, given the above error, I can't really take action without knowing the exact file the exceptions are referring too, other then deduce based on the change I've done. My setup as it is now: [ cm-r01en01 ] ( Utility Server ) [root@cm-r01en01 pki]# ls -altri
total 32
3870062 drwxr-xr-x. 5 root root 37 Apr 1 23:09 ..
69943169 lrwxrwxrwx. 1 root root 62 Apr 1 23:11 server.jks -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.keystore.jks
69943167 -rw-r--r--. 1 cloudera-scm cloudera-scm 3449 Jul 8 03:32 cm-r01en01.mws.mds.xyz.keystore.jks
67257528 -rw-r--r--. 1 cloudera-scm cloudera-scm 2775 Jul 9 23:51 cm-r01en01.mws.mds.xyz.keystore.p12
67586231 -rw-r--r--. 1 cloudera-scm cloudera-scm 1720 Jul 9 23:52 cm-r01en01.mws.mds.xyz.cert.pem
71202518 -rw-r--r--. 1 cloudera-scm cloudera-scm 1863 Jul 9 23:53 cm-r01en01.mws.mds.xyz.key.pem
71305735 -r--r-----. 1 cloudera-scm cloudera-scm 23 Jul 9 23:53 client-agent.pw
67257529 lrwxrwxrwx. 1 root root 30 Jul 9 23:55 client-key.pem -> cm-r01en01.mws.mds.xyz.key.pem
71202519 lrwxrwxrwx. 1 root root 31 Jul 9 23:55 client-cert.pem -> cm-r01en01.mws.mds.xyz.cert.pem
71305736 -rw-r--r--. 1 cloudera-scm cloudera-scm 1432 Jul 10 20:27 cm-r01en01.mws.mds.xyz.pem
71305755 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Jul 10 22:14 cm-r01nn01.mws.mds.xyz.pem
69943176 lrwxrwxrwx. 1 root root 53 Jul 10 22:14 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
69943151 drwxr-xr-x. 2 root root 4096 Jul 11 23:01 .
[root@cm-r01en01 pki]# hostname
cm-r01en01.mws.mds.xyz
[root@cm-r01en01 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep -Ei cm
Enter keystore password: changeit
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
cm-r01en01.mws.mds.xyz, Jul 11, 2019, trustedCertEntry,
cm-r01nn01.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
[root@cm-r01en01 pki]# [root@cm-r01en01 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -Eiv "#"|sed -e "/^$/d"
[General]
server_host=cm-r01nn01.mws.mds.xyz
server_port=7182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=logging.DEBUG
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/pki/agent.pem
client_key_file=/opt/cloudera/security/pki/client-key.pem
client_keypw_file=/opt/cloudera/security/pki/client-agent.pw
client_cert_file=/opt/cloudera/security/pki/client-cert.pem
[Hadoop]
[Cloudera]
[JDBC]
[Cgroup_Paths]
[root@cm-r01en01 pki]# [ cm-r01en02 ] ( Utility Server ) [root@cm-r01en02 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep -Ei cm
Enter keystore password: changeit
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Jul 10, 2019, trustedCertEntry,
cm-r01en02.mws.mds.xyz, Jul 10, 2019, trustedCertEntry,
cm-r01nn01.mws.mds.xyz, Jul 10, 2019, trustedCertEntry,
You have new mail in /var/spool/mail/root
[root@cm-r01en02 pki]# ls -altri
total 28
135616270 drwxr-xr-x. 5 root root 37 Jul 10 21:28 ..
335605256 -rw-r--r--. 1 cloudera-scm cloudera-scm 2386 Jul 10 21:29 cm-r01en02.mws.mds.xyz.keystore.jks
335605249 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Jul 10 21:51 cm-r01en02.mws.mds.xyz.pem
335605265 lrwxrwxrwx. 1 root root 62 Jul 10 21:56 server.jks -> /opt/cloudera/security/pki/cm-r01en02.mws.mds.xyz.keystore.jks
335605275 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Jul 10 22:14 cm-r01nn01.mws.mds.xyz.pem
335605382 lrwxrwxrwx. 1 root root 53 Jul 10 22:14 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
335605420 -rw-r--r--. 1 cloudera-scm cloudera-scm 2775 Jul 11 23:09 cm-r01en02.mws.mds.xyz.keystore.p12
335605426 -rw-r--r--. 1 cloudera-scm cloudera-scm 1720 Jul 11 23:10 cm-r01en02.mws.mds.xyz.cert.pem
335605425 -rw-r--r--. 1 cloudera-scm cloudera-scm 1863 Jul 11 23:11 cm-r01en02.mws.mds.xyz.key.pem
335605429 lrwxrwxrwx. 1 root root 30 Jul 11 23:12 client-key.pem -> cm-r01en02.mws.mds.xyz.key.pem
335605430 lrwxrwxrwx. 1 root root 31 Jul 11 23:12 client-cert.pem -> cm-r01en02.mws.mds.xyz.cert.pem
335860926 drwxr-xr-x. 2 root root 4096 Jul 11 23:12 .
[root@cm-r01en02 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -Eiv "#"|sed -e "/^$/d"
[General]
server_host=cm-r01nn01.mws.mds.xyz
server_port=7182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=logging.DEBUG
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/pki/agent.pem
client_key_file=/opt/cloudera/security/pki/client-key.pem
client_keypw_file=/opt/cloudera/security/pki/agent.pw
client_cert_file=/opt/cloudera/security/pki/client-cert.pem
[Hadoop]
[Cloudera]
[JDBC]
[Cgroup_Paths]
You have new mail in /var/spool/mail/root
[root@cm-r01en02 pki]# [ cm-r01nn01 ] (Name Node) [root@cm-r01nn01 pki]# keytool -list -keystore /etc/pki/ca-trust/extracted/java/jssecacerts|grep -Ei cm Enter keystore password: changeit
acraizfnmt-rcm, Mar 26, 2019, trustedCertEntry,
cm-r01nn02.mws.mds.xyz, Apr 14, 2019, trustedCertEntry,
cm-c01.mws.mds.xyz, Jul 6, 2019, trustedCertEntry,
cm-r01nn01.mws.mds.xyz, Mar 31, 2019, trustedCertEntry,
[root@cm-r01nn01 pki]#
[root@cm-r01nn01 pki]#
[root@cm-r01nn01 pki]# cd /opt/cloudera/security/pki
[root@cm-r01nn01 pki]# ls -altri
total 32
201424378 drwxr-xr-x. 5 root root 37 Mar 31 23:42 ..
135962833 lrwxrwxrwx. 1 root root 62 Mar 31 23:45 server.jks -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.keystore.jks
135962832 -rw-r--r--. 1 cloudera-scm cloudera-scm 4723 Jul 6 00:00 cm-r01nn01.mws.mds.xyz.keystore.jks
135283830 -rw-r--r--. 1 root root 1435 Jul 6 00:10 cm-c01.mws.mds.xyz.pem
135044185 -rw-r--r--. 1 root root 2751 Jul 8 03:49 cm-r01nn01.mws.mds.xyz.keystore.p12
135044187 -rw-r--r--. 1 root root 1691 Jul 8 03:50 cm-r01nn01.mws.mds.xyz.cert.pem
135044188 -rw-r--r--. 1 root root 1859 Jul 8 03:50 cm-r01nn01.mws.mds.xyz.key.pem
135962830 -rw-r--r--. 1 root root 1453 Jul 10 22:14 cm-r01nn01.mws.mds.xyz.pem
135962831 lrwxrwxrwx. 1 root root 53 Jul 10 22:14 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
135962829 drwxr-xr-x. 2 root root 4096 Jul 10 22:14 .
[root@cm-r01nn01 pki]# cat /etc/cloudera-scm-agent/config.ini|grep -Eiv "#"|sed -e "/^$/d"
[General]
server_host=cm-r01nn01.mws.mds.xyz
server_port=7182
max_collection_wait_seconds=10.0
metrics_url_timeout_seconds=30.0
task_metrics_timeout_seconds=5.0
monitored_nodev_filesystem_types=nfs,nfs4,tmpfs
local_filesystem_whitelist=ext2,ext3,ext4,xfs
impala_profile_bundle_max_bytes=1073741824
stacks_log_bundle_max_bytes=1073741824
stacks_log_max_uncompressed_file_size_bytes=5242880
orphan_process_dir_staleness_threshold=5184000
orphan_process_dir_refresh_interval=3600
scm_debug=logging.DEBUG
dns_resolution_collection_interval_seconds=60
dns_resolution_collection_timeout_seconds=30
[Security]
use_tls=1
max_cert_depth=9
verify_cert_file=/opt/cloudera/security/pki/agent.pem
[Hadoop]
[Cloudera]
[JDBC]
[Cgroup_Paths]
[root@cm-r01nn01 pki]# NOTE: I haven't reconfigured the /etc/cloudera-scm-agent/config.ini yet on this (NN) node since the two other nodes (EN) aren't working anyway. Thx, TK
... View more
07-11-2019
07:46 PM
Got steps 1 and 2 working. Tyvm. For step 3, I'm working on it now in combination with your post above and noticing this page might not be correct: https://www.cloudera.com/documentation/enterprise/latest/topics/how_to_configure_cm_tls.html#concept_xtp_q3w_wn "Cloudera Manager TLS/SSL Certificate Trust Store File" doesn't appear under Administration -> Settings. Thx, TK
... View more
07-09-2019
05:41 PM
Pointed agent.pem as follows: [root@cm-r01en01 pki]# ls -altri
total 36
69943257 -rw-r--r--. 1 cloudera-scm cloudera-scm 2385 Apr 1 23:06 cm-r01en01.mws.mds.xyz.keystore.jks-backup
69943152 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Apr 1 23:07 cm-r01en01.mws.mds.xyz.pem
3870062 drwxr-xr-x. 5 root root 37 Apr 1 23:09 ..
69943169 lrwxrwxrwx. 1 root root 62 Apr 1 23:11 server.jks -> /opt/cloudera/security/pki/cm-r01en01.mws.mds.xyz.keystore.jks
69943259 -rw-r--r--. 1 cloudera-scm cloudera-scm 1453 Jul 6 20:01 cm-r01nn01.mws.mds.xyz.pem
69943154 lrwxrwxrwx. 1 root root 53 Jul 6 20:02 rootca.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
69943167 -rw-r--r--. 1 cloudera-scm cloudera-scm 3449 Jul 8 03:32 cm-r01en01.mws.mds.xyz.keystore.jks
71202464 -rw-r--r--. 1 cloudera-scm cloudera-scm 1415 Jul 8 03:52 cm-r01nn01.mws.mds.xyz.cert.pem
71202517 -rw-r--r--. 1 cloudera-scm cloudera-scm 1859 Jul 8 03:53 cm-r01nn01.mws.mds.xyz.key.pem
71305734 -rw-r--r--. 1 cloudera-scm cloudera-scm 3119 Jul 8 03:56 cm-r01nn01.mws.mds.xyz.cert.key.pem
67257529 -rw-r--r--. 1 cloudera-scm cloudera-scm 23 Jul 8 03:57 agent.pw
67689060 lrwxrwxrwx. 1 root root 53 Jul 9 20:21 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
69943151 drwxr-xr-x. 2 root root 4096 Jul 9 20:21 .
[root@cm-r01en01 pki]# pwd
/opt/cloudera/security/pki
[root@cm-r01en01 pki]# Now the error is: [09/Jul/2019 20:22:46 +0000] 19927 MainThread agent ERROR Heartbeating to cm-r01nn01.mws.mds.xyz:7182 failed.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1387, in _send_heartbeat
self.cfg.max_cert_depth)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/https.py", line 139, in __init__
self.conn.connect()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/httpslib.py", line 69, in connect
sock.connect((self.host, self.port))
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 309, in connect
ret = self.connect_ssl()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/M2Crypto/SSL/Connection.py", line 295, in connect_ssl
return m2.ssl_connect(self.ssl, self._timeout)
SSLError: sslv3 alert bad certificate
^C
[root@cm-r01en01 pki]# Now I tried to extract the key/cert from the NN generated certs like this (convert to pkcs12 then extract the key/cert): [root@cm-r01nn01 pki]# ls -altri
total 32
135962830 -rw-r--r--. 1 root root 1453 Mar 31 22:41 cm-r01nn01.mws.mds.xyz.pem
201424378 drwxr-xr-x. 5 root root 37 Mar 31 23:42 ..
135962831 lrwxrwxrwx. 1 root root 53 Mar 31 23:44 agent.pem -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.pem
135962833 lrwxrwxrwx. 1 root root 62 Mar 31 23:45 server.jks -> /opt/cloudera/security/pki/cm-r01nn01.mws.mds.xyz.keystore.jks
135962832 -rw-r--r--. 1 cloudera-scm cloudera-scm 4723 Jul 6 00:00 cm-r01nn01.mws.mds.xyz.keystore.jks
135283830 -rw-r--r--. 1 root root 1435 Jul 6 00:10 cm-c01.mws.mds.xyz.pem
135044185 -rw-r--r--. 1 root root 2751 Jul 8 03:49 cm-r01nn01.mws.mds.xyz.keystore.p12
135044187 -rw-r--r--. 1 root root 1691 Jul 8 03:50 cm-r01nn01.mws.mds.xyz.cert.pem
135962829 drwxr-xr-x. 2 root root 4096 Jul 8 03:50 .
135044188 -rw-r--r--. 1 root root 1859 Jul 8 03:50 cm-r01nn01.mws.mds.xyz.key.pem
[root@cm-r01nn01 pki]# pwd
/opt/cloudera/security/pki
[root@cm-r01nn01 pki]# Then copied them above to the cm-r01en01.mws.mds.xyz client (as you probably noticed). Now I'm not clear on my next step. If I read you correctly, seems as if I need the client's key/cert instead? Verifying using openssl yields the same error: [root@cm-r01en01 pki]# openssl s_client -connect cm-r01nn01.mws.mds.xyz:7182 < cm-r01nn01.mws.mds.xyz.pem
CONNECTED(00000003)
depth=0 C = US, ST = California, L = Los Angeles, O = MDS, OU = MDS, CN = cm-r01nn01.mws.mds.xyz
verify return:1
139746469861264:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:s3_pkt.c:1493:SSL alert number 42
139746469861264:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177:
---
Certificate chain
0 s:/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
i:/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
---
Server certificate
-----BEGIN CERTIFICATE-----
MIID9TCCAt2gAwIBAgIEVUJkFTANBgkqhkiG9w0BAQsFADB1MQswCQYDVQQGEwJV
UzETMBEGA1UECBMKQ2FsaWZvcm5pYTEUMBIGA1UEBxMLTG9zIEFuZ2VsZXMxDDAK
BgNVBAoTA01EUzEMMAoGA1UECxMDTURTMR8wHQYDVQQDExZjbS1yMDFubjAxLm13
cy5tZHMueHl6MB4XDTE5MDQwMTAyMzc1MFoXDTI5MDMyOTAyMzc1MFowdTELMAkG
A1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFDASBgNVBAcTC0xvcyBBbmdl
bGVzMQwwCgYDVQQKEwNNRFMxDDAKBgNVBAsTA01EUzEfMB0GA1UEAxMWY20tcjAx
bm4wMS5td3MubWRzLnh5ejCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AIVFcUGgk3fKC1zuKuSTmc6HQLIxvI9fZXjWMOwYagoJjAgKEFUKtjUTgY15IqT9
VXeBuOICTzDvXQbHEh87KAVLoyQa+DkBR9yoyj5hhRUz+fG7erVgvf1AyB5V289Q
ZZir70nzrnRfGdGh4gFCPH2CKvTBndvLYIbDlSh8P5gjLyck7KMW7GrbtZfxcTGD
qBJYB8okK6/Fs6fYD5UggfYIVhEgAorciHCYlnsJagSda/8Mn5a9WtYLjMJY8hEW
RjZxLLI6aunrd/J24ulUEURaWiqjtfna7Q++mDL4GLtf/sUe4xYjt5zy7qMWPh6Q
aGNVcR+bMVgs+e2qATPbiLkCAwEAAaOBjDCBiTBFBgNVHSUEPjA8BggrBgEFBQcD
AQYIKwYBBQUHAwIGCCsGAQUFBwMDBggrBgEFBQcDBAYIKwYBBQUHAwgGCCsGAQUF
BwMJMCEGA1UdEQQaMBiCFmNtLXIwMW5uMDEubXdzLm1kcy54eXowHQYDVR0OBBYE
FCE0pxrJ9cnfzdjidWWTm5YQFoZ/MA0GCSqGSIb3DQEBCwUAA4IBAQBe9m5Tg76b
Uux3yZudIAtISKwSGP2hz4kYpGEM5ykjiY4UAtScFHLcPEJxFN6K6KtZDy02nStI
aLE9dNB3SayLspC6nC1gAB05D8viFKniOVSG8TQgTytOHG/A7UWz5yQF4uJXiETe
de82xiMt75O1jaePFrsMG2twSZxRTLpd52WGit4A5B8LM2ADTR/wsFvQMhvZA+y5
N+1/pw8bQK4SGGWZ4DWXP+Q5bC/6xP5gN1H9bmvvZILNVma7w+ko7Wr8qIfRB8RU
2j1EiHAp99CjA1on1XVH//3TAhRhvXojh+YnZAwe4JCkCI6vVEYSkbgWBK1txYEa
uZw5Rx5pl+gE
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
issuer=/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
---
Acceptable client certificate CA names
/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-r01nn01.mws.mds.xyz
/C=US/ST=California/L=Los Angeles/O=MDS/OU=MDS/CN=cm-c01.mws.mds.xyz
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Shared Requested Signature Algorithms: ECDSA+SHA512:RSA+SHA512:ECDSA+SHA384:RSA+SHA384:ECDSA+SHA256:RSA+SHA256:DSA+SHA256:ECDSA+SHA224:RSA+SHA224:DSA+SHA224:ECDSA+SHA1:RSA+SHA1:DSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1733 bytes and written 138 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 5D25323127BD39A3A3E9C85D12EA95BB46D7EF0D1A2D5EBCD68D502DC140431F
Session-ID-ctx:
Master-Key: 96673D16341D315F828D023D34D65937D8BCBA58437F9926B74450998EF22C3CBC038BD099F815CFC6053F4630490362
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
Start Time: 1562718769
Timeout : 300 (sec)
Verify return code: 0 (ok)
---
[root@cm-r01en01 pki]# Admin -> Settings: Use TLS Encryption for Admin Console (Enabled) Use TLS Encryption for Agents (Enabled) Use TLS Authentication of Agents to Server (Enabled) Verify Agent Hostname Against Certificate (Enabled) Cloudera Manager TLS/SSL Server JKS Keystore File Location = /opt/cloudera/security/pki/server.jks Cloudera Manager TLS/SSL Server JKS Keystore File Password = <PASS = cm-r01nn01.mws.mds.xyz> Supported SSL/TLS versions = SSLv2Hello, and TLSv1.2 CMS -> Configuration TLS/SSL Client Truststore File Location = /etc/pki/ca-trust/extracted/java/jssecacerts Cloudera Manager Server TLS/SSL Client Trust Store Password = <PASS = changeit > Options 2-3 above are the two that I don't fully grasp yet. Cheers, TK
... View more
