Created on 07-17-2017 12:55 PM - edited 09-16-2022 04:56 AM
I have a kerberized CDH 5.9 cluster which has been running fine for a month now since securing it. I installed Kafka through Parcels, and then went to add the service to CM, and I've been running into an error while starting the brokers. All three broker nodes are complaining because they cannot access /var/run/scm-cloudera-agent/process/*-kafka-KAFKA_BROKER/supervisor.conf, but the start script seems to create a new folder with supervisor.conf in it every time. The file is set to 600 permissions with the owner as root, so its no wonder permission is denied, but I don't understand how this is supposed to work or what I'm missing that is causing it to not work in this case. I'm including a snippet of the stderr log (the whole log was too long to fit in this post) from one of the brokers below. Note that the permissions error comes up 4 times on each broker's log.
+ for i in '`seq 1 $COUNT`' + SCRIPT=/opt/cloudera/parcels/KAFKA-2.2.0-1.2.2.0.p0.68/meta/kafka_env.sh + PARCEL_DIRNAME=KAFKA-2.2.0-1.2.2.0.p0.68 + . /opt/cloudera/parcels/KAFKA-2.2.0-1.2.2.0.p0.68/meta/kafka_env.sh ++ KAFKA_DIRNAME=KAFKA-2.2.0-1.2.2.0.p0.68 ++ export KAFKA_HOME=/opt/cloudera/parcels/KAFKA-2.2.0-1.2.2.0.p0.68/lib/kafka ++ KAFKA_HOME=/opt/cloudera/parcels/KAFKA-2.2.0-1.2.2.0.p0.68/lib/kafka + echo 'Using /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER as conf dir' + echo 'Using scripts/control.sh as process script' + replace_conf_dir + find /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER -type f '!' -path '/var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER/logs/*' '!' -name '*.log' '!' -name '*.keytab' '!' -name '*jceks' -exec perl -pi -e 's#{{CMF_CONF_DIR}}#/var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER#g' '{}' ';' Can't open /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER/supervisor.conf: Permission denied. + make_scripts_executable + find /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER -regex '.*\.\(py\|sh\)$' -exec chmod u+x '{}' ';' + export COMMON_SCRIPT=/usr/lib64/cmf/service/common/cloudera-config.sh + COMMON_SCRIPT=/usr/lib64/cmf/service/common/cloudera-config.sh + chmod u+x /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER/scripts/control.sh + exec /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER/scripts/control.sh start + DEFAULT_KAFKA_HOME=/usr/lib/kafka + KAFKA_HOME=/opt/cloudera/parcels/KAFKA-2.2.0-1.2.2.0.p0.68/lib/kafka + MIN_KAFKA_MAJOR_VERSION_WITH_SSL=2
Created 07-17-2017 01:53 PM
Created 07-17-2017 01:53 PM
Created on 07-17-2017 02:07 PM - edited 07-17-2017 02:08 PM
I've not seen anything else in either stdout or stderr. I'm attaching the entirety of stdout here for posterity. The stderr can be found at the link below:
Mon Jul 17 14:32:38 CDT 2017 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kafka_kafka-KAFKA_BROKER-c1b2741bf1f3e232c6c549ec045bcb79_pid41162.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh as CSD_JAVA_OPTS Using /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER as conf dir Using scripts/control.sh as process script Date: Mon Jul 17 14:32:38 CDT 2017 Host: svd0hdatn03.XXXX.corp Pwd: /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER CONF_DIR: /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER KAFKA_HOME: /opt/cloudera/parcels/KAFKA-2.2.0-1.2.2.0.p0.68/lib/kafka Zookeeper Quorum: svd0hdatn01.XXXX.corp:2181,svd0hmstn01.XXXX.corp:2181,svd0hmstn02.XXXX.corp:2181 Zookeeper Chroot: PORT: 9092 JMX_PORT: 9393 SSL_PORT: 9093 ENABLE_MONITORING: true METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter BROKER_HEAP_SIZE: 1024 BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true BROKER_SSL_ENABLED: false KERBEROS_AUTH_ENABLED: false KAFKA_PRINCIPAL: kafka/svd0hdatn03.XXXX.corp@XXXX.CORP SECURITY_INTER_BROKER_PROTOCOL: INFERRED AUTHENTICATE_ZOOKEEPER_CONNECTION: true SUPER_USERS: kafka Kafka version found: 0.10.2-kafka2.2.0 ZK_PRINCIPAL_NAME: zookeeper Final Zookeeper Quorum is svd0hdatn01.XXXX.corp:2181,svd0hmstn01.XXXX.corp:2181,svd0hmstn02.XXXX.corp:2181 security.inter.broker.protocol inferred as PLAINTEXT LISTENERS=listeners=PLAINTEXT://svd0hdatn03.XXXX.corp:9092, Mon Jul 17 14:32:43 CDT 2017 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kafka_kafka-KAFKA_BROKER-c1b2741bf1f3e232c6c549ec045bcb79_pid41853.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh as CSD_JAVA_OPTS Using /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER as conf dir Using scripts/control.sh as process script Date: Mon Jul 17 14:32:43 CDT 2017 Host: svd0hdatn03.XXXX.corp Pwd: /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER CONF_DIR: /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER KAFKA_HOME: /opt/cloudera/parcels/KAFKA-2.2.0-1.2.2.0.p0.68/lib/kafka Zookeeper Quorum: svd0hdatn01.XXXX.corp:2181,svd0hmstn01.XXXX.corp:2181,svd0hmstn02.XXXX.corp:2181 Zookeeper Chroot: PORT: 9092 JMX_PORT: 9393 SSL_PORT: 9093 ENABLE_MONITORING: true METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter BROKER_HEAP_SIZE: 1024 BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true BROKER_SSL_ENABLED: false KERBEROS_AUTH_ENABLED: false KAFKA_PRINCIPAL: kafka/svd0hdatn03.XXXX.corp@XXXX.CORP SECURITY_INTER_BROKER_PROTOCOL: INFERRED AUTHENTICATE_ZOOKEEPER_CONNECTION: true SUPER_USERS: kafka Kafka version found: 0.10.2-kafka2.2.0 ZK_PRINCIPAL_NAME: zookeeper Final Zookeeper Quorum is svd0hdatn01.XXXX.corp:2181,svd0hmstn01.XXXX.corp:2181,svd0hmstn02.XXXX.corp:2181 security.inter.broker.protocol inferred as PLAINTEXT LISTENERS=listeners=PLAINTEXT://svd0hdatn03.XXXX.corp:9092, Mon Jul 17 14:32:49 CDT 2017 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kafka_kafka-KAFKA_BROKER-c1b2741bf1f3e232c6c549ec045bcb79_pid42550.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh as CSD_JAVA_OPTS Using /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER as conf dir Using scripts/control.sh as process script Date: Mon Jul 17 14:32:49 CDT 2017 Host: svd0hdatn03.XXXX.corp Pwd: /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER CONF_DIR: /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER KAFKA_HOME: /opt/cloudera/parcels/KAFKA-2.2.0-1.2.2.0.p0.68/lib/kafka Zookeeper Quorum: svd0hdatn01.XXXX.corp:2181,svd0hmstn01.XXXX.corp:2181,svd0hmstn02.XXXX.corp:2181 Zookeeper Chroot: PORT: 9092 JMX_PORT: 9393 SSL_PORT: 9093 ENABLE_MONITORING: true METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter BROKER_HEAP_SIZE: 1024 BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true BROKER_SSL_ENABLED: false KERBEROS_AUTH_ENABLED: false KAFKA_PRINCIPAL: kafka/svd0hdatn03.XXXX.corp@XXXX.CORP SECURITY_INTER_BROKER_PROTOCOL: INFERRED AUTHENTICATE_ZOOKEEPER_CONNECTION: true SUPER_USERS: kafka Kafka version found: 0.10.2-kafka2.2.0 ZK_PRINCIPAL_NAME: zookeeper Final Zookeeper Quorum is svd0hdatn01.XXXX.corp:2181,svd0hmstn01.XXXX.corp:2181,svd0hmstn02.XXXX.corp:2181 security.inter.broker.protocol inferred as PLAINTEXT LISTENERS=listeners=PLAINTEXT://svd0hdatn03.XXXX.corp:9092, Mon Jul 17 14:32:57 CDT 2017 JAVA_HOME=/usr/java/jdk1.7.0_67-cloudera Using -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/kafka_kafka-KAFKA_BROKER-c1b2741bf1f3e232c6c549ec045bcb79_pid43296.hprof -XX:OnOutOfMemoryError=/usr/lib64/cmf/service/common/killparent.sh as CSD_JAVA_OPTS Using /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER as conf dir Using scripts/control.sh as process script Date: Mon Jul 17 14:32:57 CDT 2017 Host: svd0hdatn03.XXXX.corp Pwd: /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER CONF_DIR: /var/run/cloudera-scm-agent/process/1464-kafka-KAFKA_BROKER KAFKA_HOME: /opt/cloudera/parcels/KAFKA-2.2.0-1.2.2.0.p0.68/lib/kafka Zookeeper Quorum: svd0hdatn01.XXXX.corp:2181,svd0hmstn01.XXXX.corp:2181,svd0hmstn02.XXXX.corp:2181 Zookeeper Chroot: PORT: 9092 JMX_PORT: 9393 SSL_PORT: 9093 ENABLE_MONITORING: true METRIC_REPORTERS: nl.techop.kafka.KafkaHttpMetricsReporter BROKER_HEAP_SIZE: 1024 BROKER_JAVA_OPTS: -server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true BROKER_SSL_ENABLED: false KERBEROS_AUTH_ENABLED: false KAFKA_PRINCIPAL: kafka/svd0hdatn03.XXXX.corp@XXXX.CORP SECURITY_INTER_BROKER_PROTOCOL: INFERRED AUTHENTICATE_ZOOKEEPER_CONNECTION: true SUPER_USERS: kafka Kafka version found: 0.10.2-kafka2.2.0 ZK_PRINCIPAL_NAME: zookeeper Final Zookeeper Quorum is svd0hdatn01.XXXX.corp:2181,svd0hmstn01.XXXX.corp:2181,svd0hmstn02.XXXX.corp:2181 security.inter.broker.protocol inferred as PLAINTEXT LISTENERS=listeners=PLAINTEXT://svd0hdatn03.XXXX.corp:9092,
Created 07-17-2017 02:11 PM
Found the issue. In /var/log/*kafka*, there was a log file that had an exception logged in it that I've included below. After I went back and enabled Kerberos authorization for Kafka (I had not enabled it thinking I could do that later), everything started successfully. Thanks for pointing me to that log file!
java.lang.RuntimeException: Unable to create KafkaAuthBinding: keytabFile required because kerberos is enabled at org.apache.sentry.kafka.binding.KafkaAuthBindingSingleton.configure(KafkaAuthBindingSingleton.java:67) at org.apache.sentry.kafka.authorizer.SentryKafkaAuthorizer.configure(SentryKafkaAuthorizer.java:120) at kafka.server.KafkaServer$$anonfun$startup$4.apply(KafkaServer.scala:236) at kafka.server.KafkaServer$$anonfun$startup$4.apply(KafkaServer.scala:234) at scala.Option.map(Option.scala:146) at kafka.server.KafkaServer.startup(KafkaServer.scala:234) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:39) at kafka.Kafka$.main(Kafka.scala:67) at com.cloudera.kafka.wrap.Kafka$.main(Kafka.scala:76) at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala) Caused by: java.lang.IllegalArgumentException: keytabFile required because kerberos is enabled at org.apache.sentry.kafka.binding.KafkaAuthBinding.initKerberos(KafkaAuthBinding.java:561) at org.apache.sentry.kafka.binding.KafkaAuthBinding.createAuthProvider(KafkaAuthBinding.java:142) at org.apache.sentry.kafka.binding.KafkaAuthBinding.<init>(KafkaAuthBinding.java:97) at org.apache.sentry.kafka.binding.KafkaAuthBindingSingleton.configure(KafkaAuthBindingSingleton.java:63) ... 9 more
Created 03-17-2021 05:21 PM
This error shows up if you have selected Sentry/Ranger as dependencies but not checked true for the below config (i.e. did not enable Kerberos)
kerberos.auth.enable