Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache Kafka-Log4j-Appender not working spark cluster mode

Highlighted

Apache Kafka-Log4j-Appender not working spark cluster mode

New Contributor
  • i try to sends both spark driver/executor log messages to kafka topic in clusert mode. it is not sending logs to topic while try to send from local it will work.
  • i used spark submit script to run my spark application in standaloane cluster mode.
  • I added spark corespark-sqlspark stream kafkaspark streamingkafka streaming (2.11.2.2.0) and kafka-log4j-appender2.3.0 maven's but still it is showing org.apche.kafka.log4jappender ClassNotFoundException.
  • g4j.rootLogger=INFO, console, KAFKAlog4j.appender.console=org.apache.log4j.ConsoleAppenderlog4j.appender.console.target=System.err
    log4j.appender.console.layout=org.apache.log4j.PatternLayoutlog4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
    log4j.appender.KAFKA=org.apache.kafka.log4jappender.KafkaLog4jAppenderlog4j.appender.KAFKA.topic=test2
    log4j.appender.KAFKA.name=localhost
    log4j.appender.KAFKA.host=localhost
    log4j.appender.KAFKA.port=9092log4j.appender.KAFKA.brokerList=localhost:9092log4j.appender.KAFKA.compressionType=none
    log4j.appender.KAFKA.requiredNumAcks=0log4j.appender.KAFKA.syncSend=truelog4j.appender.KAFKA.layout=org.apache.log4j.PatternLayoutlog4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n

    Can anyone please help to short out this issue.