Member since
08-24-2019
1
Post
0
Kudos Received
0
Solutions
08-24-2019
10:24 PM
i try to sends both spark driver/executor log messages to kafka topic in clusert mode. it is not sending logs to topic while try to send from local it will work.
i used spark submit script to run my spark application in standaloane cluster mode.
I added spark core , spark-sql , spark stream kafka , spark streaming , kafka streaming (2.11.2.2.0) and kafka-log4j-appender2.3.0 maven's but still it is showing org.apche.kafka.log4jappender ClassNotFoundException.
g4j . rootLogger = INFO , console , KAFKA log4j . appender . console = org . apache . log4j . ConsoleAppender log4j . appender . console . target = System . err
log4j . appender . console . layout = org . apache . log4j . PatternLayout log4j . appender . console . layout . ConversionPattern =% d { yy / MM / dd HH : mm : ss } % p % c { 1 }: % m % n
log4j . appender . KAFKA = org . apache . kafka . log4jappender . KafkaLog4jAppender log4j . appender . KAFKA . topic = test2
log4j . appender . KAFKA . name = localhost
log4j . appender . KAFKA . host = localhost
log4j . appender . KAFKA . port = 9092 log4j . appender . KAFKA . brokerList = localhost : 9092 log4j . appender . KAFKA . compressionType = none
log4j . appender . KAFKA . requiredNumAcks = 0 log4j . appender . KAFKA . syncSend = true log4j . appender . KAFKA . layout = org . apache . log4j . PatternLayout log4j . appender . KAFKA . layout . ConversionPattern =% d { yyyy - MM - dd HH : mm : ss } %- 5p % c { 1 }:% L %% - % m % n
Can anyone please help to short out this issue.
... View more
Labels: