Member since
07-25-2016
5
Posts
0
Kudos Received
0
Solutions
09-29-2017
01:46 PM
Hi @Ramesh Mani , @Carter Everett I am trying to send HDFS Ranger Audit logs to kafka via Log4j. I am using HDP 2.5 Sandbox where I have Ranger 0.6 and kafka 0.10.0.1. I have added the below in Custom ranger_hdfs_audit using Ambari
xasecure.audit.destination.log4j=true xasecure.audit.log4j.is.enabled xasecure.audit.destination.log4j.logger=xaaudit I have also added the below in Advanced hdfs-log4j using Ambari #Kafka Appender ranger.logger=INFO,console,KAFKA log4j.logger.xaaudit=${ranger.logger} log4j.appender.KAFKA=org.apache.kafka.log4jappender.KafkaLog4jAppender log4j.appender.KAFKA.layout=org.apache.log4j.PatternLayout log4j.appender.KAFKA.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L %% - %m%n log4j.appender.KAFKA.BrokerList=sandbox.hortonworks.com:6667 log4j.appender.KAFKA.Topic=HDFS_AUDIT_LOG log4j.appender.KAFKA.ProducerType=sync and then restarted HDFS (namenode, Datanode and other dependencies) But now when i do a hdfs dfs -ls / , I get the below error: [root@sandbox ~]# hdfs dfs -ls / log4j:ERROR Could not instantiate class [org.apache.kafka.log4jappender.KafkaLog4jAppender]. java.lang.ClassNotFoundException: org.apache.kafka.log4jappender.KafkaLog4jAppender at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.log4j.helpers.Loader.loadClass(Loader.java:198) at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:327) at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:124) at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:785) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:768) at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:672) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:516) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:580) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.apache.log4j.Logger.getLogger(Logger.java:104) at org.apache.commons.logging.impl.Log4JLogger.getLogger(Log4JLogger.java:262) at org.apache.commons.logging.impl.Log4JLogger.<init>(Log4JLogger.java:108) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.commons.logging.impl.LogFactoryImpl.createLogFromClass(LogFactoryImpl.java:1025) at org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:790) at org.apache.commons.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541) at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292) at org.apache.commons.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269) at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657) at org.apache.hadoop.fs.FsShell.<clinit>(FsShell.java:43) log4j:ERROR Could not instantiate appender named "KAFKA". Found 12 items drwxrwxrwx - yarn hadoop 0 2016-10-25 08:10 /app-logs drwxr-xr-x - hdfs hdfs 0 2016-10-25 07:54 /apps drwxr-xr-x - yarn hadoop 0 2016-10-25 07:48 /ats drwxr-xr-x - hdfs hdfs 0 2016-10-25 08:01 /demo and so on .... Could you please help me on this. Regards, Vishnu Sure.
... View more