question Re: Kafka audit Logs stored in HDFS in Support Questions

Kafka audit Logs stored in HDFS

psilvarochagome — Tue, 02 Apr 2019 19:41:45 GMT

Hello,

I have a scenario with a Hadoop cluster installed with HDP2.6.5 and a Kafka cluster installed with HDF 3.3.0 with Ranger Service configured.

I want to store the Ranger Audit logs in HDFS so I setup in kafka the property xasecure.audit.destination.hdfs.dir pointing to the HDFS directory.

Case one: when using the namenode in the URI the logs are stored in HDFS successfully (xasecure.audit.destination.hdfs.dir=hdfs://namenode_FQDN>:8020/ranger/audit)

Case two: Using a haproxy, since i have namenode HA enabled and want to point always to the active NN, i get the following error

2019-04-02 12:00:13,841 ERROR [kafka.async.summary.multi_dest.batch_kafka.async.summary.multi_dest.batch.hdfs_destWriter] org.apache.ranger.audit.provider.BaseAuditHandler (BaseAuditHandler.java:329) - Error writing to log file.
java.io.IOException: DestHost:destPort <ha_proxy_hostname>:8085 , LocalHost:localPort <kafka_broker_hostname>/10.212.164.50:0. Failed on local exception: java.io.IOException: org.apache.hadoop.ipc.RpcException: RPC response exceeds maximum data length

Is there any extra config to be set?

Thanks

Re: Kafka audit Logs stored in HDFS

gadibalaji — Wed, 03 Apr 2019 23:42:37 GMT

Hi,

Could you please share hdf instalaltion document. I want to install HDF installation on my personal computer.

Re: Kafka audit Logs stored in HDFS

psilvarochagome — Tue, 09 Apr 2019 14:20:30 GMT

Anyone can help on this topic?

Re: Kafka audit Logs stored in HDFS

psilvarochagome — Tue, 09 Apr 2019 14:22:44 GMT

I followed the steps in this link https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.2.0/installing-hdf/content/install-ambari.html

Re: Kafka audit Logs stored in HDFS

psilvarochagome — Fri, 26 Apr 2019 19:57:18 GMT

Found how to proceed and now i can store the logs in HDFS

Re: Kafka audit Logs stored in HDFS

slim_abderrahim — Tue, 17 Sep 2019 08:44:42 GMT

Could you please share how did you proceed ?

Thank you

Re: Kafka audit Logs stored in HDFS

Shelton — Tue, 17 Sep 2019 21:11:44 GMT

@psilvarochagome
In this community, we share knowledge to advance the Cloudera community and don't get cash for that! though some are real production issues, having said that it's unfortunate people like you got a solution to a problem being faced by a member and don't want to share as requested by @slim_abderrahim

It's very unfortunate I hope member see this and tag you ... .........we open-source as opposed to proprietary code. 🙂
Happy hadooping

Re: Kafka audit Logs stored in HDFS

psilvarochagome — Thu, 19 Sep 2019 08:18:44 GMT

Actually i didn't share because i didn't get the notification about this message. Obviously will do it.

Best

Paula

Re: Kafka audit Logs stored in HDFS

psilvarochagome — Thu, 19 Sep 2019 08:32:59 GMT

Hi, this is what i did:

In Ambari, select

Kafka→ Configs→ Advanced ranger-kafka-audit and add the dfs destination dir

(if you have NameNode HA, you need to add to each kafka broker the hdfs-site.xml that has the nameservice property, so the audit logs should always hit the active namenode)

For example if you have defined the fs.defaultFS=nameservice you will add something like

xasecure.audit.destination.hdfs.dir=hdfs://nameservice/ranger/audit

Then restart the brokers.

Hope it helps

Re: Kafka audit Logs stored in HDFS

psilvarochagome — Thu, 19 Sep 2019 08:36:06 GMT

Sorry, forgot to add the port

the correct way will be

hdfs://nameservice:8020/ranger/audit

Re: Kafka audit Logs stored in HDFS

slim_abderrahim — Fri, 04 Oct 2019 12:01:27 GMT

It works only if it's the same KDC, need cross trust (realm) in my case.

Thank you.

ps : I didn't got the notification too

Regards